-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Incorrect values for memory usage being displayed #1346
Comments
Is this something that always happens always, or just sometimes? Can you paste the output of |
Not sure if im reading the endpoint correctly, although that redirects to:
Being the kubernetes bash not an API endpoint. Is that correct? If so i can dump that info |
The full endpoint is: |
{
"listMeta": {
"totalItems": 11
},
"pods": [
{
"objectMeta": {
"name": "es-client-2600856264-0bubb",
"namespace": "default",
"labels": {
"name": "elasticsearch",
"pod-template-hash": "2600856264",
"role": "client"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"es-client-2600856264\",\"uid\":\"27094f41-9a7b-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"633398\"}}\n",
"kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container es-client"
},
"creationTimestamp": "2016-10-25T06:20:40Z"
},
"typeMeta": {
"kind": "pod"
},
"podPhase": "Running",
"podIP": "10.92.0.30",
"restartCount": 0,
"metrics": {
"cpuUsage": 3,
"memoryUsage": 1843240960,
"cpuUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 13
}
],
"memoryUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 1843240960
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 1843240960
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 1843077120
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 1843077120
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 1843077120
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 1843081216
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 1843081216
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 1842909184
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 1842909184
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 1842909184
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 1844269056
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 1844187136
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 1844187136
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 1844244480
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 1847726080
}
]
}
},
{
"objectMeta": {
"name": "es-client-2600856264-3nya8",
"namespace": "default",
"labels": {
"name": "elasticsearch",
"pod-template-hash": "2600856264",
"role": "client"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"es-client-2600856264\",\"uid\":\"27094f41-9a7b-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"633650\"}}\n",
"kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container es-client"
},
"creationTimestamp": "2016-10-25T06:23:14Z"
},
"typeMeta": {
"kind": "pod"
},
"podPhase": "Running",
"podIP": "10.92.1.24",
"restartCount": 0,
"metrics": {
"cpuUsage": 53,
"memoryUsage": 1851748352,
"cpuUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 53
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 16
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 7
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 6
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 6
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 7
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 9
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 6
}
],
"memoryUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 1851748352
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 1855627264
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 1855655936
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 1855541248
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 1855733760
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 1855995904
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 1856000000
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 1856000000
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 1856094208
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 1856094208
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 1856266240
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 1856266240
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 1856266240
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 1852932096
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 1853067264
}
]
}
},
{
"objectMeta": {
"name": "es-data-3827280009-8cmer",
"namespace": "default",
"labels": {
"name": "elasticsearch",
"pod-template-hash": "3827280009",
"role": "data"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"es-data-3827280009\",\"uid\":\"3a53ecc0-9a7b-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"633665\"}}\n",
"kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container es-data"
},
"creationTimestamp": "2016-10-25T06:23:19Z"
},
"typeMeta": {
"kind": "pod"
},
"podPhase": "Running",
"podIP": "10.92.1.25",
"restartCount": 0,
"metrics": {
"cpuUsage": 5,
"memoryUsage": 1854390272,
"cpuUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 20
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 6
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 8
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 4
}
],
"memoryUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 1854390272
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 1855029248
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 1855033344
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 1855033344
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 1855033344
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 1855033344
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 1855078400
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 1855086592
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 1855086592
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 1855086592
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 1853890560
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 1853894656
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 1853915136
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 1853816832
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 1853837312
}
]
}
},
{
"objectMeta": {
"name": "es-data-3827280009-rynuu",
"namespace": "default",
"labels": {
"name": "elasticsearch",
"pod-template-hash": "3827280009",
"role": "data"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"es-data-3827280009\",\"uid\":\"3a53ecc0-9a7b-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"633459\"}}\n",
"kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container es-data"
},
"creationTimestamp": "2016-10-25T06:21:13Z"
},
"typeMeta": {
"kind": "pod"
},
"podPhase": "Running",
"podIP": "10.92.0.31",
"restartCount": 0,
"metrics": {
"cpuUsage": 93,
"memoryUsage": 1928818688,
"cpuUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 93
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 7
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 3
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 7
}
],
"memoryUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 1928818688
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 1928843264
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 1928871936
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 1928884224
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 1928884224
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 1929478144
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 1909510144
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 1909604352
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 1909776384
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 1912520704
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 1911529472
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 1911537664
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 1911386112
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 1911287808
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 1913962496
}
]
}
},
{
"objectMeta": {
"name": "es-master-709150228-sqroo",
"namespace": "default",
"labels": {
"name": "elasticsearch",
"pod-template-hash": "709150228",
"role": "master"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"es-master-709150228\",\"uid\":\"6ed710c2-9a7a-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"632952\"}}\n",
"kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container es-master"
},
"creationTimestamp": "2016-10-25T06:15:31Z"
},
"typeMeta": {
"kind": "pod"
},
"podPhase": "Running",
"podIP": "10.92.0.29",
"restartCount": 0,
"metrics": {
"cpuUsage": 4,
"memoryUsage": 1832398848,
"cpuUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 6
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 4
}
],
"memoryUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 1832398848
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 1832398848
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 1832398848
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 1832402944
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 1832407040
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 1832407040
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 1832407040
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 1832972288
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 1832980480
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 1832980480
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 1832984576
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 1832984576
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 1832988672
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 1832988672
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 1832988672
}
]
}
},
{
"objectMeta": {
"name": "es-master-709150228-tycra",
"namespace": "default",
"labels": {
"name": "elasticsearch",
"pod-template-hash": "709150228",
"role": "master"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"es-master-709150228\",\"uid\":\"6ed710c2-9a7a-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"633633\"}}\n",
"kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container es-master"
},
"creationTimestamp": "2016-10-25T06:23:07Z"
},
"typeMeta": {
"kind": "pod"
},
"podPhase": "Running",
"podIP": "10.92.1.23",
"restartCount": 0,
"metrics": {
"cpuUsage": 5,
"memoryUsage": 1852928000,
"cpuUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 19
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 6
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 4
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 5
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 6
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 5
}
],
"memoryUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 1852928000
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 1855897600
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 1855897600
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 1855897600
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 1855901696
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 1855905792
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 1855905792
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 1855905792
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 1855905792
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 1855905792
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 1855422464
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 1855422464
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 1855422464
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 1854828544
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 1854337024
}
]
}
},
{
"objectMeta": {
"name": "ethical-jobs-api-412768213-asvtr",
"namespace": "default",
"labels": {
"application": "ethical-jobs",
"name": "api",
"pod-template-hash": "412768213",
"tier": "backend"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"ethical-jobs-api-412768213\",\"uid\":\"a599952c-9a77-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"631238\"}}\n",
"kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container api; cpu request for container schedule; cpu request for container queue"
},
"creationTimestamp": "2016-10-25T05:55:35Z"
},
"typeMeta": {
"kind": "pod"
},
"podPhase": "Running",
"podIP": "10.92.0.28",
"restartCount": 0,
"metrics": {
"cpuUsage": 11,
"memoryUsage": 91222016,
"cpuUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 11
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 14
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 15
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 13
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 13
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 11
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 15
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 22
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 14
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 13
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 14
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 8
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 10
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 12
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 10
}
],
"memoryUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 91222016
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 91217920
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 90988544
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 90996736
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 90984448
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 91226112
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 91213824
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 138809344
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 139001856
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 138899456
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 91238400
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 91000832
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 91000832
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 91009024
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 83542016
}
]
}
},
{
"objectMeta": {
"name": "ethical-jobs-api-412768213-pf68h",
"namespace": "default",
"labels": {
"application": "ethical-jobs",
"name": "api",
"pod-template-hash": "412768213",
"tier": "backend"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"ethical-jobs-api-412768213\",\"uid\":\"a599952c-9a77-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"631229\"}}\n",
"kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container api; cpu request for container schedule; cpu request for container queue"
},
"creationTimestamp": "2016-10-25T05:55:35Z"
},
"typeMeta": {
"kind": "pod"
},
"podPhase": "Running",
"podIP": "10.92.1.22",
"restartCount": 0,
"metrics": {
"cpuUsage": 25,
"memoryUsage": 79966208,
"cpuUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 25
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 8
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 24
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 21
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 20
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 16
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 16
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 28
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 20
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 22
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 27
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 16
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 26
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 18
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 16
}
],
"memoryUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 79966208
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 79962112
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 79966208
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 81285120
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 81281024
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 79962112
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 79966208
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 126132224
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 127561728
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 127676416
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 79966208
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 81268736
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 79790080
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 79728640
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 79732736
}
]
}
},
{
"objectMeta": {
"name": "ethical-jobs-employers-1823474103-lnr9d",
"namespace": "default",
"labels": {
"application": "ethical-jobs",
"name": "employers",
"pod-template-hash": "1823474103",
"tier": "web"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"ethical-jobs-employers-1823474103\",\"uid\":\"43c05a5e-9a65-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"620117\"}}\n",
"kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container employers"
},
"creationTimestamp": "2016-10-25T03:44:00Z"
},
"typeMeta": {
"kind": "pod"
},
"podPhase": "Running",
"podIP": "10.92.0.25",
"restartCount": 0,
"metrics": {
"cpuUsage": 0,
"memoryUsage": 52666368,
"cpuUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 0
}
],
"memoryUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 52666368
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 52809728
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 52895744
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 53100544
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 53338112
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 53350400
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 52711424
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 52834304
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 52936704
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 53063680
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 53186560
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 53329920
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 53440512
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 53542912
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 52916224
}
]
}
},
{
"objectMeta": {
"name": "ethical-jobs-employers-1823474103-xd2nx",
"namespace": "default",
"labels": {
"application": "ethical-jobs",
"name": "employers",
"pod-template-hash": "1823474103",
"tier": "web"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"ethical-jobs-employers-1823474103\",\"uid\":\"43c05a5e-9a65-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"620127\"}}\n",
"kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container employers"
},
"creationTimestamp": "2016-10-25T03:44:00Z"
},
"typeMeta": {
"kind": "pod"
},
"podPhase": "Running",
"podIP": "10.92.1.21",
"restartCount": 0,
"metrics": {
"cpuUsage": 0,
"memoryUsage": 51040256,
"cpuUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 0
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 0
}
],
"memoryUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 51040256
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 51146752
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 51269632
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 51417088
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 51544064
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 50868224
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 50991104
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 51118080
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 51281920
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 51388416
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 51511296
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 51658752
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 51724288
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 51052544
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 51200000
}
]
}
},
{
"objectMeta": {
"name": "redis-316302323-d8xgn",
"namespace": "default",
"labels": {
"application": "ethical-jobs",
"name": "redis",
"pod-template-hash": "316302323",
"tier": "service"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"redis-316302323\",\"uid\":\"9e318153-965d-11e6-9a32-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"1272\"}}\n"
},
"creationTimestamp": "2016-10-20T00:39:11Z"
},
"typeMeta": {
"kind": "pod"
},
"podPhase": "Running",
"podIP": "10.92.1.8",
"restartCount": 0,
"metrics": {
"cpuUsage": 1,
"memoryUsage": 2826240,
"cpuUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 1
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 1
}
],
"memoryUsageHistory": [
{
"timestamp": "2016-10-25T06:54:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T06:55:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T06:56:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T06:57:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T06:58:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T06:59:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T07:00:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T07:01:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T07:02:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T07:03:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T07:04:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T07:05:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T07:06:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T07:07:00Z",
"value": 2826240
},
{
"timestamp": "2016-10-25T07:08:00Z",
"value": 2826240
}
]
}
}
],
"cumulativeMetrics": [
{
"dataPoints": [
{
"x": 1477378440,
"y": 200
},
{
"x": 1477378500,
"y": 88
},
{
"x": 1477378560,
"y": 69
},
{
"x": 1477378620,
"y": 61
},
{
"x": 1477378680,
"y": 60
},
{
"x": 1477378740,
"y": 57
},
{
"x": 1477378800,
"y": 54
},
{
"x": 1477378860,
"y": 76
},
{
"x": 1477378920,
"y": 57
},
{
"x": 1477378980,
"y": 62
},
{
"x": 1477379040,
"y": 70
},
{
"x": 1477379100,
"y": 52
},
{
"x": 1477379160,
"y": 62
},
{
"x": 1477379220,
"y": 67
},
{
"x": 1477379280,
"y": 66
}
],
"metricName": "cpu/usage_rate",
"aggregation": "sum"
},
{
"dataPoints": [
{
"x": 1477378440,
"y": 11441246208
},
{
"x": 1477378500,
"y": 11448999936
},
{
"x": 1477378560,
"y": 11448881152
},
{
"x": 1477378620,
"y": 11450462208
},
{
"x": 1477378680,
"y": 11451011072
},
{
"x": 1477378740,
"y": 11450134528
},
{
"x": 1477378800,
"y": 11429691392
},
{
"x": 1477378860,
"y": 11524198400
},
{
"x": 1477378920,
"y": 11526361088
},
{
"x": 1477378980,
"y": 11529351168
},
{
"x": 1477379040,
"y": 11433091072
},
{
"x": 1477379100,
"y": 11434377216
},
{
"x": 1477379160,
"y": 11432947712
},
{
"x": 1477379220,
"y": 11428257792
},
{
"x": 1477379280,
"y": 11426136064
}
],
"metricName": "memory/usage",
"aggregation": "sum"
}
]
} |
@bryk For some more information, running
whilst the dash report 4.81Gb usage. |
Dashboard UI includes cached MEM in the nodes view, that may be the cause. |
Investigating this.... |
I think I've found the issue. We've been using slightly different metrics for the sparklines and large graphs. Can I build a test image and pass you for verification that the fix actually works? |
I pushed an image for you to verify. Take a look at: bryk/kubernetes-dashboard-amd64:canary |
The issue was with different metrics downloaded in the list and in the graphs. The other problem was that "current usage" was off by 15 mins. Re kubernetes#1346
Shall take a look at this, and report back. Thanks! |
The issue was with different metrics downloaded in the list and in the graphs. The other problem was that "current usage" was off by 15 mins. Re #1346
I think this should be fine now. Please reopen if the problem still persists. |
Issue details
Environment
Default GKE cluster installation.
Steps to reproduce
View
Workloads >> Pods
then sum the total memory usage and compare that to the graphs. On our system there is a large disparity.Expected result
Memory usage for pods should sum to the total in the graphs?
Comments
See attached screen grab

The text was updated successfully, but these errors were encountered: