Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect values for memory usage being displayed #1346

Closed
andrewmclagan opened this issue Oct 19, 2016 · 11 comments
Closed

Incorrect values for memory usage being displayed #1346

andrewmclagan opened this issue Oct 19, 2016 · 11 comments
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@andrewmclagan
Copy link

Issue details

Environment

Default GKE cluster installation.

Dashboard version: 0.1.4
Kubernetes version: 1.4.3 
Operating system: Default GKE cluster installation.
Node.js version: Default GKE cluster installation.
Go version: Default GKE cluster installation.
Steps to reproduce

View Workloads >> Pods then sum the total memory usage and compare that to the graphs. On our system there is a large disparity.

Expected result

Memory usage for pods should sum to the total in the graphs?

Comments

See attached screen grab
screen shot 2016-10-19 at 3 52 39 pm

@floreks floreks added kind/bug Categorizes issue or PR as related to a bug. priority/P0 and removed priority/P0 labels Oct 20, 2016
@bryk
Copy link
Contributor

bryk commented Oct 21, 2016

Is this something that always happens always, or just sometimes?

Can you paste the output of http://<master-IP>/ui/api/v1/pod/default? (you can obfuscate everything that is not about metrics).

@andrewmclagan
Copy link
Author

Not sure if im reading the endpoint correctly, although that redirects to:

/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/workload?namespace=default

Being the kubernetes bash not an API endpoint. Is that correct? If so i can dump that info

@bryk
Copy link
Contributor

bryk commented Oct 24, 2016

The full endpoint is: /api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/api/v1/pod/default. Use it for the dump.

@andrewmclagan
Copy link
Author

{
  "listMeta": {
   "totalItems": 11
  },
  "pods": [
   {
    "objectMeta": {
     "name": "es-client-2600856264-0bubb",
     "namespace": "default",
     "labels": {
      "name": "elasticsearch",
      "pod-template-hash": "2600856264",
      "role": "client"
     },
     "annotations": {
      "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"es-client-2600856264\",\"uid\":\"27094f41-9a7b-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"633398\"}}\n",
      "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container es-client"
     },
     "creationTimestamp": "2016-10-25T06:20:40Z"
    },
    "typeMeta": {
     "kind": "pod"
    },
    "podPhase": "Running",
    "podIP": "10.92.0.30",
    "restartCount": 0,
    "metrics": {
     "cpuUsage": 3,
     "memoryUsage": 1843240960,
     "cpuUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 13
      }
     ],
     "memoryUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 1843240960
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 1843240960
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 1843077120
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 1843077120
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 1843077120
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 1843081216
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 1843081216
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 1842909184
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 1842909184
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 1842909184
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 1844269056
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 1844187136
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 1844187136
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 1844244480
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 1847726080
      }
     ]
    }
   },
   {
    "objectMeta": {
     "name": "es-client-2600856264-3nya8",
     "namespace": "default",
     "labels": {
      "name": "elasticsearch",
      "pod-template-hash": "2600856264",
      "role": "client"
     },
     "annotations": {
      "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"es-client-2600856264\",\"uid\":\"27094f41-9a7b-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"633650\"}}\n",
      "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container es-client"
     },
     "creationTimestamp": "2016-10-25T06:23:14Z"
    },
    "typeMeta": {
     "kind": "pod"
    },
    "podPhase": "Running",
    "podIP": "10.92.1.24",
    "restartCount": 0,
    "metrics": {
     "cpuUsage": 53,
     "memoryUsage": 1851748352,
     "cpuUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 53
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 16
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 7
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 6
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 6
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 7
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 9
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 6
      }
     ],
     "memoryUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 1851748352
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 1855627264
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 1855655936
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 1855541248
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 1855733760
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 1855995904
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 1856000000
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 1856000000
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 1856094208
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 1856094208
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 1856266240
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 1856266240
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 1856266240
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 1852932096
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 1853067264
      }
     ]
    }
   },
   {
    "objectMeta": {
     "name": "es-data-3827280009-8cmer",
     "namespace": "default",
     "labels": {
      "name": "elasticsearch",
      "pod-template-hash": "3827280009",
      "role": "data"
     },
     "annotations": {
      "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"es-data-3827280009\",\"uid\":\"3a53ecc0-9a7b-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"633665\"}}\n",
      "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container es-data"
     },
     "creationTimestamp": "2016-10-25T06:23:19Z"
    },
    "typeMeta": {
     "kind": "pod"
    },
    "podPhase": "Running",
    "podIP": "10.92.1.25",
    "restartCount": 0,
    "metrics": {
     "cpuUsage": 5,
     "memoryUsage": 1854390272,
     "cpuUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 20
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 6
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 8
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 4
      }
     ],
     "memoryUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 1854390272
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 1855029248
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 1855033344
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 1855033344
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 1855033344
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 1855033344
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 1855078400
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 1855086592
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 1855086592
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 1855086592
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 1853890560
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 1853894656
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 1853915136
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 1853816832
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 1853837312
      }
     ]
    }
   },
   {
    "objectMeta": {
     "name": "es-data-3827280009-rynuu",
     "namespace": "default",
     "labels": {
      "name": "elasticsearch",
      "pod-template-hash": "3827280009",
      "role": "data"
     },
     "annotations": {
      "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"es-data-3827280009\",\"uid\":\"3a53ecc0-9a7b-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"633459\"}}\n",
      "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container es-data"
     },
     "creationTimestamp": "2016-10-25T06:21:13Z"
    },
    "typeMeta": {
     "kind": "pod"
    },
    "podPhase": "Running",
    "podIP": "10.92.0.31",
    "restartCount": 0,
    "metrics": {
     "cpuUsage": 93,
     "memoryUsage": 1928818688,
     "cpuUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 93
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 7
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 3
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 7
      }
     ],
     "memoryUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 1928818688
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 1928843264
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 1928871936
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 1928884224
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 1928884224
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 1929478144
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 1909510144
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 1909604352
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 1909776384
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 1912520704
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 1911529472
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 1911537664
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 1911386112
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 1911287808
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 1913962496
      }
     ]
    }
   },
   {
    "objectMeta": {
     "name": "es-master-709150228-sqroo",
     "namespace": "default",
     "labels": {
      "name": "elasticsearch",
      "pod-template-hash": "709150228",
      "role": "master"
     },
     "annotations": {
      "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"es-master-709150228\",\"uid\":\"6ed710c2-9a7a-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"632952\"}}\n",
      "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container es-master"
     },
     "creationTimestamp": "2016-10-25T06:15:31Z"
    },
    "typeMeta": {
     "kind": "pod"
    },
    "podPhase": "Running",
    "podIP": "10.92.0.29",
    "restartCount": 0,
    "metrics": {
     "cpuUsage": 4,
     "memoryUsage": 1832398848,
     "cpuUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 6
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 4
      }
     ],
     "memoryUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 1832398848
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 1832398848
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 1832398848
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 1832402944
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 1832407040
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 1832407040
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 1832407040
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 1832972288
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 1832980480
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 1832980480
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 1832984576
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 1832984576
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 1832988672
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 1832988672
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 1832988672
      }
     ]
    }
   },
   {
    "objectMeta": {
     "name": "es-master-709150228-tycra",
     "namespace": "default",
     "labels": {
      "name": "elasticsearch",
      "pod-template-hash": "709150228",
      "role": "master"
     },
     "annotations": {
      "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"es-master-709150228\",\"uid\":\"6ed710c2-9a7a-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"633633\"}}\n",
      "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container es-master"
     },
     "creationTimestamp": "2016-10-25T06:23:07Z"
    },
    "typeMeta": {
     "kind": "pod"
    },
    "podPhase": "Running",
    "podIP": "10.92.1.23",
    "restartCount": 0,
    "metrics": {
     "cpuUsage": 5,
     "memoryUsage": 1852928000,
     "cpuUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 19
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 6
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 4
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 5
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 6
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 5
      }
     ],
     "memoryUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 1852928000
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 1855897600
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 1855897600
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 1855897600
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 1855901696
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 1855905792
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 1855905792
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 1855905792
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 1855905792
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 1855905792
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 1855422464
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 1855422464
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 1855422464
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 1854828544
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 1854337024
      }
     ]
    }
   },
   {
    "objectMeta": {
     "name": "ethical-jobs-api-412768213-asvtr",
     "namespace": "default",
     "labels": {
      "application": "ethical-jobs",
      "name": "api",
      "pod-template-hash": "412768213",
      "tier": "backend"
     },
     "annotations": {
      "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"ethical-jobs-api-412768213\",\"uid\":\"a599952c-9a77-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"631238\"}}\n",
      "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container api; cpu request for container schedule; cpu request for container queue"
     },
     "creationTimestamp": "2016-10-25T05:55:35Z"
    },
    "typeMeta": {
     "kind": "pod"
    },
    "podPhase": "Running",
    "podIP": "10.92.0.28",
    "restartCount": 0,
    "metrics": {
     "cpuUsage": 11,
     "memoryUsage": 91222016,
     "cpuUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 11
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 14
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 15
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 13
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 13
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 11
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 15
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 22
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 14
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 13
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 14
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 8
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 10
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 12
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 10
      }
     ],
     "memoryUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 91222016
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 91217920
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 90988544
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 90996736
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 90984448
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 91226112
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 91213824
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 138809344
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 139001856
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 138899456
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 91238400
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 91000832
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 91000832
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 91009024
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 83542016
      }
     ]
    }
   },
   {
    "objectMeta": {
     "name": "ethical-jobs-api-412768213-pf68h",
     "namespace": "default",
     "labels": {
      "application": "ethical-jobs",
      "name": "api",
      "pod-template-hash": "412768213",
      "tier": "backend"
     },
     "annotations": {
      "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"ethical-jobs-api-412768213\",\"uid\":\"a599952c-9a77-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"631229\"}}\n",
      "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container api; cpu request for container schedule; cpu request for container queue"
     },
     "creationTimestamp": "2016-10-25T05:55:35Z"
    },
    "typeMeta": {
     "kind": "pod"
    },
    "podPhase": "Running",
    "podIP": "10.92.1.22",
    "restartCount": 0,
    "metrics": {
     "cpuUsage": 25,
     "memoryUsage": 79966208,
     "cpuUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 25
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 8
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 24
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 21
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 20
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 16
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 16
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 28
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 20
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 22
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 27
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 16
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 26
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 18
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 16
      }
     ],
     "memoryUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 79966208
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 79962112
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 79966208
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 81285120
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 81281024
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 79962112
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 79966208
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 126132224
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 127561728
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 127676416
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 79966208
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 81268736
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 79790080
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 79728640
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 79732736
      }
     ]
    }
   },
   {
    "objectMeta": {
     "name": "ethical-jobs-employers-1823474103-lnr9d",
     "namespace": "default",
     "labels": {
      "application": "ethical-jobs",
      "name": "employers",
      "pod-template-hash": "1823474103",
      "tier": "web"
     },
     "annotations": {
      "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"ethical-jobs-employers-1823474103\",\"uid\":\"43c05a5e-9a65-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"620117\"}}\n",
      "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container employers"
     },
     "creationTimestamp": "2016-10-25T03:44:00Z"
    },
    "typeMeta": {
     "kind": "pod"
    },
    "podPhase": "Running",
    "podIP": "10.92.0.25",
    "restartCount": 0,
    "metrics": {
     "cpuUsage": 0,
     "memoryUsage": 52666368,
     "cpuUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 0
      }
     ],
     "memoryUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 52666368
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 52809728
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 52895744
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 53100544
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 53338112
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 53350400
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 52711424
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 52834304
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 52936704
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 53063680
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 53186560
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 53329920
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 53440512
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 53542912
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 52916224
      }
     ]
    }
   },
   {
    "objectMeta": {
     "name": "ethical-jobs-employers-1823474103-xd2nx",
     "namespace": "default",
     "labels": {
      "application": "ethical-jobs",
      "name": "employers",
      "pod-template-hash": "1823474103",
      "tier": "web"
     },
     "annotations": {
      "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"ethical-jobs-employers-1823474103\",\"uid\":\"43c05a5e-9a65-11e6-8a15-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"620127\"}}\n",
      "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container employers"
     },
     "creationTimestamp": "2016-10-25T03:44:00Z"
    },
    "typeMeta": {
     "kind": "pod"
    },
    "podPhase": "Running",
    "podIP": "10.92.1.21",
    "restartCount": 0,
    "metrics": {
     "cpuUsage": 0,
     "memoryUsage": 51040256,
     "cpuUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 0
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 0
      }
     ],
     "memoryUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 51040256
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 51146752
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 51269632
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 51417088
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 51544064
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 50868224
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 50991104
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 51118080
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 51281920
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 51388416
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 51511296
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 51658752
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 51724288
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 51052544
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 51200000
      }
     ]
    }
   },
   {
    "objectMeta": {
     "name": "redis-316302323-d8xgn",
     "namespace": "default",
     "labels": {
      "application": "ethical-jobs",
      "name": "redis",
      "pod-template-hash": "316302323",
      "tier": "service"
     },
     "annotations": {
      "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"redis-316302323\",\"uid\":\"9e318153-965d-11e6-9a32-42010af000c6\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"1272\"}}\n"
     },
     "creationTimestamp": "2016-10-20T00:39:11Z"
    },
    "typeMeta": {
     "kind": "pod"
    },
    "podPhase": "Running",
    "podIP": "10.92.1.8",
    "restartCount": 0,
    "metrics": {
     "cpuUsage": 1,
     "memoryUsage": 2826240,
     "cpuUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 1
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 1
      }
     ],
     "memoryUsageHistory": [
      {
       "timestamp": "2016-10-25T06:54:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T06:55:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T06:56:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T06:57:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T06:58:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T06:59:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T07:00:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T07:01:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T07:02:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T07:03:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T07:04:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T07:05:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T07:06:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T07:07:00Z",
       "value": 2826240
      },
      {
       "timestamp": "2016-10-25T07:08:00Z",
       "value": 2826240
      }
     ]
    }
   }
  ],
  "cumulativeMetrics": [
   {
    "dataPoints": [
     {
      "x": 1477378440,
      "y": 200
     },
     {
      "x": 1477378500,
      "y": 88
     },
     {
      "x": 1477378560,
      "y": 69
     },
     {
      "x": 1477378620,
      "y": 61
     },
     {
      "x": 1477378680,
      "y": 60
     },
     {
      "x": 1477378740,
      "y": 57
     },
     {
      "x": 1477378800,
      "y": 54
     },
     {
      "x": 1477378860,
      "y": 76
     },
     {
      "x": 1477378920,
      "y": 57
     },
     {
      "x": 1477378980,
      "y": 62
     },
     {
      "x": 1477379040,
      "y": 70
     },
     {
      "x": 1477379100,
      "y": 52
     },
     {
      "x": 1477379160,
      "y": 62
     },
     {
      "x": 1477379220,
      "y": 67
     },
     {
      "x": 1477379280,
      "y": 66
     }
    ],
    "metricName": "cpu/usage_rate",
    "aggregation": "sum"
   },
   {
    "dataPoints": [
     {
      "x": 1477378440,
      "y": 11441246208
     },
     {
      "x": 1477378500,
      "y": 11448999936
     },
     {
      "x": 1477378560,
      "y": 11448881152
     },
     {
      "x": 1477378620,
      "y": 11450462208
     },
     {
      "x": 1477378680,
      "y": 11451011072
     },
     {
      "x": 1477378740,
      "y": 11450134528
     },
     {
      "x": 1477378800,
      "y": 11429691392
     },
     {
      "x": 1477378860,
      "y": 11524198400
     },
     {
      "x": 1477378920,
      "y": 11526361088
     },
     {
      "x": 1477378980,
      "y": 11529351168
     },
     {
      "x": 1477379040,
      "y": 11433091072
     },
     {
      "x": 1477379100,
      "y": 11434377216
     },
     {
      "x": 1477379160,
      "y": 11432947712
     },
     {
      "x": 1477379220,
      "y": 11428257792
     },
     {
      "x": 1477379280,
      "y": 11426136064
     }
    ],
    "metricName": "memory/usage",
    "aggregation": "sum"
   }
  ]
 }

@andrewmclagan
Copy link
Author

@bryk For some more information, running kubectl top node returns:

NAME                                                  CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%
gke-ethical-jobs-cluster-default-pool-04458a56-kphb   81m          4%        1760Mi          23%
gke-ethical-jobs-cluster-default-pool-04458a56-t5cy   90m          4%        1248Mi          16%

whilst the dash report 4.81Gb usage.

@bryk
Copy link
Contributor

bryk commented Oct 28, 2016

Dashboard UI includes cached MEM in the nodes view, that may be the cause.

@bryk bryk self-assigned this Oct 28, 2016
@bryk
Copy link
Contributor

bryk commented Oct 28, 2016

Investigating this....

@bryk
Copy link
Contributor

bryk commented Oct 28, 2016

I think I've found the issue. We've been using slightly different metrics for the sparklines and large graphs.

Can I build a test image and pass you for verification that the fix actually works?

@bryk
Copy link
Contributor

bryk commented Oct 28, 2016

I pushed an image for you to verify. Take a look at: bryk/kubernetes-dashboard-amd64:canary

bryk added a commit to bryk/dashboard that referenced this issue Oct 28, 2016
The issue was with different metrics downloaded in the list and in the graphs. The other problem was that "current usage" was off by 15 mins.

Re kubernetes#1346
@andrewmclagan
Copy link
Author

Shall take a look at this, and report back. Thanks!

rf232 pushed a commit that referenced this issue Oct 31, 2016
The issue was with different metrics downloaded in the list and in the graphs. The other problem was that "current usage" was off by 15 mins.

Re #1346
@bryk
Copy link
Contributor

bryk commented Nov 3, 2016

I think this should be fine now. Please reopen if the problem still persists.

@bryk bryk closed this as completed Nov 3, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

3 participants