Possible memory leak in api .load() when calling multiple times with interval or timeout #429

rshingleton · 2018-05-29T20:00:17Z

I have a vue application built on a model where I have a websocket connection that received periodic updates from several monitoring servers. When app gets an update, I do a call to chart.load(chartData) where the chartData contain is a populated object {xs: {},columns: [],colors: {}}. I noticed after migrating from another graphing package to Billboard that if I leave my page open that it will die after a very short interval, like within an hour.

In order to eliminate all other potential memory leaks, I created a simple page where I poll a single json endpoint and get an array of data with approximately 120 elements with a timestamp and datapoint. I then load this into a timeseries chart. I set a refresh interval with setTimeout at 20 seconds. After about 120 seconds the heap and listeners begin rising.

NOTE: There is also an open issue for c3.js for what appears to be the same thing:
https://github.com/c3js/c3/issues/1961

Chrome profiler with loading:

Chrome profiler without loading:

Sample Code:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
    <link rel="stylesheet" type="text/css" href="https://naver.github.io/billboard.js/release/latest/dist/billboard.min.css">

    <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.3/jquery.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/axios/0.18.0/axios.min.js"></script>
    <script src="https://d3js.org/d3.v4.min.js"></script>
    <script src="https://naver.github.io/billboard.js/release/latest/dist/billboard.min.js"></script>

</head>
<body>
<div id="chart"></div>
<script>
    var chart;
    var timer;
    $(function () {


        getData();

    })

    function getData() {
        let url = '/status/frontpageIncidentsPerMinute';
        return axios.get(url).then((response) => {
            console.log({data: response.data});
            if (!chart) {
                chart = makeChart(prepChartData(response.data));
            } else {
                var data = prepChartData(response.data);
                chart.load(data);
            }
        }, (err) => {
            console.log('Status API: updateApplicationStatus', err)
        }).then(() => {
            
            if (timer)
                clearTimeout(timer)

            timer = setTimeout(function () {
                console.log('Updating');
                getData();
            }, 20000);
            
        });
    }

    function makeChart(chartData) {
        return bb.generate({
            bindto: '#chart',
            data: chartData,
            zoom: {
                enabled: false
            },
            axis: {
                x: {
                    type: 'timeseries',
                    tick: {
                        format: "%m-%d %H:%M:%S" // https://github.com/mbostock/d3/wiki/Time-Formatting#wiki-format
                    }
                }
            }
        });
    }

    function prepChartData(res) {

        let chartData = {
            xs: {},
            columns: [],
            colors: {}
        };

        let seriesData = {};

        chartData.xs['local'] = 'local_x';
        chartData.colors['local'] = '#ff9999';
        let cols = ['local_x'];
        let data = ['local'];
        res.forEach((sd) => {
            let x = sd[0];
            let y = sd[1];
            cols.push(x);
            data.push(y);
        });
        chartData.columns.push(cols, data);

        return chartData;
    }
</script>

</body>

</html>

Sample Data:

{
  "data": [
    [
      1527616620000,
      15
    ],
    [
      1527616680000,
      67
    ],
    [
      1527616740000,
      19
    ],
    [
      1527616800000,
      12
    ],
    [
      1527616860000,
      58
    ],
    [
      1527616920000,
      70
    ],
    [
      1527616980000,
      20
    ],
    [
      1527617040000,
      67
    ],
    [
      1527617100000,
      25
    ],
    [
      1527617160000,
      68
    ],
    [
      1527617220000,
      31
    ],
    [
      1527617280000,
      67
    ],
    [
      1527617340000,
      19
    ],
    [
      1527617400000,
      57
    ],
    [
      1527617460000,
      38
    ],
    [
      1527617520000,
      52
    ],
    [
      1527617580000,
      27
    ],
    [
      1527617640000,
      86
    ],
    [
      1527617700000,
      11
    ],
    [
      1527617760000,
      78
    ],
    [
      1527617820000,
      22
    ],
    [
      1527617880000,
      71
    ],
    [
      1527617940000,
      25
    ],
    [
      1527618000000,
      53
    ],
    [
      1527618060000,
      33
    ],
    [
      1527618120000,
      50
    ],
    [
      1527618180000,
      23
    ],
    [
      1527618240000,
      75
    ],
    [
      1527618300000,
      12
    ],
    [
      1527618360000,
      77
    ],
    [
      1527618420000,
      22
    ],
    [
      1527618480000,
      67
    ],
    [
      1527618540000,
      18
    ],
    [
      1527618600000,
      16
    ],
    [
      1527618660000,
      57
    ],
    [
      1527618720000,
      62
    ],
    [
      1527618780000,
      17
    ],
    [
      1527618840000,
      74
    ],
    [
      1527618900000,
      25
    ],
    [
      1527618960000,
      78
    ],
    [
      1527619020000,
      18
    ],
    [
      1527619080000,
      60
    ],
    [
      1527619140000,
      18
    ],
    [
      1527619200000,
      58
    ],
    [
      1527619260000,
      39
    ],
    [
      1527619320000,
      62
    ],
    [
      1527619380000,
      31
    ],
    [
      1527619440000,
      75
    ],
    [
      1527619500000,
      9
    ],
    [
      1527619560000,
      70
    ],
    [
      1527619620000,
      9
    ],
    [
      1527619680000,
      63
    ],
    [
      1527619740000,
      15
    ],
    [
      1527619800000,
      60
    ],
    [
      1527619860000,
      43
    ],
    [
      1527619920000,
      66
    ],
    [
      1527619980000,
      24
    ],
    [
      1527620040000,
      66
    ],
    [
      1527620100000,
      6
    ],
    [
      1527620160000,
      71
    ],
    [
      1527620220000,
      24
    ],
    [
      1527620280000,
      68
    ],
    [
      1527620340000,
      22
    ],
    [
      1527620400000,
      16
    ],
    [
      1527620460000,
      58
    ],
    [
      1527620520000,
      65
    ],
    [
      1527620580000,
      18
    ],
    [
      1527620640000,
      75
    ],
    [
      1527620700000,
      31
    ],
    [
      1527620760000,
      81
    ],
    [
      1527620820000,
      26
    ],
    [
      1527620880000,
      63
    ],
    [
      1527620940000,
      12
    ],
    [
      1527621000000,
      73
    ],
    [
      1527621060000,
      31
    ],
    [
      1527621120000,
      55
    ],
    [
      1527621180000,
      37
    ],
    [
      1527621240000,
      82
    ],
    [
      1527621300000,
      15
    ],
    [
      1527621360000,
      92
    ],
    [
      1527621420000,
      37
    ],
    [
      1527621480000,
      81
    ],
    [
      1527621540000,
      17
    ],
    [
      1527621600000,
      61
    ],
    [
      1527621660000,
      29
    ],
    [
      1527621720000,
      61
    ],
    [
      1527621780000,
      36
    ],
    [
      1527621840000,
      76
    ],
    [
      1527621900000,
      30
    ],
    [
      1527621960000,
      93
    ],
    [
      1527622020000,
      16
    ],
    [
      1527622080000,
      77
    ],
    [
      1527622140000,
      29
    ],
    [
      1527622200000,
      28
    ],
    [
      1527622260000,
      67
    ],
    [
      1527622320000,
      92
    ],
    [
      1527622380000,
      27
    ],
    [
      1527622440000,
      71
    ],
    [
      1527622500000,
      28
    ],
    [
      1527622560000,
      95
    ],
    [
      1527622620000,
      36
    ],
    [
      1527622680000,
      77
    ],
    [
      1527622740000,
      19
    ],
    [
      1527622800000,
      57
    ],
    [
      1527622860000,
      42
    ],
    [
      1527622920000,
      84
    ],
    [
      1527622980000,
      36
    ],
    [
      1527623040000,
      79
    ],
    [
      1527623100000,
      19
    ],
    [
      1527623160000,
      103
    ],
    [
      1527623220000,
      15
    ],
    [
      1527623280000,
      72
    ],
    [
      1527623340000,
      27
    ],
    [
      1527623400000,
      70
    ],
    [
      1527623460000,
      41
    ],
    [
      1527623520000,
      75
    ],
    [
      1527623580000,
      22
    ],
    [
      1527623640000,
      67
    ],
    [
      1527623700000,
      7
    ],
    [
      1527623760000,
      44
    ]
  ]
}

The text was updated successfully, but these errors were encountered:

netil · 2018-05-30T08:13:51Z

@rshingleton thanks for reporting.

There's already a similar issue #347. If you read the comments, I made an example for the test, but I couldn't find any weird behavior at that time.

I'll try to reproduce based on your example. If you find anything related let me know or plz leave some comments.

rshingleton · 2018-05-30T20:07:01Z

I think the issue may be stemming from the generateWait() method in billboard.js/src/internals/ChartInternal.js.

I believe the current implementation is creating a lot of timers and not clearing them properly. In fact, I think it just keeps creating timers because the function is broken. In all the tests I've run, this test:

if (done === transitionsToWait.length) {
  clearTimeout(timer);
  callback && callback();
} else {
  timer = setTimeout(loop, 20);
}

Always seems to fail. The done variable always seems to be equal to 6 when using a timeseries graph and loading data whereas the transitionsToWait.length is equal to (120 * number of series) + 8 being loaded.

I've switched from a timer based recursion to a promise and was running tests and so far it seems to be better, I need to let it run longer to be sure, but so far this looks better. I'm just not sure why the number of done is always 6 as yet, I certainly haven't dug that far into the code. I'm hesitant about the while loop, here, but I'm sure some better handling can be implemented. This is a quick test.

generateWait() {
	let transitionsToWait = [];
	const f = function (transition, callback) {
		const sleep = time => new Promise(resolve => setTimeout(resolve, time));

		function loop() {
			const p = new Promise((resolve, reject) => {
				const check = () => {
					let count = 0;

					transitionsToWait.forEach(t => {
						if (t.empty()) {
							count += 1;
							return;
						}

						try {
							t.transition();
						} catch (e) {
							count += 1;
						}
					});
					return count;
				};

				let done = check();

				while (done !== 6) {
					done = check();
					sleep(20);
				}

				resolve();
			});

			p.then(() => {
				callback && callback();
			});
		}

		loop();
	};

	f.add = function (transition) {
		// console.log(transition)
		if (Array.isArray(transition)) {
			transitionsToWait = [...transitionsToWait, ...transition];
		} else {
			transitionsToWait.push(transition);
		}
	};

	return f;
}

New test page structure

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>CHART LOAD TEST</title>
    <link rel="stylesheet" type="text/css" href="https://naver.github.io/billboard.js/release/latest/dist/billboard.min.css">
    <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.3/jquery.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/axios/0.18.0/axios.min.js"></script>
    <script src="https://d3js.org/d3.v4.min.js"></script>
    <script src="/dist/billboard.js"></script>

</head>
<body>
<div id="chart"></div>
<script>
    var chart;
    var timer;
    const millisecondsPerMinute = 60000;
    let chartData = {
        xs: {},
        columns: [],
        colors: {}
    };

    $(function () {
        getData();
    })


    function makeChart(chartData) {
        return bb.generate({
            bindto: '#chart',
            data: chartData,
            zoom: {
                enabled: false
            },
            axis: {
                x: {
                    type: 'timeseries',
                    tick: {
                        format: "%m-%d %H:%M:%S" // https://github.com/mbostock/d3/wiki/Time-Formatting#wiki-format
                    }
                }
            }
        });
    }

    function prepChartData(res, name, color) {
        chartData.xs[name] = `${name}_x`;
        chartData.colors[name] = color;
        let cols = [`${name}_x`];
        let data = [name];
        res.forEach((sd) => {
            let x = sd[0];
            let y = sd[1];
            cols.push(x);
            data.push(y);
        });
        chartData.columns.push(cols, data);

        return chartData;
    }


    function getData() {
        prepChartData(randomizeValues(sample), 'sample1', '#ff0000');
        prepChartData(randomizeValues(sample), 'sample2', '#80ff00');
        prepChartData(randomizeValues(sample), 'sample3', '#867979');
        prepChartData(randomizeValues(sample), 'sample4', '#4000ff');
        prepChartData(randomizeValues(sample), 'sample5', '#ff00dc');
        prepChartData(randomizeValues(sample), 'sample6', '#000000');

        if (chart)
            chart.load(chartData);
        else
            chart = makeChart(chartData);


        if (timer)
            clearTimeout(timer)

        timer = setTimeout(function () {
            console.log('Updating');
            chartData = {
                xs: {},
                columns: [],
                colors: {}
            };
            sample.forEach((sd) => {
                sd[0] += millisecondsPerMinute;
            });
            getData();
        }, 5000);
    }

    function randomizeValues(data, withTime) {
        var d = JSON.parse(JSON.stringify(data));
        d.forEach((sd) => {
            sd[0] += millisecondsPerMinute;
            let variance = sd[1] / 2;
            sd[1] = (Math.random() * ((sd[1] + variance) - (sd[1] - variance) + 1) + (sd[1] - variance) + 1);
        });
        return d;
    }


    var sample =
        [
            [
                1527616620000,
                15
            ],
            [
                1527616680000,
                67
            ],
            [
                1527616740000,
                19
            ],
            [
                1527616800000,
                12
            ],
            [
                1527616860000,
                58
            ],
            [
                1527616920000,
                70
            ],
            [
                1527616980000,
                20
            ],
            [
                1527617040000,
                67
            ],
            [
                1527617100000,
                25
            ],
            [
                1527617160000,
                68
            ],
            [
                1527617220000,
                31
            ],
            [
                1527617280000,
                67
            ],
            [
                1527617340000,
                19
            ],
            [
                1527617400000,
                57
            ],
            [
                1527617460000,
                38
            ],
            [
                1527617520000,
                52
            ],
            [
                1527617580000,
                27
            ],
            [
                1527617640000,
                86
            ],
            [
                1527617700000,
                11
            ],
            [
                1527617760000,
                78
            ],
            [
                1527617820000,
                22
            ],
            [
                1527617880000,
                71
            ],
            [
                1527617940000,
                25
            ],
            [
                1527618000000,
                53
            ],
            [
                1527618060000,
                33
            ],
            [
                1527618120000,
                50
            ],
            [
                1527618180000,
                23
            ],
            [
                1527618240000,
                75
            ],
            [
                1527618300000,
                12
            ],
            [
                1527618360000,
                77
            ],
            [
                1527618420000,
                22
            ],
            [
                1527618480000,
                67
            ],
            [
                1527618540000,
                18
            ],
            [
                1527618600000,
                16
            ],
            [
                1527618660000,
                57
            ],
            [
                1527618720000,
                62
            ],
            [
                1527618780000,
                17
            ],
            [
                1527618840000,
                74
            ],
            [
                1527618900000,
                25
            ],
            [
                1527618960000,
                78
            ],
            [
                1527619020000,
                18
            ],
            [
                1527619080000,
                60
            ],
            [
                1527619140000,
                18
            ],
            [
                1527619200000,
                58
            ],
            [
                1527619260000,
                39
            ],
            [
                1527619320000,
                62
            ],
            [
                1527619380000,
                31
            ],
            [
                1527619440000,
                75
            ],
            [
                1527619500000,
                9
            ],
            [
                1527619560000,
                70
            ],
            [
                1527619620000,
                9
            ],
            [
                1527619680000,
                63
            ],
            [
                1527619740000,
                15
            ],
            [
                1527619800000,
                60
            ],
            [
                1527619860000,
                43
            ],
            [
                1527619920000,
                66
            ],
            [
                1527619980000,
                24
            ],
            [
                1527620040000,
                66
            ],
            [
                1527620100000,
                6
            ],
            [
                1527620160000,
                71
            ],
            [
                1527620220000,
                24
            ],
            [
                1527620280000,
                68
            ],
            [
                1527620340000,
                22
            ],
            [
                1527620400000,
                16
            ],
            [
                1527620460000,
                58
            ],
            [
                1527620520000,
                65
            ],
            [
                1527620580000,
                18
            ],
            [
                1527620640000,
                75
            ],
            [
                1527620700000,
                31
            ],
            [
                1527620760000,
                81
            ],
            [
                1527620820000,
                26
            ],
            [
                1527620880000,
                63
            ],
            [
                1527620940000,
                12
            ],
            [
                1527621000000,
                73
            ],
            [
                1527621060000,
                31
            ],
            [
                1527621120000,
                55
            ],
            [
                1527621180000,
                37
            ],
            [
                1527621240000,
                82
            ],
            [
                1527621300000,
                15
            ],
            [
                1527621360000,
                92
            ],
            [
                1527621420000,
                37
            ],
            [
                1527621480000,
                81
            ],
            [
                1527621540000,
                17
            ],
            [
                1527621600000,
                61
            ],
            [
                1527621660000,
                29
            ],
            [
                1527621720000,
                61
            ],
            [
                1527621780000,
                36
            ],
            [
                1527621840000,
                76
            ],
            [
                1527621900000,
                30
            ],
            [
                1527621960000,
                93
            ],
            [
                1527622020000,
                16
            ],
            [
                1527622080000,
                77
            ],
            [
                1527622140000,
                29
            ],
            [
                1527622200000,
                28
            ],
            [
                1527622260000,
                67
            ],
            [
                1527622320000,
                92
            ],
            [
                1527622380000,
                27
            ],
            [
                1527622440000,
                71
            ],
            [
                1527622500000,
                28
            ],
            [
                1527622560000,
                95
            ],
            [
                1527622620000,
                36
            ],
            [
                1527622680000,
                77
            ],
            [
                1527622740000,
                19
            ],
            [
                1527622800000,
                57
            ],
            [
                1527622860000,
                42
            ],
            [
                1527622920000,
                84
            ],
            [
                1527622980000,
                36
            ],
            [
                1527623040000,
                79
            ],
            [
                1527623100000,
                19
            ],
            [
                1527623160000,
                103
            ],
            [
                1527623220000,
                15
            ],
            [
                1527623280000,
                72
            ],
            [
                1527623340000,
                27
            ],
            [
                1527623400000,
                70
            ],
            [
                1527623460000,
                41
            ],
            [
                1527623520000,
                75
            ],
            [
                1527623580000,
                22
            ],
            [
                1527623640000,
                67
            ],
            [
                1527623700000,
                7
            ],
            [
                1527623760000,
                44
            ]
        ];
</script>

</body>

</html>

netil · 2018-05-31T06:59:02Z

I monitored for hours based on your example, and the JS heap size didn't showed accumulated increase. It maintained as 110 ~ 130MB.
The only changes I did was to change the interval of getData() to 1000ms to get quick updates.

Anyway, I'll try look on .generateWait(), if there's some potential pitfalls.

- Avoid the call of generateWait() when the condition aren't met. - Change 'onrendered' option's default value to undefined to not fulfill condition for generateWait(). - Timer interval to check transition end, increased from 20 to 50. Ref #429 Close #430

netil · 2018-05-31T10:14:37Z

@rshingleton, I did some refactoring to avoid unnecessary transition wait.

Could you make a test with the latest build? And plz let me know how it was.

https://github.com/naver/billboard.js/tree/nightly/dist

If there's no necessity on using onrendered option, it will not generate the transition wait.
I'm seeing some improvement from this changes.

rshingleton · 2018-05-31T17:17:30Z

This seems to be doing much better, I need to run some additional tests against it.

TheWitness · 2021-03-31T23:10:38Z

@netil, Why is this issue still open?

netil · 2021-04-01T00:09:29Z

@TheWitness, is because was waiting the confirmation after the test.

I need to run some additional tests against it.

I guess is okay to close now, where the issue not been active for long time.

TheWitness · 2021-04-01T00:12:45Z

Just seemed a long time. We just swapped over from c3 and I was fighting a memory leak and noted the age of the ticket without an update.

netil · 2021-04-01T01:10:27Z

@TheWitness, if you find any issue, plz open new issue.

netil added the need-investigate label May 30, 2018

netil mentioned this issue May 31, 2018

refactor(ChartInternal): Improve redraw #430

Merged

netil closed this as completed Apr 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Possible memory leak in api .load() when calling multiple times with interval or timeout #429

Possible memory leak in api .load() when calling multiple times with interval or timeout #429

rshingleton commented May 29, 2018

netil commented May 30, 2018

rshingleton commented May 30, 2018 •

edited

Loading

netil commented May 31, 2018

netil commented May 31, 2018

rshingleton commented May 31, 2018

TheWitness commented Mar 31, 2021

netil commented Apr 1, 2021

TheWitness commented Apr 1, 2021

netil commented Apr 1, 2021

Possible memory leak in api .load() when calling multiple times with interval or timeout #429

Possible memory leak in api .load() when calling multiple times with interval or timeout #429

Comments

rshingleton commented May 29, 2018

netil commented May 30, 2018

rshingleton commented May 30, 2018 • edited Loading

netil commented May 31, 2018

netil commented May 31, 2018

rshingleton commented May 31, 2018

TheWitness commented Mar 31, 2021

netil commented Apr 1, 2021

TheWitness commented Apr 1, 2021

netil commented Apr 1, 2021

rshingleton commented May 30, 2018 •

edited

Loading