Fluentd pods memory usage increasing steadily.

[dewsingh]

What is a problem?

I have a K8s environment which uses fluentd 1.14.6.
Fluentd was configured with http and prometheus plugins.
when I checked the memory usage of fluentd over a period of 5 days, it was observed that the average memory usage of fluentd was gradually increasing. There was no active traffic on the system.
The graph for the same is below:

When the fluentd handles traffic then the memory utilization keeps on increasing and ultimately there is a silent OOM kill of the ruby process determined only by the process timestamp.

Describe the configuration of Fluentd

<source>
@type http
port 8888
bind "0.0.0.0"
</source>
<source>
@type prometheus
bind 0.0.0.0
port 24231
metrics_path /metrics
</source>

<source>
@type prometheus_monitor
<labels>
host ${hostname}
</labels>
</source>

<source>
@type prometheus_output_monitor
<labels>
host ${hostname}
</labels>
</source>

<source>
@type prometheus_tail_monitor
<labels>
host ${hostname}
</labels>
</source>

Describe the logs of Fluentd

No logs

Environment

- Fluentd version: 1.14.6
- TD Agent version: 4.3.1
- Fluent Package version:
- Docker image (tag):
- Operating system:
- Kernel version: