[Q&A] DiskPressure due to 80 GB /home/fluentd/core. files

[Alisher-Nabiev]

What is a problem?

hi everyone,

i got on my eks fluentd that sents log to elastiserach 7.17.

after upgrade my cluster from 1.24 to 1.25 my nodes got DiskPressure due to 80 GB /home/fluentd/core.  files.

when the fluentd pod fills up the space of the node its evicted and the node in DiskPressure stat for 3-4 min after that its gone and fluentd pod runs again.

Describe the configuration of Fluentd

# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /templates/conf/fluent.conf.erb   

@include "#{ENV['FLUENTD_SYSTEMD_CONF'] || 'systemd'}.conf"
@include "#{ENV['FLUENTD_PROMETHEUS_CONF'] || 'prometheus'}.conf"       
@include kubernetes.conf
@include conf.d/*.conf

<filter kubernetes.var.log.containers.**.log>
  @type parser
  key_name log
  reserve_data true
  reserve_time true
  hash_value_field ******-fields
  remove_key_name_field false
  replace_invalid_sequence false
  emit_invalid_record_to_error true
  <parse>
    @type json
  </parse>
</filter>

<filter kubernetes.var.log.containers.****.log kubernetes.var.log.containers.**.log kubernetes.var.log.containers.**.log kubernetes.var.log.containers.**.log>
  @type parser
  key_name log
  reserve_data true
  reserve_time true
  hash_value_field fields
  remove_key_name_field false
  replace_invalid_sequence false
  emit_invalid_record_to_error true
  <parse>
    @type json
  </parse>
</filter>

<label @ERROR>
  <match **>
    @type null
  </match>
</label>

<match kubernetes.var.log.containers.**.log kubernetes.var.log.containers**.log kubernetes.var.log.containers.**.log kubernetes.var.log.containers.**.log>
  @type elasticsearch
  @id out_es
  @log_level debug 
  include_tag_key true
  hosts "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
  port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
  path "#{ENV['FLUENT_ELASTICSEARCH_PATH']}"
  scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
  ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"     
  ssl_version "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
  user "#{ENV['FLUENT_ELASTICSEARCH_USER'] || use_default}"
  password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD'] || use_default}"
  reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'false'}"
  reconnect_on_error "#{ENV['FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR'] || 'true'}"
  reload_on_failure "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE'] || 'true'}"
  log_es_400_reason "#{ENV['FLUENT_ELASTICSEARCH_LOG_ES_400_REASON'] || 'false'}"
  logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
  logstash_dateformat "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_DATEFORMAT'] || '%Y.%m.%d'}"
  logstash_format "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT'] || 'true'}"
  index_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME'] || 'logstash'}"
  target_index_key "#{ENV['FLUENT_ELASTICSEARCH_TARGET_INDEX_KEY'] || use_nil}"
  type_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_TYPE_NAME'] || 'fluentd'}"
  include_timestamp "#{ENV['FLUENT_ELASTICSEARCH_INCLUDE_TIMESTAMP'] || 'false'}"
  template_name "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_NAME'] || use_nil}"
  template_file "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_FILE'] || use_nil}"
  template_overwrite "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_OVERWRITE'] || use_default}"
  sniffer_class_name "#{ENV['FLUENT_SNIFFER_CLASS_NAME'] || 'Fluent::Plugin::ElasticsearchSimpleSniffer'}"
  request_timeout "#{ENV['FLUENT_ELASTICSEARCH_REQUEST_TIMEOUT'] || '5s'}"
  suppress_type_name "#{ENV['FLUENT_ELASTICSEARCH_SUPPRESS_TYPE_NAME'] || 'true'}"
  enable_ilm "#{ENV['FLUENT_ELASTICSEARCH_ENABLE_ILM'] || 'false'}"
  ilm_policy_id "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY_ID'] || use_default}"
  ilm_policy "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY'] || use_default}"
  ilm_policy_overwrite "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY_OVERWRITE'] || 'false'}"
  <buffer>
    flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"
    flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}"
    chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '10M'}"
    queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '64'}"
    retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
    retry_forever true
  </buffer>
</match>

Describe the logs of Fluentd

fluentd.log

Environment

- Fluentd version: fluentd 1.16.3
- Fluent Package version: Debian GNU/Linux 12 (bookworm)
- Docker image (tag): fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
- Kernel version: 5.10.209-198.812.amzn2.x86_64

[anumantha-sw]

We are also facing the same disk pressure issue in the EKS cluster. Attaching relevant logs from the fluentd containers and environment details.

df -lh
Filesystem Size Used Avail Use% Mounted on
overlay 128G 105G 24G 82% /
tmpfs 64M 0 64M 0% /dev
tmpfs 16G 0 16G 0% /sys/fs/cgroup
....
....

core files from the container

302M core.9443
326M core.9454
333M core.9506
317M core.9611
209M core.9634
325M core.9768
158M core.9823
302M core.9853
222M core.9904
pwd
/home/fluent

EKS Version: 1.25
Fluentd Version: v1.12

[Alisher-Nabiev]

@anumantha-sw
I resolved the issue by downgrading the version from 1.16 to 1.15

Docker image (tag): fluent/fluentd-kubernetes-daemonset:v1.15-debian-elasticsearch7

[daipom]

Is it possible to identify the files occupying disk space? Buffer files, etc.

[dcarrith]

We just experienced the worst weekend in the history of IT (or, maybe the second worst because it's "only" the staging environment). But, the team I'm on was ready to skip the Superbowl and make it the highest priority call usually reserved for production disasters. On the weekend I chose to upgrade our EKS v1.28.5 clusters to 1.29 then 1.30, someone else on our team decided it was a good Friday to upgrade Fluentd from v1.14.5-711 to 1.16.5-80 or some shit. Well, the next 48 hours that ensued was the most brutal work experience of my life. It was worse than when I worked for Symantec and had to travel to Houston, Texas to fix a customer's SSIM issues. They sat me down in the middle of the room and locked me in until I fixed the problems. Anyway, yes, there is something very wrong with v1.16.5-80. It was filling up the /var/lib/containerd/io.containerd.snapshotter.overlayfs/snapshots directory with many hundreds of GBs of snapshots for some reason. I went from our normal 137GB volume to a 1TB volume that it filled up in about 10 hours. It wasn't until I happened to try downgrading the thing that the other person on my team installed (fluentd) and then relaunched the cluster (for the third time) with a 2TB volume for my next tests (trying to literally "buy" ourselves some more time to troubleshoot) but then it magically wasn't happening anymore on the relaunched nodes (after doing this process numerous times while fighting IP constraints because the agency I work for doesn't seem to have enough network space to do what they are doing). It was a fucking nightmare. Have y'all tested with EKS v1.29 and v1.30 yet? Please do the world a favor and test more on those versions. The issue takes a while to start happening.

[kenhys]

I guess that it might be already fixed issue.

fluent/fluentd-docker-image#385

If so, it was caused by old version of systemd-journal and fluent-plugin-systemd.
fluent-kubernetes-daemonset v1.17.1-1.1 image or later version will solve issue.