Duplicate logs being dumped in ES intermittently.

[nishant24011998]

What is a problem?

Duplicate logs being dumped in ES intermittently. This is random and not for all the logs.

Describe the configuration of Fluentd

I am using logging operator version 103.1.1+up4.4.0. I witnessed that duplicate logs are being dumped in ES.

Below is my flow config

Name: smnts-semanticsearchbatch-json-applogs
Namespace: jc-pd
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: smnts-semanticsearchbatch-pd
meta.helm.sh/release-namespace: cattle-logging-system
API Version: logging.banzaicloud.io/v1beta1
Kind: Flow
Metadata:
Creation Timestamp: 2024-12-20T05:43:07Z
Generation: 1
Resource Version: 6359244
UID: 25ef9de3-5e5d-40ff-8069-c808f8ee0690
Spec:
Filters:
record_modifier:
Records:
Platform: application-logs
Parser:
Parse:
Patterns:
Expression: /^[[^ ]* [(?[^\]])] [(?[^\]])] [(?[^ ])] [(?[^ ])] [(?[^ ])] [(?[^ ])] [(?[^ ])] [(?[^ ])] [(?[^ ])] [(?[^ ])] [(?[^ ])] [(?[^ ])] (?[^\n](\n^[^\[].|$))/
Format: regexp
Format: json
time_format: %Y-%m-%dT%H:%M:%S.%NZ
Type: multi_format
remove_key_name_field: true
reserve_data: true
reserve_time: true
Grep:
Exclude:
Key: log
Pattern: /.+|\s+/
Key: logType
Pattern: /business_event/
Global Output Refs:
pd-es-output
Match:
Select:
container_names:
smnts-semanticsearchbatch**

Name: pd-es-output
Namespace: cattle-logging-system
Labels:
Annotations:
API Version: logging.banzaicloud.io/v1beta1
Kind: ClusterOutput
Metadata:
Creation Timestamp: 2024-12-20T05:43:54Z
Generation: 2
Resource Version: 27785653
UID: 5c387eb0-e622-4afa-af87-26ab6c883074
Spec:
Elasticsearch:
Buffer:
chunk_limit_size: 50m
flush_interval: 10s
queued_chunks_limit_size: 10240
retry_max_times: 8
retry_timeout: 60s
retry_wait: 5s
Timekey: 1m
timekey_use_utc: true
timekey_wait: 30s
Host: ******************
include_timestamp: true
index_name: ************
Password:
Value From:
Secret Key Ref:
Key: elastic
Name: elastic-user
Port: ******
reload_connections: true
reload_on_failure: true
request_timeout: 120s
Scheme: http
suppress_type_name: true
time_key_format: %Y-%m-%dT%H:%M:%S.%N%z
User: *******

I am facing this issue intermittently.

Kindly help with this problem.

Describe the logs of Fluentd

No response

Environment

Logging Operator Version: 103.1.1+up4.4.0
Fluentd Image: rancher/mirrored-banzaicloud-fluentd:v1.14.6-alpine-5
Operator Image: rancher/mirrored-kube-logging-logging-operator:4.4.0
Fluentbit Image: rancher/mirrored-fluent-fluent-bit:2.2.0