Linear memory increase after update to 5.0.6

[matteosilv]

Bug Report

Describe the bug
After updating from 5.0.5 to 5.0.6 promql query shows a linear memory increase (check screenshot below):

container_memory_working_set_bytes{
  namespace="fluent-bit",
  pod=~"fluent-bit.*",
  container="fluent-bit"
}

To Reproduce

just updating to 5.0.6 caused the issue.

Expected behavior
No memory increase happens.

Screenshots

Your Environment

this is our config (chart version 0.57.5):

  # Resources
  resources:
    limits:
      cpu: 250m
      memory: 250Mi
    requests:
      cpu: 250m
      memory: 250Mi

  # Scheduling
  priorityClassName: system-node-critical
  tolerations:
    - operator: Exists   # tolerates all taints
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet

  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        # Exclude Fargate nodes
        - key: eks.amazonaws.com/compute-type
          operator: NotIn
          values: [fargate]
        # Exclude "core" nodes
        - key: <REDACTED>/node-type
          operator: NotIn
          values: [core]

  # Logging
  logLevel: error

  # Environment
  env:
    APP_NAME_SYSTEMD: systemd
    SUB_SYSTEM_SYSTEMD: kubelet
    HOSTNAME: <from fieldRef: metadata.name>
    ENDPOINT: <from coralogix.endpoint>
    LOG_LEVEL: <from logLevel>

  # Secrets (injected via envFrom)
  envFrom:
    - secretRef: coralogix-keys   # provides PRIVATE_KEY

  # Coralogix
  coralogix:
    endpoint: <REDACTED>   # private ingress endpoint (eu2 region)

  # Pipeline: Inputs
  inputs:
    - name: tail
      tag: kube.*
      path: /var/log/containers/*.log
      exclude: terraform-operator, fluent-bit, tf-runner containers
      multiline.parser: docker, cri
      mem_buf_limit: 25MB
      processors:
        - kubernetes enrichment (merge_log on, annotations/labels off)
        - nest: lift kubernetes metadata
        - modify: remove kubernetes_docker_id
        - nest: re-nest kubernetes_* fields
        - nest: wrap everything under "json"
        - lua: addcrxmetadata (from mounted script.lua)

    - name: systemd
      tag: host.*
      filter: kubelet.service
      read_from_tail: on
      mem_buf_limit: 5MB
      processors:
        - add applicationName, copy subsystemName
        - nest: _HOSTNAME, SYSLOG_IDENTIFIER, _CMDLINE, MESSAGE → json

  # Pipeline: Output
  outputs:
    - name: http
      match: kube.* and host.*
      host: <REDACTED>   # ENDPOINT from secret/configmap
      port: 443
      uri: /logs/v1/singles
      format: json_lines
      tls: on
      auth: Bearer <PRIVATE_KEY from secret>
      compress: gzip
      retry_limit: false
      net.keepalive: off

  # Observability
  serviceMonitor:
    enabled: true
    selector:
      prometheus: prometheus-platform

  # Extra volumes
  extraVolumes:
    - name: crxluascript
      configMap: fluent-bit-http-crxluascript
      mountPath: /fluent-bit/scripts/script.lua

Additional context

[JohnPolansky]

We are also experiencing this issue causing repeated crashes due to out of memory in multiple clusters. We were using 5.0 tag so picking up patch versions. However we had to downgraded back to 5.0.5 to resolve the memory issues.

Ignore the aws-for-fluent-bit name thats a hold over from previous setup, we are using fluent-bit directly now.

[duj4]

We met the similar issue till 5.0.5, now the memory issue is gone after the upgrade. If the issue came back in 5.0.6, we might postpone the upgrade.

[lecaros]

hey @matteosilv @JohnPolansky @duj4
I couldn't reproduce this locally yet.
can any of you share a reproducer config? Make sure to also include the estimated load (record size, rate, format) you're sending through tailing the files.
If you can't provide a repro, could you please try building from the latest source? The team have recently made some changes and could be relevant here.

[duj4]

@lecaros The memory issue was gone after upgrading to 5.0.5 in my case, did not try 5.0.6 yet.

[MichaelEischer]

We've also seen this memory increase even on nodes where the path provided to tail does not match anything.

Our configuration looks like the following:

[SERVICE]
    Daemon Off
    Flush 1
    Log_Level info
    Parsers_File /fluent-bit/etc/parsers.conf
    Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
    HTTP_Server On
    HTTP_Listen 0.0.0.0
    HTTP_Port 2020
    Health_Check On

[INPUT]
    Name tail
    Path /var/log/containers/*_pod-name-container-name-*.log
    Parser cri
    Tag data.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On

[FILTER]
    Name parser
    Match data.*
    Key_Name message
    Parser json
[FILTER]
    name   grep
    match  data.*
    regex  message value
[FILTER]
    Name  modify
    Match data.*
    Add   hostname ${NODE_NAME}

[OUTPUT]
    Name              splunk
    Match             data.*
    Host              <host>
    Port              8088
    retry_limit       no_limits
    TLS               On
    TLS.Verify        On
    tls.verify_hostname on
    Event_Index       <index>
    Event_Sourcetype  source
    Splunk_Token      ${SPLUNK_TOKEN}

Note that using the stdout output seems to avoid the memory usage issues.

[lecaros]

hi @MichaelEischer,
a few questions:

• for the config you have, what the load look like? (size, rate, type/format)
• Do you see retries, errors, backpressure, when this happens?
• Did you compare with an image built from latest source?
• Do you only see this with Splunk as destination?

I'll try to use your config to reproduce.

[duj4]

@lecaros here is the screenshot for the memory consumption during the past 36 hours of Fluent-Bit running as Daemonset in our lab environment, the workload is quite steady:

The 2 memory consumption increase in the middle is from 5.0.6 and once we downgraded back to 5.0.5, it became steady again.

[lecaros]

Thanks. I've used the config from #11846 (comment) but unable to reproduce. The container mem usage is pretty much the same from the start until almost 3 hours later

== 14:11:58 ==
  fluent-bit-66q8p                                   ip-10-0-11-71.ec2.internal              7.0 MiB
  fluent-bit-z5jkl                                   ip-10-0-3-40.ec2.internal               6.4 MiB

...

== 17:01:07 ==
  fluent-bit-66q8p                                   ip-10-0-11-71.ec2.internal             10.4 MiB
  fluent-bit-z5jkl                                   ip-10-0-3-40.ec2.internal               7.1 MiB

If any of you can provide a reproducer, that will accelerate things.

[duj4]

We have several configurations one Fluent-Bit daemonset, below is one of them:

pipeline:
  inputs:
    - name: tail
      tag: ${tag_loki-query-frontend}
      path: ${path_loki-query-frontend}
      refresh_interval: 2
      parser: cri_loki
      skip_empty_lines: on
      read_from_head: false
      db: ${flb_storage}/loki.db
      storage.type: filesystem
      rotate_wait: 30
      threaded: true

      processors:
        logs:
          - name: parser
            parser: parser_loki-query
            key_name: message
            reserve_data: true

          - name: grep
            regex: caller ^metrics\.go

          - name: modify
            remove:
              - logtag
              - stream

          - name: kubernetes
            kube_tag_prefix: ${prefix_loki-query-frontend}
            merge_log: off
            labels: on
            annotations: off
            buffer_size: 64k
            kube_meta_cache_ttl: 60s

    - name: tail
      tag: ${tag_loki-querier}
      path: ${path_loki-querier}
      refresh_interval: 2
      parser: cri_loki
      skip_empty_lines: on
      read_from_head: false
      db: ${flb_storage}/loki.db
      storage.type: filesystem
      rotate_wait: 30

      processors:
        logs:
          - name: parser
            parser: parser_loki-query
            key_name: message
            reserve_data: true

          - name: grep
            regex: caller ^metrics\.go

          - name: modify
            remove:
              - logtag
              - stream

          - name: kubernetes
            kube_tag_prefix: ${prefix_loki-querier}
            merge_log: off
            labels: on
            annotations: off
            buffer_size: 64k
            kube_meta_cache_ttl: 60s

    - name: tail
      tag: ${tag_loki-gateway}
      path: ${path_loki-gateway}
      refresh_interval: 2
      parser: cri_loki
      skip_empty_lines: on
      read_from_head: false
      db: ${flb_storage}/loki.db
      storage.type: filesystem

      processors:
        logs:
          - name: parser
            parser: parser_loki-gateway
            key_name: message
            reserve_data: true

          - name: grep
            regex: msg (^request rejected)

          - name: modify
            remove:
              - logtag
              - stream

          - name: kubernetes
            kube_tag_prefix: ${prefix_loki-gateway}
            merge_log: off
            labels: on
            annotations: off
            buffer_size: 64k
            kube_meta_cache_ttl: 60s

  outputs:
    - name: loki
      match_regex: ^kube\.loki.*
      host: laas-qa.sh.lab.ms.com.cn
      port: 443
      labels: app=$kubernetes['labels']['app.kubernetes.io/name'], component=$kubernetes['labels']['app.kubernetes.io/component'], host=$kubernetes['host'], pod=$kubernetes['pod_name'], namespace=$kubernetes['namespace_name']
      remove_keys: kubernetes
      line_format: json
      tenant_id: ${loki_tenantid}
      tls: on
      tls.verify: on
      tls.debug: 1
      tls.ca_file: /fluent-bit/etc/certs/ca.crt
      tls.crt_file: /fluent-bit/etc/certs/tls.crt
      tls.key_file: /fluent-bit/etc/certs/tls.key
      storage.total_limit_size: 10G
      net.connect_timeout: 310
      net.keepalive_idle_timeout: 30
      net.keepalive_max_recycle: 120
      net.dns.resolver: ASYNC
      retry_limit: no_limits
      workers: 2

And they are sharing the same service config as below:

env:
  flb_storage: /var/log/fluent-bit-storage

service:
  daemon: off
  flush: {{ .Values.flush }}
  log_level: {{ .Values.log_level }}
  http_server: on
  http_listen: 0.0.0.0
  http_port: {{ .Values.metricsPort }}
  health_check: on
  storage.path: ${flb_storage}
  storage.sync: normal
  storage.checksum: on
  storage.max_chunks_up: 128
  storage.delete_irrecoverable_chunks: on
  scheduler.base: 5
  scheduler.cap: 600

includes:
- Tenant A
- Tenant B
- Tenant C

[lecaros]

Hello again.
So, I was able to reproduce the issue and found the guilty. We're leaking through the http_server. Good news is, this is already solved in the master branch.
Can you build your image from the master branch and test?

Leaking test

baseline RSS=10208 kB
running 30 rounds x 20 conns, holding idle for 12s each round
round  conns_opened  RSS_kB  delta_kB
    1            20    10064      -144
    2            40    11120     +1056
    3            60    11936      +816
    4            80    12240      +304
    5           100    11920      -320
    6           120    12272      +352
    7           140    12848      +576
    8           160    13728      +880
    9           180    13488      -240
   10           200    14560     +1072
   11           220    14608       +48
   12           240    14192      -416
   13           260    14960      +768
   14           280    14832      -128
   15           300    15104      +272
   16           320    16000      +896
   17           340    15296      -704
   18           360    16080      +784
   19           380    16208      +128
   20           400    16736      +528
   21           420    17024      +288
   22           440    17280      +256
   23           460    17648      +368
   24           480    17200      -448
   25           500    17712      +512
   26           520    18768     +1056
   27           540    19824     +1056
   28           560    19344      -480
   29           580    19616      +272
   30           600    20688     +1072

baseline=10208 kB  final=20688 kB  delta=+10480 kB  over 600 connections  (17886 bytes/conn)

Non leaking from master

baseline RSS=9136 kB
running 30 rounds x 20 conns, holding idle for 12s each round
round  conns_opened  RSS_kB  delta_kB
    1            20    10528     +1392
    2            40    10544       +16
    3            60    10592       +48
    4            80    10496       -96
    5           100    10576       +80
    6           120    10512       -64
    7           140    10544       +32
    8           160    10544        +0
    9           180    10448       -96
   10           200    10368       -80
   11           220    10480      +112
   12           240    10464       -16
   13           260    10480       +16
   14           280    10480        +0
   15           300    10512       +32
   16           320    10400      -112
   17           340    10480       +80
   18           360    10416       -64
   19           380    10464       +48
   20           400    10464        +0
   21           420    10320      -144
   22           440    10448      +128
   23           460    10464       +16
   24           480    10464        +0
   25           500    10464        +0
   26           520    10464        +0
   27           540    10464        +0
   28           560    10432       -32
   29           580    10320      -112
   30           600    10064      -256

baseline=9136 kB  final=10064 kB  delta=+928 kB  over 600 connections  (1584 bytes/conn)

[JohnPolansky]

Unfortunately I don't have an easy way to use or deploy with master in our setup as it's all automated based on published images. Hopefully someone else can try 🤞