fluent-bit getting unhealthy because of stdout output?

[reisman234]

Hello,
I'm currently getting familiar with fluent-bit, by deploing it into my local minikube installation and log messages from a dummy container. To deploy fluent-bit, I use helm and adapt the values file accordingly.

Until now, I only changed the outputs to stdout and the inputs to only my specific container.

After the deployment, everything seems running normally. My dummy container is running and producing logs, and fluent-bit is collecting the logs and print it to stdout. But after one minute the fluent-bit pod is getting into the unready state because the /api/v1/health endpoint is returning error all the time. From the /api/v1/metrics endpoint, I get the following information

{
  "input": {
    "tail.0": {
      "records": 985,
      "bytes": 109233,
      "files_opened": 1,
      "files_closed": 0,
      "files_rotated": 0
    }
  },
  "filter": {
    "kubernetes.0": {
      "drop_records": 0,
      "add_records": 0,
      "records": 985,
      "bytes": 109233
    }
  },
  "output": {
    "stdout.0": {
      "proc_records": 0,
      "proc_bytes": 0,
      "errors": 984,
      "retries": 0,
      "retries_failed": 0,
      "dropped_records": 984,
      "retried_records": 0
    }
  }
}

Which shows only errors for the stdout output, but I don't know why this happens.

For completeness, this is the data in the fluent-bit configmap

data:
  custom_parsers.conf: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Log_Level info
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On
        Flush 1
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf

    [INPUT]
        Name tail
        Path /var/log/containers/*_example_*.log
        Exclude_Path /var/log/containers/*_example_fluent-bit*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [OUTPUT]
        Name stdout
        Match kube.*