How to parse different log formats in Kubernetes

[alongwill]

Hi,

I'm using the Helm chart to run fluent-bit in our Kubernetes cluster. The configuration used is below:

config:
  service: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File /fluent-bit/etc/parsers.conf
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On
  customParsers: |
    [PARSER]
      Name          containerd
      Format        regex
      Regex         ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
      Time_Key      time
      Time_Format   %Y-%m-%dT%H:%M:%S.%L%z
  inputs: |
    [INPUT]
        Name tail
        Alias kubernetes
        Path /var/log/containers/*.log
        Parser containerd
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On
        Skip_Empty_Lines On
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
  outputs: |
    [OUTPUT]
        name         loki
        match        kube.*
        host         vlogs-vlinsert.monitoring.svc
        uri          /insert/loki/api/v1/push
        port         9428
        Tenant_Id    0
        label_keys   $path,$time,$stream,$source,$hostname,$kubernetes['pod_name'],$kubernetes['container_name'],$kubernetes['namespace_name']
        header       VL-Msg-Field msg
        header       VL-Time-Field time
        header       VL-Stream-Field ,source,kubernetes_namespace_name,kubernetes_container_name
        remove_keys  path,stream,time,kubernetes
        drop_single_key raw
        line_format  key_value
        Auto_Kubernetes_Labels true

Our kubernetes application logs have different formats:

Example: fluent-bit

logtag="F" log="[2025/05/06 08:38:42] [ info] [input:tail:kubernetes] inotify_fs_remove(): inode=67257744 watch_fd=3539"

we want:

• 2025/05/06 08:38:42 to be parsed as the log entry time (_time field)
• info to be set as the level
• input:tail:kubernetes to be set as input
• the rest of the text as the main log (_msg field)

Example: victoria metrics

logtag="F" level="info" ts="2025-05-06T08:53:45Z" logger="controllers.VMAgent" msg="selected PodScrapes" vmagent="map[name:"test" namespace:"monitoring"]" controller="vmagent.crud" namespace="monitoring" vmagent="vmagent-external-services" podscrapes="caddy/caddy" namespace="monitoring" vmagent="test"

we want

• info to be set as level
• ts to be set as _time
• msg to be set as _msg
• all other key=value pairs to be set as labels

Example: node exporter

ts=2025-04-17T19:06:15.488Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9100

we want:

• ts to be set as _time
• level to be set as level
• msg to be set as _msg
• all other key=value pairs to be set as labels

Is there a recommended way to configure parsing for each application in this scenario? Should I be configuring individual inputs and parsers for these?

[patrick-stephens]

This comes up a few times and I really should write up a post on how to do it.

I did something like this a few years ago: https://github.com/couchbase/couchbase-fluent-bit

You can do this fairly easily with some combination of routing and multi-stage parsing.
You always need to first parse the CRI/docker format logs to reconstruct the potentially multi-line output from the kubelet format.
You can then have further filters invoke different parsers: https://docs.fluentbit.io/manual/pipeline/filters/parser
Either by some combination of matching on tag or just using the "if parser does not match it is skipped" rule. A good idea is when your data is parsed it ensures a field is removed/present always to allow for easy detection and matching of failed parsing or future parsers.

https://www.couchbase.com/blog/fluent-bit-tips-tricks-log-forwarding-couchbase/ was something I put together a few years back with some advice too: this is based on 1.9 series at the time.

[patrick-stephens]

Unrelated but do not make up your own containerd parser - you should use the built in ones to cope with all the edge cases and be more performant. Your current parser fails to cope with partial lines for example which the kubelet can decide to split on for any reason it likes.

I would also add a simple filter as well to enforce some of the label keys you are using and/or simplify them:

        label_keys   $path,$time,$stream,$source,$hostname,$kubernetes['pod_name'],$kubernetes['container_name'],$kubernetes['namespace_name']

What happens if any of those are missing, using invalid characters or some other nonsense? The K8S API may reject the request to look up metadata for example.

[alongwill]

Thanks @patrick-stephens

One strange thing: I tried using record_modifier and was able to remove keys like stream and logtag and one application key called ts (which was part of the main log message).

[FILTER]
    Name record_modifier
    Match *
    Remove_key stream
    Remove_key logtag
    Remove_key ts

I don't understand what's happening here 😅 . i.e. if I'm able to remove keys then the log must be structured, so why do all the application log keys end up in the main log message? It's like fluent-bit thinks they are keys, but by the time it gets to Victoria Logs, that information is lost 🤔 Do I need to set the main log message as the msg key somehow?

[patrick-stephens]

Merge Log is doing this I believe: https://docs.fluentbit.io/manual/pipeline/filters/kubernetes