FluentBit Convert message to seperate fields

[marcelvanheemst]

I'm struggling to convert a json message into seperate fields.

The logfile contains the lines below:
{"level":"info","msg":"config storage: postgres","time":"2024-10-02T09:00:57Z"}
{"level":"info","msg":"config static client: Argo Continuous Delivery","time":"2024-10-02T09:00:57Z"}
{"level":"info","msg":"config refresh tokens rotation enabled: true","time":"2024-10-02T09:00:57Z"}
{"level":"info","msg":"listening (http) on 0.0.0.0:8080","time":"2024-10-02T09:00:57Z"}

FluentBit configuration:
[INPUTS]

    - name: tail
      tag: idp.logs
      key: idp
      path: /output/logs/*.log
      db: /output/fluent/fluent.db

[FILTERS]

    - name: parser
      match: idp.*
      parser: json-parser
      key_name: log

    - name: nest
      match: idp.*
      operation: lift
      nested_under: message

[PARSER]

  name                    json-parser
  format                  json
  time_key                time
  time_tormat             %Y-%m-%dT%H:%M:%S%z
  Decode_Field_As   json       message

[OUTPUTS]

    - name:  gelf
      match: idp.*
      host: graylog-inputs.mydomain.nl
      port: 12201
      mode: tcp
      gelf_short_message_key: idp

The graylog server received the message and shows it as below:

I tried regex and nest/lift to convert the json message to seperate fields but without success, I need seperate field so I'm able to filter on severity level using graylog.

Is there a way to convert the json message received from the /output/logs/*.log to seperate fields?

[patrick-stephens]

Simplify first, I think it is being parsed correctly but it may be the gelf output confusing things.

What does it look like with stdout output?

Your parser definition does not look right either:
Decode_Field_As json message
Your input has no such message key:
{"level":"info","msg":"config storage: postgres","time":"2024-10-02T09:00:57Z"}

[patrick-stephens]

OK, maybe try the latest 3.1.9 version to confirm?

That was working for me.

[marcelvanheemst]

Next monday I will upgrade to 3.1.9

[marcelvanheemst]

Fluent Bit v3.1.9
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  __
|  ___| |                | |   | ___ (_) |         |____ |/  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

[2024/10/07 08:23:09] [ info] [fluent bit] version=3.1.9, commit=, pid=1
[2024/10/07 08:23:09] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/10/07 08:23:09] [ info] [cmetrics] version=0.9.6
[2024/10/07 08:23:09] [ info] [ctraces ] version=0.5.6
[2024/10/07 08:23:09] [ info] [input:tail:tail.0] initializing
[2024/10/07 08:23:09] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/10/07 08:23:09] [ info] [input:tail:tail.0] db: delete unmonitored stale inodes from the database: count=0
[2024/10/07 08:23:09] [ info] [sp] stream processor started
[2024/10/07 08:23:09] [ info] [output:stdout:stdout.0] worker #0 started
[2024/10/07 08:23:09] [ info] [input:tail:tail.0] inotify_fs_add(): inode=263196 watch_fd=1 name=/output/logs/dex.log

I still have the same issue

[patrick-stephens]

I'm not sure what to say, using your exact provided example works fine for me...

[patrick-stephens]

How is your config different to mine?
Does my usage work for you?
#9458 (reply in thread)

[marcelvanheemst]

Yesterday I removed the fluentbit config and started with an empty config.
Looks like the [FILTERS] part doesn't work as expected and avoid the json-parser.

I moved the parser reference to the [INPUTS] part and everything is working as expected.