[Q&A] How to define buffer chunk keys in yaml format

[plloppii]

What is a problem?

Hey folks, im trying to define the chuck keys in the new yaml format, but anything i try is unsuccessful,
The syntax that I am trying is

buffer tag, time:

However I get the following warn output after running a dry run with that yaml:

2024-09-19 14:55:04 +0000 [warn]: section <buffer tag, time> is not used in <store>

The interesting thing here is that the yaml gets interpreted as the following, with the chuck keys within both the opening and closing XML tags

        <buffer tag, time>
          @type file
          path /fluentd/s3/
          timekey 1h
          timekey_wait 1m
          timekey_use_utc true
          chunk_limit_size 256m
          queued_chunks_limit_size 5
          flush_thread_count 5
          overflow_action drop_oldest_chunk
        </buffer tag, time>

Describe the configuration of Fluentd

        - match:
            $tag: '**'
            $type: copy
            store:
              - $type: s3
                $id: out_s3
                s3_bucket: test.logs
                s3_region: us-east-1
                path: service
                s3_object_key_format: !fluent/s "%{path}/${tag[0]}/%Y/%m/%d/%{time_slice}_%{index}_%{hostname}_#{worker_id}.%{file_extension}"
                # if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
                # need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
                
                # Buffer chunks are written as 1 chunk per timekey (an hour, in this case) per service
                # As for 2024 we have ~ 250 production service tags
                buffer tag,time:
                  $type: file
                  path: !fluent/s "/fluentd/s3/#{worker_id}"
                  timekey: 1h
                  timekey_wait: 1m
                  timekey_use_utc: true
                  chunk_limit_size: 256m
                  # We have seen issues where staged chunks are never written into queued chunks,
                  # leading to a catastrophic backlog that reaches the file descriptor limit and
                  # locks up the process. This limit defines how many queued chunks there can be
                  # at a time, and defaults to 1. Setting it higher increases the number of chunks
                  # that can come out of the staged state. See https://www.fluentd.org/blog/fluentd-v1.1.3-has-been-released                  
                  queued_chunks_limit_size: 5
                  flush_thread_count: 5
                  # In case of buffer overflow, drop the oldest QUEUED chunk
                  overflow_action: drop_oldest_chunk

Describe the logs of Fluentd

No response

Environment

- Fluentd version: v1.17
- TD Agent version:
- Fluent Package version:
- Docker image (tag): v1.17
- Operating system:
- Kernel version:

[daipom]

You can use $arg for directive args.

• https://docs.fluentd.org/configuration/config-file-yaml#special-yaml-elements

config:
  - source:
      $type: sample
      tag: test
  - match:
      $type: stdout
      $tag: test.**
      buffer:
        $arg: tag, time
        $type: memory
        time_key: 1h
        flush_mode: immediate

[daipom]

It would be better we have some setting examples in the document.

• https://github.com/fluent/fluentd-docs-gitbook

It would be nice if someone sends a PullRequest.