Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fluentd command: add --with-source-only #4661

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

daipom
Copy link
Contributor

@daipom daipom commented Oct 8, 2024

Which issue(s) this PR fixes:
None.
Related to

This feature may be useful on its own, but mainly, this feature will help #4622 feature.

What this PR does / why we need it:
Add --with-source-only option to fluentd command.
(Ref: Similar to existing --without-source option)

It launches Fluentd with Input plugins only.
Here is the specification.

  • Those Input plugins emits the events to SourceOnlyBufferAgent.
  • The events is kept in the buf_file during the source-only-mode.
  • SIGRTMIN(34) cancels the mode.
  • After canceled, the new agent starts to load the buffer.

Docs Changes:
TODO

Release Note:

  • fluentd command: add --with-source-only
  • embedded plugin: add out_buffer plugin, which can be used for buffering and relabeling events.

new plugin: out_buffer

This feature internally uses the new plugin out_buffer.
This plugin can be used for buffering and relabeling events.

Example: in_udp -> out_buffer -> out_stdout

<source>
  @type udp
  tag foo.udp
  @label @buffer
  <parse>
    @type none
  </parse>
</source>

<match foo.**>
  @type stdout
</match>

<label @buffer>
  <match **>
    @type buffer
    @label @ROOT
    <buffer>
      path /path/to/buffer
    </buffer>
  </match>
</label>

TODO

  • Consider race condition
  • Fix and add tests
  • Consider how to stop the loading agent
  • Consider configurablity of SourceOnlyBufferAgent.
  • Consider Windows support

@daipom daipom force-pushed the command-option-add-with-source-only branch 3 times, most recently from 09f05a5 to f55690e Compare October 10, 2024 08:49
It launches Fluentd with Input plugins only.
Here is the specification.

* Those Input plugins emits the events to SourceOnlyBufferAgent.
* The events is kept in the buf_file during the source-only-mode.
* SIGRTMIN(34) cancels the mode.
* After canceled, the new agent starts to load the buffer.

Signed-off-by: Daijiro Fukuda <[email protected]>
@daipom daipom force-pushed the command-option-add-with-source-only branch from f55690e to 7ab49c3 Compare October 10, 2024 14:35
daipom added a commit that referenced this pull request Oct 11, 2024
Add a new feature: Update/Reload without downtime.

1. The supervisor receives SIGUSR2.
2. Spawn a new supervisor.
3. Take over shared sockets.
4. Launch new workers, and stop old processes in parallel.
   * Launch new workers with source-only mode
     * Limit to restart_without_downtime_ready? input plugin
   * Send SIGTERM to the old supervisor after 10s delay from 3.
5. The old supervisor stops and sends SIGRTMIN(34) to the new one.
6. The new workers run fully.

Problem to solve:

Updating Fluentd or reloading a config causes downtime.
Plugins that receive data as a server, such as `in_udp`, `in_tcp`,
and `in_syslog`, cannot receive data during this time.
This means that the data sent by a client is lost during this
time unless the client has a re-sending feature.
This makes updating Fluentd or reloading a config difficult in
some cases.

Note: need these feature

* #4661
* treasure-data/serverengine#146

Signed-off-by: Daijiro Fukuda <[email protected]>
@daipom daipom mentioned this pull request Oct 11, 2024
3 tasks
daipom added a commit that referenced this pull request Oct 11, 2024
Add a new feature: Update/Reload without downtime.

1. The supervisor receives SIGUSR2.
2. Spawn a new supervisor.
3. Take over shared sockets.
4. Launch new workers, and stop old processes in parallel.
   * Launch new workers with source-only mode
     * Limit to restart_without_downtime_ready? input plugin
   * Send SIGTERM to the old supervisor after 10s delay from 3.
5. The old supervisor stops and sends SIGRTMIN(34) to the new one.
6. The new workers run fully.

Problem to solve:

Updating Fluentd or reloading a config causes downtime.
Plugins that receive data as a server, such as `in_udp`, `in_tcp`,
and `in_syslog`, cannot receive data during this time.
This means that the data sent by a client is lost during this
time unless the client has a re-sending feature.
This makes updating Fluentd or reloading a config difficult in
some cases.

Note: need these feature

* #4661
* treasure-data/serverengine#146

Co-authored-by: Shizuo Fujita <[email protected]>
Signed-off-by: Daijiro Fukuda <[email protected]>
daipom added a commit that referenced this pull request Oct 31, 2024
Add a new feature: Update/Reload without downtime.

1. The supervisor receives SIGUSR2.
2. Spawn a new supervisor.
3. Take over shared sockets.
4. Launch new workers, and stop old processes in parallel.
   * Launch new workers with source-only mode
     * Limit to restart_without_downtime_ready? input plugin
   * Send SIGTERM to the old supervisor after 10s delay from 3.
5. The old supervisor stops and sends SIGRTMIN(34) to the new one.
6. The new workers run fully.

Problem to solve:

Updating Fluentd or reloading a config causes downtime.
Plugins that receive data as a server, such as `in_udp`, `in_tcp`,
and `in_syslog`, cannot receive data during this time.
This means that the data sent by a client is lost during this
time unless the client has a re-sending feature.
This makes updating Fluentd or reloading a config difficult in
some cases.

Note: need these feature

* #4661
* treasure-data/serverengine#146

Co-authored-by: Shizuo Fujita <[email protected]>
Signed-off-by: Daijiro Fukuda <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant