-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fluentd command: add --with-source-only #4661
Draft
daipom
wants to merge
1
commit into
fluent:master
Choose a base branch
from
daipom:command-option-add-with-source-only
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
daipom
force-pushed
the
command-option-add-with-source-only
branch
3 times, most recently
from
October 10, 2024 08:49
09f05a5
to
f55690e
Compare
It launches Fluentd with Input plugins only. Here is the specification. * Those Input plugins emits the events to SourceOnlyBufferAgent. * The events is kept in the buf_file during the source-only-mode. * SIGRTMIN(34) cancels the mode. * After canceled, the new agent starts to load the buffer. Signed-off-by: Daijiro Fukuda <[email protected]>
daipom
force-pushed
the
command-option-add-with-source-only
branch
from
October 10, 2024 14:35
f55690e
to
7ab49c3
Compare
daipom
added a commit
that referenced
this pull request
Oct 11, 2024
Add a new feature: Update/Reload without downtime. 1. The supervisor receives SIGUSR2. 2. Spawn a new supervisor. 3. Take over shared sockets. 4. Launch new workers, and stop old processes in parallel. * Launch new workers with source-only mode * Limit to restart_without_downtime_ready? input plugin * Send SIGTERM to the old supervisor after 10s delay from 3. 5. The old supervisor stops and sends SIGRTMIN(34) to the new one. 6. The new workers run fully. Problem to solve: Updating Fluentd or reloading a config causes downtime. Plugins that receive data as a server, such as `in_udp`, `in_tcp`, and `in_syslog`, cannot receive data during this time. This means that the data sent by a client is lost during this time unless the client has a re-sending feature. This makes updating Fluentd or reloading a config difficult in some cases. Note: need these feature * #4661 * treasure-data/serverengine#146 Signed-off-by: Daijiro Fukuda <[email protected]>
daipom
added a commit
that referenced
this pull request
Oct 11, 2024
Add a new feature: Update/Reload without downtime. 1. The supervisor receives SIGUSR2. 2. Spawn a new supervisor. 3. Take over shared sockets. 4. Launch new workers, and stop old processes in parallel. * Launch new workers with source-only mode * Limit to restart_without_downtime_ready? input plugin * Send SIGTERM to the old supervisor after 10s delay from 3. 5. The old supervisor stops and sends SIGRTMIN(34) to the new one. 6. The new workers run fully. Problem to solve: Updating Fluentd or reloading a config causes downtime. Plugins that receive data as a server, such as `in_udp`, `in_tcp`, and `in_syslog`, cannot receive data during this time. This means that the data sent by a client is lost during this time unless the client has a re-sending feature. This makes updating Fluentd or reloading a config difficult in some cases. Note: need these feature * #4661 * treasure-data/serverengine#146 Co-authored-by: Shizuo Fujita <[email protected]> Signed-off-by: Daijiro Fukuda <[email protected]>
daipom
added a commit
that referenced
this pull request
Oct 31, 2024
Add a new feature: Update/Reload without downtime. 1. The supervisor receives SIGUSR2. 2. Spawn a new supervisor. 3. Take over shared sockets. 4. Launch new workers, and stop old processes in parallel. * Launch new workers with source-only mode * Limit to restart_without_downtime_ready? input plugin * Send SIGTERM to the old supervisor after 10s delay from 3. 5. The old supervisor stops and sends SIGRTMIN(34) to the new one. 6. The new workers run fully. Problem to solve: Updating Fluentd or reloading a config causes downtime. Plugins that receive data as a server, such as `in_udp`, `in_tcp`, and `in_syslog`, cannot receive data during this time. This means that the data sent by a client is lost during this time unless the client has a re-sending feature. This makes updating Fluentd or reloading a config difficult in some cases. Note: need these feature * #4661 * treasure-data/serverengine#146 Co-authored-by: Shizuo Fujita <[email protected]> Signed-off-by: Daijiro Fukuda <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue(s) this PR fixes:
None.
Related to
This feature may be useful on its own, but mainly, this feature will help #4622 feature.
What this PR does / why we need it:
Add
--with-source-only
option tofluentd
command.(Ref: Similar to existing --without-source option)
It launches Fluentd with Input plugins only.
Here is the specification.
Docs Changes:
TODO
Release Note:
fluentd
command: add--with-source-only
out_buffer
plugin, which can be used for buffering and relabeling events.new plugin: out_buffer
This feature internally uses the new plugin
out_buffer
.This plugin can be used for buffering and relabeling events.
Example: in_udp -> out_buffer -> out_stdout
TODO