Revert backup chunk specification for suppressing useless chunk backup sets#247
Open
bysnupy wants to merge 1 commit intofluent-plugins-nursery:masterfrom
Open
Revert backup chunk specification for suppressing useless chunk backup sets#247bysnupy wants to merge 1 commit intofluent-plugins-nursery:masterfrom
bysnupy wants to merge 1 commit intofluent-plugins-nursery:masterfrom
Conversation
…p sets "Fluent::Unrecoverable Error" make the log set backup per a whole chunk unit, not only relevant log message. It cause irrelevant logs lost due to whole chunk backup, in other words, not sending and backup whole chunk data set including the bigger log than 256 kb. In addition the chunk backup file is remained as useless things, because there is no solution to restore the backup sets now. Refer fluent/fluentd#3813
cosmo0920
requested changes
Apr 4, 2024
Member
cosmo0920
left a comment
cosmo0920
left a comment
There was a problem hiding this comment.
This change shouldn't be accepted. Because this change introduces malformed requests which contain too large data for ingested AWS CloudWatch Logs. This should cause unnecessary consuming data bandwidths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Fluent::Unrecoverable Errormakes the log set backup per a whole chunk unit, not only relevant log message.It even causes irrelevant logs lost due to not sending whole chunk sets for backup, in other words, unrecoverable error can block sending the whole chunk data set which has any bigger logs than 256kb. Ideally, we should skip only to send the certain bigger log message than 256kb, not all chunk data including multiple log messages.
In addition the chunk backup file would keep accumulating as useless things, because there is no solution to restore the chunk backup set binary files now. Refer fluent/fluentd#3813 for potential risk of this behavior please.