-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wildcard match without suffix algorithm seems like a hazard #32
Comments
fluent-bit-go and its using go output plugin cannot handle tag information at startup. tag information will be handled in FLBPluginFlushCtx. |
My thought was that the Match directive could be checked, but it seems this isn't exposed in the config property map or any other api. Since opening this I've realized that collisions can happen due to the batching of records, even without the use of a wildcard match. It seems not using suffixes is problematic except in certain low throughput circumstances. |
I added warning message in 06bddb9. Yeah, I'm checking this plugin in low throughput environment. |
Problem
Using
Match foo.*
with no suffix can lead to cause multiple uploads to the same file if there is more than one distinct tag being flushed in the same second.#24 added a suffix mechanism to avoid this, but it isn't enabled by default. Because this issue can lead to data being lost silently, I think more should be done to avoid this scenario.
Mitigation
A warning could be generated at startup if a wildcard match is used and suffix algorithm is not specified explicitly.
Alternately, the default suffix algorithm could be changed when a wildcard match is used.
The text was updated successfully, but these errors were encountered: