Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wildcard match without suffix algorithm seems like a hazard #32

Open
groner opened this issue Apr 2, 2020 · 3 comments
Open

Wildcard match without suffix algorithm seems like a hazard #32

groner opened this issue Apr 2, 2020 · 3 comments

Comments

@groner
Copy link
Contributor

groner commented Apr 2, 2020

Problem

Using Match foo.* with no suffix can lead to cause multiple uploads to the same file if there is more than one distinct tag being flushed in the same second.

#24 added a suffix mechanism to avoid this, but it isn't enabled by default. Because this issue can lead to data being lost silently, I think more should be done to avoid this scenario.

Mitigation

A warning could be generated at startup if a wildcard match is used and suffix algorithm is not specified explicitly.

Alternately, the default suffix algorithm could be changed when a wildcard match is used.

@cosmo0920
Copy link
Owner

cosmo0920 commented Apr 3, 2020

Mitigation

A warning could be generated at startup if a wildcard match is used and suffix algorithm is not specified explicitly.

Alternately, the default suffix algorithm could be changed when a wildcard match is used.

fluent-bit-go and its using go output plugin cannot handle tag information at startup. tag information will be handled in FLBPluginFlushCtx.

@groner
Copy link
Contributor Author

groner commented Apr 3, 2020

My thought was that the Match directive could be checked, but it seems this isn't exposed in the config property map or any other api.

Since opening this I've realized that collisions can happen due to the batching of records, even without the use of a wildcard match. It seems not using suffixes is problematic except in certain low throughput circumstances.

cosmo0920 added a commit that referenced this issue Apr 14, 2020
@cosmo0920
Copy link
Owner

I added warning message in 06bddb9.

Yeah, I'm checking this plugin in low throughput environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants