Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for reading files encoded in UTF-16 for Tail Input #9321

Open
PrzemyslawDusza opened this issue Sep 2, 2024 · 0 comments · May be fixed by #9468
Open

Add support for reading files encoded in UTF-16 for Tail Input #9321

PrzemyslawDusza opened this issue Sep 2, 2024 · 0 comments · May be fixed by #9468

Comments

@PrzemyslawDusza
Copy link

Is your feature request related to a problem? Please describe.
Microsoft SQL Server instance logs errors and information to ERRORLOG files which is stored on a drive. That files are encoded in 'UTF-16 LE BOM'. Fluent-bit is able to read and parse that file and finally to send it to output (e.g. OpenTelemetry) however it does not track offsets. Every time when file is modified, fluent-bit reads whole content of the file instead of only tail which was changed which results with hundreds or thousands of duplicated entries in output.

Describe the solution you'd like
Introduce support for UTF-16 LE BOM for tail input.

Describe alternatives you've considered
There is no way to force SQL Server instance to change encoding, one of the alternative ways is to convert SQL Server error logs from unsupported format to UTF-8 on the fly using external script and configure fluent-bit to read converted file.

Additional context
In environment with thousands of SQL Server instances it requires to introduce script which will convert the log files frequently to export up-to-date logs to OpenTelemetry

image
ERRORLOG.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant