simdutf: simdutf_connector: in_tail: Implement UTF-16LE/UTF-16BE encoder #9468
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cosmo0920
force-pushed
the
cosmo0920-try-to-bundle-simdutf-amalgamation
branch
from
d1b404a
to
4053bbd
Compare
cosmo0920
force-pushed
the
cosmo0920-try-to-bundle-simdutf-amalgamation
branch
from
4053bbd
to
2a515ea
Compare
cosmo0920
force-pushed
the
cosmo0920-try-to-bundle-simdutf-amalgamation
branch
from
655a31a
to
19a0a1a
Compare
Signed-off-by: Hiroshi Hatake <[email protected]>
cosmo0920
force-pushed
the
cosmo0920-try-to-bundle-simdutf-amalgamation
branch
from
df32358
to
36537a4
Compare
Signed-off-by: Hiroshi Hatake <[email protected]>
Signed-off-by: Hiroshi Hatake <[email protected]>
patrick-stephens
approved these changes
Oct 28, 2024
| # This is provided for simplifying the build pipeline | ||
| FLB_UNICODE_ENCODER=${FLB_UNICODE_ENCODER:-On} | ||
|
|
||
| if [ "${FLB_DISTRO}" = "centos/6" ] || [ "${FLB_DISTRO}" = "centos/7" ] || |
There was a problem hiding this comment.
We don't support CentOS 6 in OSS
|
|
||
| if [ "${FLB_DISTRO}" = "centos/6" ] || [ "${FLB_DISTRO}" = "centos/7" ] || | ||
| [ "${FLB_DISTRO}" = "centos/7.arm64v8" ]; then | ||
| FLB_UNICODE_ENCODER=Off |
There was a problem hiding this comment.
I thought the plan was to disable it in the PR check but enable it (with all the SCL requirements added) to the container images for release builds?
There was a problem hiding this comment.
Yeah, but, I didn't find a way how to enable SCL on CentOS7 container builds on our packaging system...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In Windows, there are lots of using UTF-16LE programs. This is because Unicode on Windows means UTF-16LE with BOM(Byte Order Mark).
In addition, there is lots of differences between UTF-16LE/UTF-16BE and UTF-8.
I added some of C, J and subdivision flags test cases for converting from UTF-16LE/UTF-16BE to UTF-8 in unit tests for in_tail plugin. This is because in_tail is the main usages to process non-UTF-8 encodings.
At first, we need to process UTF-16LE and UTF-16BE encodings.
Note that simdutf library is written in C++. So, we also provide an option (
FLB_UNICODE_ENCODER) to turn on/off this feature.Closes #9321
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
fluent/fluent-bit-docs#1471
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.