-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-48715][SQL] Integrate UTF8String validation into collation-aware string function implementations #47131
Conversation
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mkaravel ready for another round of review
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
Outdated
Show resolved
Hide resolved
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cloud-fan ready for review
thanks, merging to master! |
…re string function implementations ### What changes were proposed in this pull request? Use our own invalid UTF-8 byte sequence replacement logic in UTF8String, before all `.toString()` method calls. ### Why are the changes needed? Avoid relying on Java to perform invalid UTF-8 byte sequence replacement, and ensure consistent results. ### Does this PR introduce _any_ user-facing change? Yes, collation aware string function implementations will now rely on our own invalid UTF-8 string replacement implementation, instead of Java's. ### How was this patch tested? Existing tests, with some changes in `UTF8StringSuite` and `CollationSupportSuite`. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47131 from uros-db/make-valid. Authored-by: Uros Bojanic <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
What changes were proposed in this pull request?
Use our own invalid UTF-8 byte sequence replacement logic in UTF8String, before all
.toString()
method calls.Why are the changes needed?
Avoid relying on Java to perform invalid UTF-8 byte sequence replacement, and ensure consistent results.
Does this PR introduce any user-facing change?
Yes, collation aware string function implementations will now rely on our own invalid UTF-8 string replacement implementation, instead of Java's.
How was this patch tested?
Existing tests, with some changes in
UTF8StringSuite
andCollationSupportSuite
.Was this patch authored or co-authored using generative AI tooling?
No.