-
Notifications
You must be signed in to change notification settings - Fork 236
Document considerations related to lowering the collation strength from the default #6662
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -67,7 +67,43 @@ | |||||||||||||||
| //! | ||||||||||||||||
| //! ## Strength | ||||||||||||||||
| //! | ||||||||||||||||
| //! The degree of sensitivity in how to determine that strings are distinct. | ||||||||||||||||
| //! The collation strength indicates how many levels to compare. The primary | ||||||||||||||||
| //! level considers base letters, i.e. 'a' and 'b' are unequal but 'E' and 'é' | ||||||||||||||||
| //! are equal, with further levels dealing with distinctions such as accents | ||||||||||||||||
| //! and case. | ||||||||||||||||
| //! | ||||||||||||||||
| //! If an earlier level isn't equal, the earlier level is decisive. | ||||||||||||||||
| //! If the result is equal on a level, but the strength is higher, | ||||||||||||||||
| //! the comparison proceeds to the next level. | ||||||||||||||||
|
Comment on lines
+75
to
+77
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||
| //! | ||||||||||||||||
| //! Note that lowering the strength means that more user-perceptible differences | ||||||||||||||||
| //! compare as equal. This may make sense when sorting more complex structures | ||||||||||||||||
|
Comment on lines
+79
to
+80
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||
| //! where the string to be compared is just one field, and ties between strings | ||||||||||||||||
| //! that differ only in case, accent, or similar are resolved by comparing some | ||||||||||||||||
| //! secondary field in the larger structure to be sorted. | ||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. optional: you could mention that some browsers implement the "Find" functionality for a page based on collation, ex: via the StringSearch API in ICU. This allows them to choose between primary strength so that it can ignore diacritics and case ("koln" matching on "Köln") or secondary strength to only match exactly on diacritics ("Bár" not matching "Bär"). |
||||||||||||||||
| //! | ||||||||||||||||
| //! However, if the sort is just a string sort without some other field for | ||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In general for this paragraph, I think it should be worded as a detailed caution to the user. It expounds on the main point of the previous paragraph, IMO.
Suggested change
|
||||||||||||||||
| //! resolving ties, lowering the strength means that factors that don't make | ||||||||||||||||
| //! sense to the user (such as the order of items prior to sorting with a stable | ||||||||||||||||
| //! sort algorithm or the internal details of a sorting algorithm that doesn't | ||||||||||||||||
| //! provide the stability property) affect the relative order of strings that | ||||||||||||||||
| //! do have user-perceptible differences particularly in accents or case. | ||||||||||||||||
| //! | ||||||||||||||||
| //! Lowering the strength is less of a perfomance optimization that it may seem | ||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||
| //! directly from the above description. As described above, in the case | ||||||||||||||||
| //! of identical strings to be compared, the algorithm has to work though all | ||||||||||||||||
| //! the levels included in the strength without an early exit. However, this | ||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||
| //! collator implements an identical prefix optimization, which examines the | ||||||||||||||||
| //! code units of the strings to be compared to skip the identical prefix before | ||||||||||||||||
| //! starting the actual collation algorithm. When the strings to be compared | ||||||||||||||||
| //! are identical on the byte level, they are found to be equal without the | ||||||||||||||||
| //! actual collation algorithm running at all! Therefore, the strength setting | ||||||||||||||||
| //! only has an effect (whether order effect or performance effect) for | ||||||||||||||||
| //! comparisons where the strings to be compared are not equal on the byte level | ||||||||||||||||
| //! but are equal on the primary level/strength. The common cases are that | ||||||||||||||||
| //! a comparison is decided on the primary level or the strings are byte | ||||||||||||||||
| //! equal, which narrows the performance effect of lowering the strength | ||||||||||||||||
| //! setting. | ||||||||||||||||
| //! | ||||||||||||||||
| //! ``` | ||||||||||||||||
| //! use core::cmp::Ordering; | ||||||||||||||||
|
|
||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,13 +15,45 @@ use crate::{ | |
| CollatorPreferences, | ||
| }; | ||
|
|
||
| /// The collation strength that indicates how many levels to compare. | ||
| /// The collation strength that indicates how many levels to compare. The primary | ||
| /// level considers base letters, i.e. 'a' and 'b' are unequal but 'E' and 'é' | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is it guaranteed to be the "base letter"? How does that map to other writing systems, or to locales that sort certain diacritics nonadjacent to their base letter (like Danish Å)?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. On the one hand: I don't think it's guaranteed to be a base letter, per se. For example, the Rupee sign U+20A8 used to sort with other currency symbols. Now it sorts with the ASCII letter sequence "Rs". The German sharp S ( On the other hand: the ICU User Guide uses the terms "base letter" and "base character": https://unicode-org.github.io/icu/userguide/collation/concepts.html I think we can reuse ICU terminology for ICU4X docs, at least here for the purposes of illustrating the effects of different strength levels to explain strength. |
||
| /// are equal, with further levels dealing with distinctions such as accents | ||
| /// and case. | ||
| /// | ||
| /// If an earlier level isn't equal, the earlier level is decisive. | ||
| /// If the result is equal on a level, but the strength is higher, | ||
| /// the comparison proceeds to the next level. | ||
| /// | ||
| /// Note: The bit layout of `CollatorOptions` requires `Strength` | ||
| /// Note that lowering the strength means that more user-perceptible differences | ||
| /// compare as equal. This may make sense when sorting more complex structures | ||
| /// where the string to be compared is just one field, and ties between strings | ||
| /// that differ only in case, accent, or similar are resolved by comparing some | ||
| /// secondary field in the larger structure to be sorted. | ||
|
Comment on lines
+28
to
+31
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does this use case actually make sense? You say in the next paragraph that it doesn't really make sense.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think the subsequent paragraph is just expounding on the main point of this paragraph. If so, then I think that the subsequent beginning with the word "However" is confusing for the reader. |
||
| /// | ||
| /// However, if the sort is just a string sort without some other field for | ||
| /// resolving ties, lowering the strength means that factors that don't make | ||
| /// sense to the user (such as the order of items prior to sorting with a stable | ||
| /// sort algorithm or the internal details of a sorting algorithm that doesn't | ||
| /// provide the stability property) affect the relative order of strings that | ||
| /// do have user-perceptible differences particularly in accents or case. | ||
| /// | ||
| /// Lowering the strength is less of a perfomance optimization that it may seem | ||
| /// directly from the above description. As described above, in the case | ||
| /// of identical strings to be compared, the algorithm has to work though all | ||
| /// the levels included in the strength without an early exit. However, this | ||
| /// collator implements an identical prefix optimization, which examines the | ||
| /// code units of the strings to be compared to skip the identical prefix before | ||
| /// starting the actual collation algorithm. When the strings to be compared | ||
| /// are identical on the byte level, they are found to be equal without the | ||
| /// actual collation algorithm running at all! Therefore, the strength setting | ||
| /// only has an effect (whether order effect or performance effect) for | ||
| /// comparisons where the strings to be compared are not equal on the byte level | ||
| /// but are equal on the primary level/strength. The common cases are that | ||
| /// a comparison is decided on the primary level or the strings are byte | ||
| /// equal, which narrows the performance effect of lowering the strength | ||
| /// setting. | ||
| /// | ||
| /// Note: The bit layout of `CollatorOptionsBitField` requires `Strength` | ||
| /// to fit in 3 bits. | ||
| #[derive(Eq, PartialEq, Debug, Copy, Clone, PartialOrd, Ord)] | ||
| #[repr(u8)] | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to clarify that the "high" / "low" relationship corresponds to primary = lowest, indentical = highest