Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 37 additions & 1 deletion components/collator/README.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

38 changes: 37 additions & 1 deletion components/collator/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,43 @@
//!
//! ## Strength
//!
//! The degree of sensitivity in how to determine that strings are distinct.
//! The collation strength indicates how many levels to compare. The primary
//! level considers base letters, i.e. 'a' and 'b' are unequal but 'E' and 'é'
//! are equal, with further levels dealing with distinctions such as accents
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify that the "high" / "low" relationship corresponds to primary = lowest, indentical = highest

Suggested change
//! are equal, with further levels dealing with distinctions such as accents
//! are equal, with higher levels dealing with distinctions such as accents

//! and case.
//!
//! If an earlier level isn't equal, the earlier level is decisive.
//! If the result is equal on a level, but the strength is higher,
//! the comparison proceeds to the next level.
Comment on lines +75 to +77
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
//! If an earlier level isn't equal, the earlier level is decisive.
//! If the result is equal on a level, but the strength is higher,
//! the comparison proceeds to the next level.
//! If an lower level isn't equal, the lower level is decisive.
//! If the comparison result is equal on one level,
//! but the collator's strength input value is higher than that,
//! then the collator comparison iteratively proceeds to the next higher level.

//!
//! Note that lowering the strength means that more user-perceptible differences
//! compare as equal. This may make sense when sorting more complex structures
Comment on lines +79 to +80
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
//! Note that lowering the strength means that more user-perceptible differences
//! compare as equal. This may make sense when sorting more complex structures
//! Note that lowering the strength value given to the collator means that more user-perceptible
//! differences will compare as equal. This may make sense when sorting more complex structures

//! where the string to be compared is just one field, and ties between strings
//! that differ only in case, accent, or similar are resolved by comparing some
//! secondary field in the larger structure to be sorted.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional: you could mention that some browsers implement the "Find" functionality for a page based on collation, ex: via the StringSearch API in ICU. This allows them to choose between primary strength so that it can ignore diacritics and case ("koln" matching on "Köln") or secondary strength to only match exactly on diacritics ("Bár" not matching "Bär").

//!
//! However, if the sort is just a string sort without some other field for
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general for this paragraph, I think it should be worded as a detailed caution to the user. It expounds on the main point of the previous paragraph, IMO.

Suggested change
//! However, if the sort is just a string sort without some other field for
//! Therefore, if the sort is just a string sort without some other field for

//! resolving ties, lowering the strength means that factors that don't make
//! sense to the user (such as the order of items prior to sorting with a stable
//! sort algorithm or the internal details of a sorting algorithm that doesn't
//! provide the stability property) affect the relative order of strings that
//! do have user-perceptible differences particularly in accents or case.
//!
//! Lowering the strength is less of a perfomance optimization that it may seem
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
//! Lowering the strength is less of a perfomance optimization that it may seem
//! Lowering the strength is less of a perfomance optimization than it may seem

//! directly from the above description. As described above, in the case
//! of identical strings to be compared, the algorithm has to work though all
//! the levels included in the strength without an early exit. However, this
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
//! the levels included in the strength without an early exit. However, this
//! the levels, from primary up to the provided strength value given to collator, without an early exit. However, this

//! collator implements an identical prefix optimization, which examines the
//! code units of the strings to be compared to skip the identical prefix before
//! starting the actual collation algorithm. When the strings to be compared
//! are identical on the byte level, they are found to be equal without the
//! actual collation algorithm running at all! Therefore, the strength setting
//! only has an effect (whether order effect or performance effect) for
//! comparisons where the strings to be compared are not equal on the byte level
//! but are equal on the primary level/strength. The common cases are that
//! a comparison is decided on the primary level or the strings are byte
//! equal, which narrows the performance effect of lowering the strength
//! setting.
//!
//! ```
//! use core::cmp::Ordering;
Expand Down
36 changes: 34 additions & 2 deletions components/collator/src/options.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,45 @@ use crate::{
CollatorPreferences,
};

/// The collation strength that indicates how many levels to compare.
/// The collation strength that indicates how many levels to compare. The primary
/// level considers base letters, i.e. 'a' and 'b' are unequal but 'E' and 'é'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it guaranteed to be the "base letter"? How does that map to other writing systems, or to locales that sort certain diacritics nonadjacent to their base letter (like Danish Å)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the one hand: I don't think it's guaranteed to be a base letter, per se. For example, the Rupee sign U+20A8 used to sort with other currency symbols. Now it sorts with the ASCII letter sequence "Rs". The German sharp S (ß) sorts with ss, I believe.

On the other hand: the ICU User Guide uses the terms "base letter" and "base character": https://unicode-org.github.io/icu/userguide/collation/concepts.html

I think we can reuse ICU terminology for ICU4X docs, at least here for the purposes of illustrating the effects of different strength levels to explain strength.

/// are equal, with further levels dealing with distinctions such as accents
/// and case.
///
/// If an earlier level isn't equal, the earlier level is decisive.
/// If the result is equal on a level, but the strength is higher,
/// the comparison proceeds to the next level.
///
/// Note: The bit layout of `CollatorOptions` requires `Strength`
/// Note that lowering the strength means that more user-perceptible differences
/// compare as equal. This may make sense when sorting more complex structures
/// where the string to be compared is just one field, and ties between strings
/// that differ only in case, accent, or similar are resolved by comparing some
/// secondary field in the larger structure to be sorted.
Comment on lines +28 to +31
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this use case actually make sense? You say in the next paragraph that it doesn't really make sense.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the subsequent paragraph is just expounding on the main point of this paragraph. If so, then I think that the subsequent beginning with the word "However" is confusing for the reader.

///
/// However, if the sort is just a string sort without some other field for
/// resolving ties, lowering the strength means that factors that don't make
/// sense to the user (such as the order of items prior to sorting with a stable
/// sort algorithm or the internal details of a sorting algorithm that doesn't
/// provide the stability property) affect the relative order of strings that
/// do have user-perceptible differences particularly in accents or case.
///
/// Lowering the strength is less of a perfomance optimization that it may seem
/// directly from the above description. As described above, in the case
/// of identical strings to be compared, the algorithm has to work though all
/// the levels included in the strength without an early exit. However, this
/// collator implements an identical prefix optimization, which examines the
/// code units of the strings to be compared to skip the identical prefix before
/// starting the actual collation algorithm. When the strings to be compared
/// are identical on the byte level, they are found to be equal without the
/// actual collation algorithm running at all! Therefore, the strength setting
/// only has an effect (whether order effect or performance effect) for
/// comparisons where the strings to be compared are not equal on the byte level
/// but are equal on the primary level/strength. The common cases are that
/// a comparison is decided on the primary level or the strings are byte
/// equal, which narrows the performance effect of lowering the strength
/// setting.
///
/// Note: The bit layout of `CollatorOptionsBitField` requires `Strength`
/// to fit in 3 bits.
#[derive(Eq, PartialEq, Debug, Copy, Clone, PartialOrd, Ord)]
#[repr(u8)]
Expand Down
Loading