Why doesn't max/min Graphemes have an associated Locale?

[nklisch]

Grapheme length is dependent on the locale context that the string is parse in.
The lexicon string field has a constraint for maxGraphemes and minGraphemes. But without a locale associated with the context of the parsing operation, we don't have a way to consistently "count".

For example:
String: é (U+0065 LATIN SMALL LETTER E + U+0301 COMBINING ACUTE ACCENT)
Locale 1 (e.g., French): May treat this as 1 grapheme because it recognizes é as a single user-perceived character.
Locale 2 (Generic or Root): May treat this as 2 graphemes (one for e and one for the combining accent).

This leads to inconsistent implementation of validation unless there is someway to require a locale on every message or have a default/standard locale.

I noticed the reference validation for this is just counting 16bit UTF block , which does garantee that the amount of graphemes are less then the constraint but one service could consider one record valid - because the amount of graphemes is within the min/max, and a different one using a different locale would consider it invalid.

atproto/packages/lexicon/src/validators/primitives.ts

Line 236 in 011e73e

if (value.length <= def.maxGraphemes) {

See my bluesky post where I tested some limits with different large grapheme cluster emojis: https://bsky.app/profile/the.nathanklisch.com/post/3lbkd5cw43c2v

Am I missing something?

[DavidBuchanan314]

The source line you link is just an early-exit optimisation, full grapheme counting happens later, using the Graphemer package https://www.npmjs.com/package/graphemer

Graphemer claims "it is an implementation on the Default Grapheme Cluster Boundary of UAX #29." You're right that locale-specific rule variations can be used, but the default seems well-defined. (and the atproto docs should probably say something about that)

[nklisch]

Ok, I guess we need to specify that that is the standard. I am implementing validations in Java and the protocol needs to specify how we are doing our breaks.

[nklisch]

Thanks for the additional edits and clarifications. I'll do default for my implementation, but yes, needs to be called out in the protocol specs.
Thanks again.