Flesh out testing and fix bugs for string primary key probes by peterdukelarsen · Pull Request #38030 · MaterializeInc/materialize

peterdukelarsen · 2026-08-03T23:41:37Z

Motivation

Split out some testing from #38022 to keep that PR manageable.

Part of: SS-97

Description

Adds testing for more diverse character types, collations, and for asserting searches use the B-tree rather than scanning the full table.

For Context

I've caught a couple of bugs in the later PR from discussion with Marty and a review with Dennis: #38093.

For PAD SPACE collations certain characters will sort before a shorter string. Our algorithm still works cleanly for that case at a high level, just throwing out the lower bounds which for utf8mb4_bin are all things like null, carriage return, and tab. The one tricky spot we needed to handle correctly was the upper bound, for which we're able to append NUL characters to get behavior like what we would have expected initially. I tried fixing this directly initially, but thought the fix was more complex than the perf issue was worth (see here).
Certain collations can't be supported with this approach because multiple characters in a row can in fact sort differently from it's prefix. There are tests here highlighting that and then most of the tests have been moved to utf8mb4_bin to match the first collation we intend to support.

Restores the broader live test suite on top of the probe interface PR: case-insensitive traversal, LIKE wildcard and metacharacter data, multibyte and emoji keys, ULID and UUID primary keys, collation behavior, stale table statistics, and sargability via Handler_read session counters.

ublubu

Thanks for all these tests. Walking through the examples gave me some ideas.

ublubu · 2026-08-06T19:09:52Z

+        // Although the sorting is case-insensitive the values returned by mysql are not normalized, so
+        // we need to use the correct character representation here to get the tests to pass.


Does this comment refer to L352 above? "utf8mb4_0900_ai_ci"

Indirectly I guess. It's more about the immediate following lines being mixed case.

Line 352 is relevant because it is what sets it to case-insensitive sorting explicitly (although that's mysql's default). It is more specifically about why I have a specific mix of upper and lower case "A" "b" "C" in the following lines.

ublubu · 2026-08-06T19:21:40Z


+    #[mz_ore::test(tokio::test)]
+    #[cfg_attr(miri, ignore)]
+    async fn test_case_insensitive_prefix_traversal() -> Result<(), anyhow::Error> {


I know what's happening here, but it'd be nice to have some comments during the traversal to say things like:

Segment the key space with 1-character prefixes as fenceposts.

Subdivide the [A, b) segment with 2-character prefixes as fenceposts.

" [b, C) "

" [C, ...) ". No further subdivision is possible, so we find no new fenceposts.

It may be worth noting that we must keep all the too-short fenceposts because of the a, aa, aaa situation.
That is, row a lives in its own partition [a, aa). If we forgot about fencepost (prefix) a, we'd start at aa (and row a would be lost).

Re: too-short prefixes

When we call prefix_of_first_row_not_matching_prefix, there are two ways we might get len(prefix) < max_prefix_length:

We got prefix from a previous iteration with a lower max_prefix_length.

Our last call to prefix_of_first_row_not_matching_prefix found a too-short row.

If we start with max_prefix_length=1 and only increment the length by 1 between rounds, scenario 2 can't happen...
unless there's a row with length 0.
(But if we start at max_prefix_length=2 or increment by 2, then scenario 2 can happen.)

Regardless of whether it's scenario 1 or 2, it's possible that a too-short prefix means we have a short row that we must account for. (See suggested note in previous section.)

I had a think, and I believe there's a clean way to explain everything:

Suppose we're segmenting the key range [a, aa, aaa, b, ba, c].
Suppose we want to use 2-character prefixes as our fenceposts.

problem: a isn't long enough to match any 2-character prefix.
solution: Let 0 be a NULL character. Treat each key as having a bunch of NULLs after it.
For example, a, a0, and a00 are equivalent. Now we can use the prefix a0 to match a.

Now, the algorithm (which we're already following) can be described like this:

You have a range of keys. For example, the range encompassing all possible keys is ['', inf), from the empty prefix up to infinity.

You subdivide the range of keys by increasing the prefix length.

'' (length 0) becomes '<null>' (length 1), which only matches the empty string. Then prefix_of_first_row_not_matching_prefix gives you the prefix 'a'. Then 'b', etc.

'a' becomes 'a<null>', so the next 2-character prefix is 'aa'.

If you go straight from 0 to 2, '' becomes '<null><null>'. The next prefix is 'a<null>'. Then 'aa', etc.

Return to step 1 for any sub-range that is still too big.

At the end, you should have a contiguous sequence of ranges, each fenceposted by a single prefix.

This way we don't have to think too hard about any special cases. And we can easily explain why we use different logic to match against too-short prefixes.

This is really well-said and gets at the core of what I was discussing with Marty yesterday. What I landed on doing was actually intentionally throwing out the short keys if the buckets are too big and we need to step down into them.

The two key contextual pieces there are:

I'm planning to have a lower bound of 10-20k for a single prefix that we try to split. So losing 1 row out of 10k doesn't meaningfully impact the results

I'm planning to limit the budget of specific calls to the DB to exit early -- i.e. ~3k for a big table. This means we're already sunk in the case of a jagged key.

All-together for the actual algorithm I think that leaves us safe to represent it as a list of Prefixes, where a prefix is defined by the prefix itself and an upper bound that we carry through.

ublubu · 2026-08-06T19:47:24Z

+        drop_db(&mut conn, DB).await?;
+        conn.disconnect().await?;
+        Ok(())
+    }


I've reviewed up to here 😅

peterdukelarsen · 2026-08-10T14:42:27Z

        const DB: &str = "mz_probe_explain_test";
        let ids: Vec<String> = (0..1000).map(|i| format!("a{i:05}")).collect();
-        let table = setup_table(&mut conn, DB, "utf8mb4_0900_ai_ci", &ids).await?;
+        let table = setup_table(&mut conn, DB, "utf8mb4_bin", &ids).await?;


Moved tests to "utf8mb4_bin" by default because we're not going to support utf8mb4_0900_ai_ci to start. See the "ß" and NULL tests I added at the bottom for "utf8mb4_0900_ai_ci". Got the idea for looking into NULLs from @ublubu's idea of inserting nulls, and found the ß issue after @martykulma called out the Czech collation issue.

def-

No further complaints from QA side

ublubu

looks good to me

ublubu · 2026-08-11T19:17:58Z

+    ///
+    /// The upper bound is padded with NUL characters so that no key it
+    /// prefixes falls inside the range. Under PAD SPACE collations like
+    /// `utf8mb4_bin` "ab\0" sorts before "ab", which is treated as "ab ".


This is bananas.

Actual review comment:
It's not clear to me what it means for "ab" to be treated as "ab " when it comes to the placement of "ab\0" in the ordering.

IIUC, we're is trying to say:

"ab\0" sorts before both "ab" and "ab ". We can't use "ab" because "ab" is not before "ab " in PAD SPACE collations like utf8mb4_bin.

So, "ab" == "ab " == "ab " == "ab " for PAD SPACE collations. Null characters (and some other control characters such as tab) are sorted below space and therefor also "ab".

I updated the comment to be:

/// Under PAD SPACE collations like /// `utf8mb4_bin` "ab" is ordered as equivalent to "ab ..." /// (however many spaces are needed to fill remaining char/varchar length), so "ab\0" sorts /// before either of those (because NUL is below all other characters in /// `utf8mb4_bin`).

ublubu · 2026-08-11T20:15:06Z

+    /// `utf8mb4_bin` compares character by character but is PAD SPACE, so
+    /// keys starting below space sort below the empty string. A walk seeded
+    /// with the empty string drops them, they land in the snapshot range
+    /// left of the first boundary.


Similar question here. I see this is weird and worth documenting in a test. But what impact does it have on us?

Yeah, this will be fine, it just means we'll have somewhat poorly balanced partitioning. The walk will still converge. Added a comment explaining it's ok.

to catch potential `PAD SPACE` upper-bound issues Follow-up to #38030

peterdukelarsen force-pushed the pl/mysql-key-probes-tests branch 11 times, most recently from 3e56f33 to 45ef8e6 Compare August 6, 2026 17:31

peterdukelarsen changed the title ~~storage: Flesh out testing for string primary key probes~~ Flesh out testing for string primary key probes Aug 6, 2026

peterdukelarsen force-pushed the pl/mysql-key-probes-tests branch from 45ef8e6 to 8d946b5 Compare August 6, 2026 18:12

peterdukelarsen marked this pull request as ready for review August 6, 2026 18:13

peterdukelarsen requested a review from a team as a code owner August 6, 2026 18:13

peterdukelarsen requested a review from a team August 6, 2026 18:13

peterdukelarsen mentioned this pull request Aug 6, 2026

Flesh out testing for string primary key probes #38092

Closed

ublubu reviewed Aug 6, 2026

View reviewed changes

def- requested changes Aug 7, 2026

View reviewed changes

Comment thread src/mysql-util/src/probe.rs

peterdukelarsen force-pushed the pl/mysql-key-probes-tests branch 7 times, most recently from 2081b17 to 527925c Compare August 10, 2026 14:34

peterdukelarsen commented Aug 10, 2026

View reviewed changes

peterdukelarsen force-pushed the pl/mysql-key-probes-tests branch 2 times, most recently from 947274e to b6db40e Compare August 10, 2026 15:01

peterdukelarsen force-pushed the pl/mysql-key-probes-tests branch 6 times, most recently from cee8cb4 to 91dc051 Compare August 10, 2026 17:56

peterdukelarsen changed the title ~~Flesh out testing for string primary key probes~~ Flesh out testing and fix bugs for string primary key probes Aug 10, 2026

peterdukelarsen force-pushed the pl/mysql-key-probes-tests branch 7 times, most recently from 8135596 to 4819f94 Compare August 10, 2026 20:20

mysql-util: extend probe tests and NUL-pad the upper prefix bounds

19161b5

peterdukelarsen force-pushed the pl/mysql-key-probes-tests branch from 4819f94 to 19161b5 Compare August 10, 2026 21:10

peterdukelarsen requested a review from a team August 10, 2026 21:22

mysql-util: soften the KeyProber collation support claim

55149b8

def- approved these changes Aug 11, 2026

View reviewed changes

peterdukelarsen requested a review from ublubu August 11, 2026 19:41

mysql-util: name the probed key length cap

600c6b1

ublubu approved these changes Aug 11, 2026

View reviewed changes

mysql-util: expand collation test coverage and clarify probe docs

3b1f44d

peterdukelarsen enabled auto-merge (squash) August 11, 2026 22:44

peterdukelarsen disabled auto-merge August 11, 2026 22:53

mysql-util: drop tests of unsupported collations

876ed83

peterdukelarsen enabled auto-merge (squash) August 11, 2026 23:07

peterdukelarsen merged commit efd5eb7 into MaterializeInc:main Aug 11, 2026
82 of 83 checks passed

def- mentioned this pull request Aug 13, 2026

cargo test: Sharpen mysql probe tests #38193

Merged

def- added a commit that referenced this pull request Aug 18, 2026

cargo test: Sharpen mysql probe tests (#38193)

0f7c52f

to catch potential `PAD SPACE` upper-bound issues Follow-up to #38030

		// Although the sorting is case-insensitive the values returned by mysql are not normalized, so
		// we need to use the correct character representation here to get the tests to pass.

Uh oh!

Conversation

peterdukelarsen commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

For Context

Uh oh!

ublubu left a comment

Choose a reason for hiding this comment

Uh oh!

ublubu Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

ublubu Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ublubu Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

peterdukelarsen Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

def- left a comment

Choose a reason for hiding this comment

Uh oh!

ublubu left a comment

Choose a reason for hiding this comment

Uh oh!

ublubu Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ublubu Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

peterdukelarsen commented Aug 3, 2026 •

edited

Loading

ublubu Aug 6, 2026 •

edited

Loading

peterdukelarsen Aug 11, 2026 •

edited

Loading

peterdukelarsen Aug 11, 2026 •

edited

Loading