-
Notifications
You must be signed in to change notification settings - Fork 512
docs: draft MySQL snapshot parallelism page #38091
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| --- | ||
| headless: true | ||
| --- | ||
|
|
||
| Materialize can parallelize snapshotting across the workers of the cluster | ||
| hosting the source. | ||
|
|
||
| - **PostgreSQL sources** are parallelized by table, i.e., different tables | ||
| are read concurrently by different workers. On PostgreSQL 14 and later, | ||
| Materialize additionally attempts to partition each table's read across | ||
| workers. Tables that cannot be partitioned fall back to a single worker. | ||
|
|
||
| - **MySQL sources** are parallelized by table, i.e., different tables are | ||
| read concurrently by different workers. For tables that meet certain | ||
| requirements, Materialize can additionally partition the table's read | ||
| across workers {{< private-preview-inline />}}. See [MySQL snapshot | ||
| parallelism](/ingest-data/mysql/snapshot-parallelism/). | ||
|
|
||
| - **Kafka sources** are parallelized by topic partition, with partitions | ||
| distributed across workers, so parallelism is bounded by the topic's | ||
| partition count. | ||
|
|
||
| - **SQL Server sources** are not parallelized: a single worker reads all | ||
| tables. | ||
|
|
||
| The degree of snapshot parallelism depends on the number of workers. A | ||
| cluster's [size](/sql/create-cluster/#available-sizes) determines its number | ||
| of workers, so a larger cluster can shorten the snapshot, to the extent the | ||
| work parallelizes and the upstream database keeps up. The volume read from | ||
| the upstream database is unchanged, it is compressed into a shorter window | ||
| of more concurrent queries and connections. To determine whether | ||
| snapshotting is overloading the upstream database, and for ways to mitigate | ||
| the load, see [Is the upstream database | ||
| overloaded?](/ingest-data/troubleshooting/#is-the-upstream-database-overloaded) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,108 @@ | ||
| --- | ||
| title: "Snapshot parallelism" | ||
| description: "How Materialize splits the snapshot of a single MySQL table across the workers of a cluster." | ||
| menu: | ||
| main: | ||
| parent: "mysql" | ||
| name: "Snapshot parallelism" | ||
| identifier: "mysql-snapshot-parallelism" | ||
| weight: 70 | ||
| --- | ||
|
|
||
| {{< private-preview />}} | ||
|
|
||
| When you create a [MySQL source](/sql/create-source/mysql-v2/), Materialize | ||
|
kay-kim marked this conversation as resolved.
|
||
| performs an initial, snapshot-based sync of the selected tables before it | ||
| starts ingesting change events from the binlog. For large tables, this | ||
| snapshot dominates the time until the source becomes healthy. | ||
|
|
||
| How snapshot work is spread across the workers of a cluster, and what that | ||
| means for the upstream database, is covered in | ||
| [Snapshotting](/concepts/snapshotting/#parallelism). Materialize can split | ||
| the read of a **single table** across all the workers of the cluster, so | ||
| that even a source dominated by one very large table benefits from a larger | ||
| cluster. This page covers what is specific to MySQL: which tables are | ||
| eligible for splitting, and how their reads are partitioned. | ||
|
|
||
| ## Which tables are split | ||
|
|
||
| Materialize splits the snapshot of an individual table across workers when | ||
| all of the following conditions are met: | ||
|
|
||
| - The table has a **single-column primary key**. Composite primary keys are | ||
| not supported. | ||
| - The primary key column is of type **`CHAR` or `VARCHAR`**, with a declared | ||
| length of **at most 768 characters**. Other types, including numeric keys, | ||
| are not supported. | ||
| - The primary key column uses the **`utf8mb4` character set** with the | ||
| **`utf8mb4_bin` collation**. | ||
| - The table is **large enough to be worth splitting**. Small tables are read | ||
| by a single worker, where splitting would add overhead without benefit. | ||
|
|
||
| How evenly the split lands also depends on the distribution of the key | ||
| values. See [How a table is partitioned](#how-a-table-is-partitioned). | ||
|
|
||
| If a table does not meet these requirements, or if the [boundary | ||
| sampling](#how-a-table-is-partitioned) fails, its snapshot is not split: a | ||
| single worker reads the table in full. Different tables are still read | ||
| concurrently by different workers. | ||
|
|
||
| ## How a table is partitioned | ||
|
|
||
| Materialize partitions an [eligible](#which-tables-are-split) table using the | ||
| leading characters of its primary key values. Before reading the table, | ||
| Materialize probes the primary key index to discover key prefixes and uses | ||
| the MySQL optimizer's row estimates to gauge how many rows fall under each | ||
| prefix. It extends the prefixes as needed to find boundaries that divide the | ||
| table into roughly even ranges. The probes are inexpensive point lookups, | ||
| capped in proportion to the table's estimated size, so the sampling phase | ||
| stays negligible next to the snapshot itself. | ||
|
|
||
| Each worker then reads only its assigned range, within the same consistent | ||
| snapshot of the upstream database, so the result is identical to a | ||
| single-worker snapshot, only faster. | ||
|
|
||
| Because partitioning is based on key prefixes and optimizer estimates, how | ||
| evenly the work divides depends on the shape of your keys: | ||
|
|
||
| - **Evenly distributed keys partition well.** Keys whose leading characters | ||
| spread rows uniformly, such as UUIDs, hashes, or other randomized | ||
| identifiers, produce well-balanced ranges. | ||
|
|
||
| - **Skewed keys partition less evenly.** If a large share of the table's rows | ||
| sort under a few common prefixes, some ranges end up with more rows than | ||
| others, and the workers assigned to them finish later. | ||
|
|
||
| - **The probe budget can run out.** If finding even boundaries would require | ||
| examining very many distinct prefixes, Materialize stops probing and uses | ||
| the coarser boundaries found so far, which can also leave ranges uneven. | ||
|
|
||
| Uneven partitioning is never incorrect. It only reduces the speedup, since | ||
| the snapshot finishes when the busiest worker finishes. | ||
|
|
||
| ## MySQL-specific upstream considerations | ||
|
|
||
| - **Connection count.** While the snapshot is being set up, Materialize | ||
| briefly holds up to two connections per worker, plus one. Once reading is | ||
| underway, this settles to one connection per worker reading a range, plus | ||
| one coordination connection. After the snapshot completes, the source drops | ||
| back to a single replication connection. If your MySQL server or connection | ||
| pooler enforces a low | ||
| [`max_connections`](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_connections) | ||
| limit, account for this burst when sizing it. | ||
|
|
||
| - **Statistics freshness.** Range boundaries are placed using the MySQL | ||
| optimizer's row estimates. Stale statistics don't affect correctness, but | ||
| can skew how evenly work divides across workers. Running | ||
| [`ANALYZE TABLE`](https://dev.mysql.com/doc/refman/8.0/en/analyze-table.html) | ||
| on very large tables before creating the source can improve balance. | ||
|
Comment on lines
+84
to
+98
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd keep these here for now: both bullets only apply when parallel snapshotting is active, which is private preview and flag-off, so they'd be noise in the general MySQL considerations. Worth revisiting when the feature is on by default. |
||
|
|
||
| For general guidance on read load, IOPS, and other upstream impact, which is | ||
| not specific to MySQL, see [Is the upstream database | ||
| overloaded?](/ingest-data/troubleshooting/#is-the-upstream-database-overloaded) | ||
|
|
||
| ## Observability | ||
|
|
||
| To observe the progress of an ongoing snapshot, see [Monitoring the | ||
| snapshotting | ||
| progress](/ingest-data/monitoring-data-ingestion/#monitoring-the-snapshotting-progress). | ||
Uh oh!
There was an error while loading. Please reload this page.