docs: draft MySQL snapshot parallelism page

doc/user/content/concepts/snapshotting.md

@@ -18,6 +18,10 @@ menu:
 
 {{% include-headless "/headless/ingestion/snapshotting-duration" %}}
 
+### Parallelism
+
+{{% include-headless "/headless/ingestion/snapshotting-parallelism" %}}
+
 ## Queries during snapshotting
 
 {{% include-headless "/headless/ingestion/snapshotting-queries" %}}
@@ -27,7 +31,8 @@ menu:
 Snapshotting has the following upstream impacts:
 
 - **Read load.** Snapshotting puts read, CPU, and network load on the upstream
-  system, proportional to the data volume.
+  system, proportional to the data volume and concentrated in proportion to
+  the source cluster's [parallelism](#parallelism).
 
 - **Change-log retention for CDC database sources.** When ingesting data from
   CDC database sources (PostgreSQL, MySQL, SQL Server), the upstream system must
@@ -41,3 +46,4 @@ Snapshotting has the following upstream impacts:
 
 - [Ingest data](/ingest-data/)
 - [Sources](/concepts/sources/)
+- [Troubleshooting data ingestion](/ingest-data/troubleshooting/)

doc/user/content/headless/ingestion/snapshotting-parallelism.md

@@ -0,0 +1,22 @@
+---
+headless: true
+---
+
+Materialize parallelizes snapshotting across the workers of the cluster
+hosting the source. For PostgreSQL and MySQL sources, work is distributed by
+table, with different tables read concurrently by different workers.
+PostgreSQL sources additionally partition every table, splitting its read
+across workers (on PostgreSQL 14 and later). MySQL sources partition tables
+that meet certain requirements. See [MySQL snapshot
+parallelism](/ingest-data/mysql/snapshot-parallelism/). Kafka sources are
+parallelized by topic partition, with partitions distributed across workers,
+so parallelism is bounded by the topic's partition count. SQL Server sources
+are not parallelized: a single worker reads all tables.
+
+A cluster's [size](/sql/create-cluster/#available-sizes) determines its
+number of workers, so a larger cluster shortens the snapshot. The volume
+read from the upstream database is unchanged, it is compressed into a
+shorter window of more concurrent queries and connections. To tell whether
+the upstream database is struggling under this load, and for options if it
+is, see [Is the upstream database
+overloaded?](/ingest-data/troubleshooting/#is-the-upstream-database-overloaded)

doc/user/content/ingest-data/_index.md

@@ -60,6 +60,10 @@ we recommend:
   the steady-state resource needs of your upsert source(s). See [Best practices:
   Upsert sources](#upsert-sources).
 
+### Parallelism
+
+{{% include-headless "/headless/ingestion/snapshotting-parallelism" %}}
+
 ### Monitoring progress
 
 While snapshotting is taking place, you can monitor the progress of the

doc/user/content/ingest-data/mysql/snapshot-parallelism.md

@@ -0,0 +1,104 @@
+---
+title: "Snapshot parallelism"
+description: "How Materialize splits the snapshot of a single MySQL table across the workers of a cluster."
+menu:
+  main:
+    parent: "mysql"
+    name: "Snapshot parallelism"
+    identifier: "mysql-snapshot-parallelism"
+    weight: 70
+---
+
+When you create a [MySQL source](/sql/create-source/mysql-v2/), Materialize
+performs an initial, snapshot-based sync of the selected tables before it
+starts ingesting change events from the binlog. For large tables, this
+snapshot dominates the time until the source becomes healthy.
+
+How snapshot work is spread across the workers of a cluster, and what that
+means for the upstream database, is covered in
+[Snapshotting](/concepts/snapshotting/#parallelism). This page covers what is
+specific to MySQL: Materialize can split the read of a **single table**
+across all the workers of the cluster, so that even a source dominated by one
+very large table benefits from a larger cluster.
+
+## Which tables are split
+
+The snapshot of an individual table is split across workers when all of the
+following hold:
+
+- The table has a **single-column primary key**. Composite primary keys are
+  not supported.
+- The primary key column is of type **`CHAR` or `VARCHAR`**, with a declared
+  length of **at most 768 characters**. Other types, including numeric keys,
+  are not supported.
+- The primary key column uses the **`utf8mb4` character set** with the
+  **`utf8mb4_bin` collation**.
+- The table is **large enough to be worth splitting**. Small tables are read
+  by a single worker, where splitting would add overhead without benefit.
+
+How evenly the split lands also depends on the distribution of the key
+values. See [How a table is partitioned](#how-a-table-is-partitioned).
+
+Tables that don't meet these requirements, or whose boundary sampling fails
+for any reason, still snapshot correctly: each is read in full by a single
+worker, and different tables are still read concurrently.
+
+## How a table is partitioned
+
+Materialize partitions a table by the unique leading characters of its
+primary keys. Before reading the table, it probes the primary key index to
+discover key prefixes and uses the MySQL optimizer's row estimates to gauge
+how many rows fall under each one, extending prefixes until it finds
+boundaries that divide the table into roughly even ranges. The probes are
+inexpensive point lookups, capped in proportion to the table's estimated
+size, so this sampling phase stays negligible next to the snapshot itself.
+Each worker then reads only its assigned range, within the same consistent
+snapshot of the upstream database, so the result is identical to a
+single-worker snapshot, only faster.
+
+Because partitioning is based on key prefixes and optimizer estimates, how
+evenly the work divides depends on the shape of your keys:
+
+- **Evenly distributed keys partition well.** Keys whose leading characters
+  spread rows uniformly, such as UUIDs, hashes, or other randomized
+  identifiers, produce well-balanced ranges.
+
+- **Skewed keys partition less evenly.** If a large share of the table's rows
+  sort under a few common prefixes, some ranges end up with more rows than
+  others, and the workers assigned to them finish later.
+
+- **The probe budget can run out.** If finding even boundaries would require
+  examining very many distinct prefixes, Materialize stops probing and uses
+  the coarser boundaries found so far, which can also leave ranges uneven.
+
+Uneven partitioning is never incorrect. It only reduces the speedup, since
+the snapshot finishes when the busiest worker finishes.
+
+## MySQL-specific upstream considerations
+
+- **Connection count.** While the snapshot is being set up, Materialize
+  briefly holds up to two connections per worker, plus one. Once reading is
+  underway, this settles to one connection per worker reading a range, plus
+  one coordination connection. After the snapshot completes, the source drops
+  back to a single replication connection. If your MySQL server or connection
+  pooler enforces a low
+  [`max_connections`](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_connections)
+  limit, account for this burst when sizing it.
+
+- **Statistics freshness.** Range boundaries are placed using the MySQL
+  optimizer's row estimates. Stale statistics don't affect correctness, but
+  can skew how evenly work divides across workers. Running
+  [`ANALYZE TABLE`](https://dev.mysql.com/doc/refman/8.0/en/analyze-table.html)
+  on very large tables before creating the source can improve balance.
+
+For general guidance on read load, IOPS, and other upstream impact, which is
+not specific to MySQL, see [Is the upstream database
+overloaded?](/ingest-data/troubleshooting/#is-the-upstream-database-overloaded)
+
+## Observability
+
+The progress of an ongoing snapshot is visible in the
+[`mz_internal.mz_source_statistics`](/reference/system-catalog/mz_internal/#mz_source_statistics)
+system catalog view: `snapshot_records_known` is the estimated total size of
+the snapshot and `snapshot_records_staged` is how much of it has been read so
+far.

doc/user/content/ingest-data/troubleshooting.md

@@ -99,6 +99,32 @@ also be necessary to support increased memory usage during the process. For more
 information, see [Use a larger cluster for upsert source
 snapshotting](/ingest-data/#use-a-larger-cluster-for-upsert-source-snapshotting).
 
+## Is the upstream database overloaded?
+
+Snapshotting puts significant load on the upstream database (see [Impact on
+upstream system](/concepts/snapshotting/#impact-on-upstream-system)).
+
+Check the upstream database when a snapshot progresses more slowly than the
+data volume suggests, when applications sharing the database slow down while
+it runs, or when the source reports upstream connection errors or timeouts.
+The relevant metrics are in your cloud provider's monitoring console, or in
+OS tools like `iostat` and the database's activity views for self-hosted
+databases. Look for:
+
+- **CPU** pinned at the instance's limit for the duration of the snapshot.
+- **Read IOPS or throughput** flat at a provisioned cap while disk queue
+  depth and read latency climb.
+- **Network throughput** at the instance type's cap.
+- **Memory** pressure, or a falling cache hit rate as large scans evict the
+  normal workload's working set.
+- **Connections** near the database's limit. Snapshotting opens connections
+  in proportion to the source cluster's workers.
+
+If the database is overloaded, snapshot during off-peak hours, ingest from a
+read replica, use a smaller source cluster to spread the load over a longer
+window, [limit the volume of data](/ingest-data/#limit-the-volume-of-data)
+you sync, or provision more IOPS, throughput, or instance capacity.
+
 ## Adding a new subsource to an existing source blocks replication. Should I just create a new source instead?
 
 It depends. Materialize provides transactional guarantees for subsource of the