docs: separate ingest data step w some single-sourcing (rough-cut) (#33926)

Author: kay-kim

doc/user/config.toml

@@ -67,7 +67,7 @@ weight = 12
 [[menu.main]]
 identifier = "reference"
 name = "Reference"
-weight = 15
+weight = 80
 
 [[menu.main]]
 name = "SQL commands"
@@ -83,7 +83,7 @@ weight = 100
 [[menu.main]]
 identifier = "releases-previews"
 name = "Releases"
-weight = 65
+weight = 80
 
 [[menu.main]]
 identifier = "changelogs"
@@ -99,10 +99,10 @@ url = "https://materialize.com/changelog/"
 [[menu.main]]
 identifier = "about"
 name = "About"
-weight = 70
+weight = 90
 
 [[menu.main]]
-name = "Security overview"
+name = "Materialize Cloud Security and Compliance"
 parent = "about"
 url = "https://materialize.com/security-overview"
 weight = 25

doc/user/content/ingest-data/cdc-cockroachdb.md

@@ -89,12 +89,8 @@ scenarios, we recommend separating your workloads into multiple clusters for
 
 {{% kafka/cockroachdb/create-a-cluster %}}
 
-### 2. Start ingesting data
+### 2. Create a connection
 
-[//]: # "TODO(morsapaes) Incorporate all options for network security and
-authentication later on. Starting with a simplified version that is consistent
-with the PostgreSQL/MySQL guides and can be used to model other Kafka-relate
-integration guides after."
 
 Now that you've created an ingestion cluster, you can connect Materialize to
 your Kafka broker and start ingesting data. The exact steps depend on your
@@ -137,6 +133,8 @@ guidance.
     );
     ```
 
+## 3. Start ingesting data
+
 1. Use the [`CREATE SOURCE`](/sql/create-source/) command to connect Materialize
    to your Kafka broker and start ingesting data from the target topic:
 
@@ -153,15 +151,15 @@ guidance.
     By default, the source will be created in the active cluster; to use a
     different cluster, use the `IN CLUSTER` clause.
 
-### 3. Monitor the ingestion status
+### 4. Monitor the ingestion status
 
 {{% kafka/cockroachdb/check-the-ingestion-status %}}
 
-### 4. Create a view
+### 5. Create a view
 
 {{% ingest-data/ingest-data-kafka-debezium-view %}}
 
-### 5. Create an index on the view
+### 6. Create an index on the view
 
 {{% ingest-data/ingest-data-kafka-debezium-index %}}
 

doc/user/content/ingest-data/mysql/amazon-aurora.md

@@ -282,7 +282,7 @@ configuration of resources for an SSH tunnel. For more details, see the
 
 ## C. Ingest data in Materialize
 
-### 1. (Optional) Create a cluster
+### 1. (Optional) Create a source cluster
 
 {{< note >}}
 If you are prototyping and already have a cluster to host your MySQL
@@ -293,14 +293,10 @@ scenarios, we recommend separating your workloads into multiple clusters for
 
 {{% mysql-direct/create-a-cluster %}}
 
-### 2. Start ingesting data
+### 2. Create a connection
 
-[//]: # "TODO(morsapaes) MySQL connections support multiple SSL modes. We should
-adapt to that, rather than just state SSL MODE REQUIRED."
-
-Now that you've configured your database network, you can connect Materialize to
-your MySQL database and start ingesting data. The exact steps depend on your
-networking configuration, so start by selecting the relevant option.
+Once you have configured your network, create a connection in Materialize per
+your networking configuration.
 
 {{< tabs >}}
 
@@ -318,15 +314,23 @@ networking configuration, so start by selecting the relevant option.
 
 {{< /tabs >}}
 
+### 3. Start ingesting data
+
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="create-source" %}}
+
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="create-source-options" %}}
+
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="schema-changes" %}}
+
 [//]: # "TODO(morsapaes) Replace these Step 6. and 7. with guidance using the
 new progress metrics in mz_source_statistics + console monitoring, when
 available (also for PostgreSQL)."
 
-### 3. Monitor the ingestion status
+### 4. Monitor the ingestion status
 
 {{% mysql-direct/check-the-ingestion-status %}}
 
-### 4. Right-size the cluster
+### 5. Right-size the cluster
 
 {{% mysql-direct/right-size-the-cluster %}}
 

doc/user/content/ingest-data/mysql/amazon-rds.md

@@ -304,15 +304,13 @@ scenarios, we recommend separating your workloads into multiple clusters for
 
 {{% mysql-direct/create-a-cluster %}}
 
-### 2. Start ingesting data
+### 2. Create a connection
 
 [//]: # "TODO(morsapaes) MySQL connections support multiple SSL modes. We should
 adapt to that, rather than just state SSL MODE REQUIRED."
 
-Now that you've configured your database network and created an ingestion
-cluster, you can connect Materialize to your MySQL database and start
-ingesting data. The exact steps depend on your networking configuration, so
-start by selecting the relevant option.
+Once you have configured your network, create a connection in Materialize per
+your networking configuration.
 
 {{< tabs >}}
 
@@ -330,15 +328,23 @@ start by selecting the relevant option.
 
 {{< /tabs >}}
 
+### 3. Start ingesting data
+
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="create-source" %}}
+
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="create-source-options" %}}
+
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="schema-changes" %}}
+
 [//]: # "TODO(morsapaes) Replace these Step 6. and 7. with guidance using the
 new progress metrics in mz_source_statistics + console monitoring, when
 available (also for PostgreSQL)."
 
-### 3. Monitor the ingestion status
+### 4. Monitor the ingestion status
 
 {{% mysql-direct/check-the-ingestion-status %}}
 
-### 4. Right-size the cluster
+### 5. Right-size the cluster
 
 {{% mysql-direct/right-size-the-cluster %}}
 

doc/user/content/ingest-data/mysql/azure-db.md

@@ -128,14 +128,13 @@ scenarios, we recommend separating your workloads into multiple clusters for
 
 {{% mysql-direct/create-a-cluster %}}
 
-### 2. Start ingesting data
+### 2. Create a connection
 
 [//]: # "TODO(morsapaes) MySQL connections support multiple SSL modes. We should
 adapt to that, rather than just state SSL MODE REQUIRED."
 
-Now that you've configured your database network, you can connect Materialize to
-your MySQL database and start ingesting data. The exact steps depend on your
-networking configuration, so start by selecting the relevant option.
+Once you have configured your network, create a connection in Materialize per
+your networking configuration.
 
 {{< tabs >}}
 
@@ -149,15 +148,19 @@ networking configuration, so start by selecting the relevant option.
 
 {{< /tabs >}}
 
-[//]: # "TODO(morsapaes) Replace these Step 6. and 7. with guidance using the
-new progress metrics in mz_source_statistics + console monitoring, when
-available(also for PostgreSQL)."
+### 3. Start ingesting data
 
-### 3. Monitor the ingestion status
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="create-source" %}}
+
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="create-source-options" %}}
+
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="schema-changes" %}}
+
+### 4. Monitor the ingestion status
 
 {{% mysql-direct/check-the-ingestion-status %}}
 
-### 4. Right-size the cluster
+### 5. Right-size the cluster
 
 {{% mysql-direct/right-size-the-cluster %}}
 

doc/user/content/ingest-data/mysql/google-cloud-sql.md

@@ -122,14 +122,13 @@ scenarios, we recommend separating your workloads into multiple clusters for
 
 {{% mysql-direct/create-a-cluster %}}
 
-### 2. Start ingesting data
+### 2. Create a connection
 
 [//]: # "TODO(morsapaes) MySQL connections support multiple SSL modes. We should
 adapt to that, rather than just state SSL MODE REQUIRED."
 
-Now that you've configured your database network, you can connect Materialize to
-your MySQL database and start ingesting data. The exact steps depend on your
-networking configuration, so start by selecting the relevant option.
+Once you have configured your network, create a connection in Materialize per
+your networking configuration.
 
 {{< tabs >}}
 
@@ -143,11 +142,19 @@ networking configuration, so start by selecting the relevant option.
 
 {{< /tabs >}}
 
-### 3. Monitor the ingestion status
+### 3. Start ingesting data
+
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="create-source" %}}
+
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="create-source-options" %}}
+
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="schema-changes" %}}
+
+### 4. Monitor the ingestion status
 
 {{% mysql-direct/check-the-ingestion-status %}}
 
-### 4. Right-size the cluster
+### 5. Right-size the cluster
 
 {{% mysql-direct/right-size-the-cluster %}}
 

doc/user/content/ingest-data/mysql/self-hosted.md

@@ -122,11 +122,10 @@ scenarios, we recommend separating your workloads into multiple clusters for
 
 {{% mysql-direct/create-a-cluster %}}
 
-### 2. Start ingesting data
+### 2. Create a connection
 
-Now that you've configured your database network, you can connect Materialize to
-your MySQL database and start ingesting data. The exact steps depend on your
-networking configuration, so start by selecting the relevant option.
+Once you have configured your network, create a connection in Materialize per
+your networking configuration.
 
 {{< tabs >}}
 
@@ -140,16 +139,19 @@ networking configuration, so start by selecting the relevant option.
 
 {{< /tabs >}}
 
+### 3. Start ingesting data
 
-[//]: # "TODO(morsapaes) Replace these Step 6. and 7. with guidance using the
-new progress metrics in mz_source_statistics + console monitoring, when
-available(also for PostgreSQL)."
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="create-source" %}}
 
-### 3. Monitor the ingestion status
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="create-source-options" %}}
+
+{{% include-example file="examples/ingest_data/mysql/create_source_cloud" example="schema-changes" %}}
+
+### 4. Monitor the ingestion status
 
 {{% mysql-direct/check-the-ingestion-status %}}
 
-### 4. Right-size the cluster
+### 5. Right-size the cluster
 
 {{% mysql-direct/right-size-the-cluster %}}
 

doc/user/content/ingest-data/postgres/alloydb.md

@@ -132,28 +132,80 @@ scenarios, we recommend separating your workloads into multiple clusters for
 
 {{% postgres-direct/create-a-cluster %}}
 
-### 2. Start ingesting data
+### 2. Create a connection
 
-With the network configured and an ingestion pipeline in place, connect
-Materialize to your AlloyDB instance and begin the data ingestion process.
+Once you have configured your network, create a connection in Materialize per
+your networking configuration.
 
 {{< tabs >}}
 
 {{< tab "Allow Materialize IPs">}}
-{{% postgres-direct/ingesting-data/allow-materialize-ips %}}
+
+1. {{% include-example
+   file="examples/ingest_data/postgres/create_connection_ips_cloud"
+   example="create-secret" indent="true" %}}
+
+1. {{% include-example
+   file="examples/ingest_data/postgres/create_connection_ips_cloud"
+   example="create-connection" indent="true" %}}
+
+   {{% include-example
+   file="examples/ingest_data/postgres/create_connection_ips_cloud"
+   example="create-connection-options-general" indent="true" %}}
+
 {{< /tab >}}
 
 {{< tab "Use an SSH tunnel">}}
-{{% postgres-direct/ingesting-data/use-ssh-tunnel %}}
+
+1. {{% include-example
+   file="examples/ingest_data/postgres/create_connection_ssh_cloud"
+   example="create-ssh-tunnel-connection" indent="true" %}}
+
+   {{% include-example
+   file="examples/ingest_data/postgres/create_connection_ssh_cloud"
+   example="create-ssh-tunnel-connection-options" indent="true" %}}
+
+1. {{% include-example
+   file="examples/ingest_data/postgres/create_connection_ssh_cloud"
+   example="get-public-keys-general" indent="true" %}}
+
+1. {{% include-example
+   file="examples/ingest_data/postgres/create_connection_ssh_cloud"
+   example="login-to-ssh-bastion-host" indent="true" %}}
+
+1. {{% include-example
+   file="examples/ingest_data/postgres/create_connection_ssh_cloud"
+   example="validate-ssh-tunnel-connection" indent="true" %}}
+
+1. {{% include-example
+   file="examples/ingest_data/postgres/create_connection_ssh_cloud"
+   example="create-secret" indent="true" %}}
+
+1. {{% include-example
+   file="examples/ingest_data/postgres/create_connection_ssh_cloud"
+   example="create-connection" indent="true" %}}
+
+   {{% include-example
+   file="examples/ingest_data/postgres/create_connection_ssh_cloud"
+   example="create-connection-options-general" indent="true" %}}
 {{< /tab >}}
 
 {{< /tabs >}}
 
-### 3. Monitor the ingestion status
+### 3. Start ingesting data
+
+{{% include-example file="examples/ingest_data/postgres/create_source_cloud" example="create-source" %}}
+
+{{% include-example file="examples/ingest_data/postgres/create_source_cloud" example="create-source-options" %}}
+
+{{% include-example file="examples/ingest_data/postgres/create_source_cloud"
+example="schema-changes" %}}
+
+### 4. Monitor the ingestion status
 
 {{% postgres-direct/check-the-ingestion-status %}}
 
-### 4. Right-size the cluster
+### 5. Right-size the cluster
 
 {{% postgres-direct/right-size-the-cluster %}}