Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
c6cf863
DM-11841: Add cdc improved private link document.
ginkgoch Oct 17, 2025
cb217fc
DM-11841: Add sink endpoint to TOC.
ginkgoch Oct 17, 2025
e7df3ce
DM-11841: Document polish.
ginkgoch Oct 17, 2025
c8373cb
Update tidb-cloud/changefeed-sink-to-mysql.md
ginkgoch Oct 17, 2025
6176f8a
Update tidb-cloud/changefeed-sink-to-mysql.md
ginkgoch Oct 17, 2025
95129f8
Update tidb-cloud/changefeed-sink-to-mysql.md
ginkgoch Oct 17, 2025
000fc79
Update tidb-cloud/set-up-sink-private-endpoint.md
ginkgoch Oct 17, 2025
0c941dd
Update tidb-cloud/set-up-sink-private-endpoint.md
ginkgoch Oct 17, 2025
d4cddd1
DM-1184: Correct a role restriction.
ginkgoch Oct 17, 2025
d95c973
Add more info for domain.
ginkgoch Oct 17, 2025
00816bc
revise tidb-cloud/set-up-sink-private-endpoint.md
qiancai Oct 20, 2025
cafc654
Apply suggestions from code review
qiancai Oct 20, 2025
b33b50e
tidb-cloud/set-up-sink-private-endpoint.md: fix format issues
qiancai Oct 20, 2025
9ac73e9
tidb-cloud/set-up-sink-private-endpoint.md: revise wording
qiancai Oct 20, 2025
0c9d84e
tidb-cloud/set-up-sink-private-endpoint.md: remove unnecessary empty …
qiancai Oct 20, 2025
30b224c
tidb-cloud/changefeed-sink-to-apache-kafka.md: refine wording
qiancai Oct 20, 2025
bed99d1
tidb-cloud/changefeed-sink-to-mysql.md: revise changes
qiancai Oct 20, 2025
c37d346
update UI text according to UI
qiancai Oct 21, 2025
ffa3906
move the "Create Private Endpoint" line to step 2
qiancai Oct 21, 2025
6fedc1e
Apply suggestions from code review
qiancai Oct 21, 2025
5e31121
add an empty line
qiancai Oct 21, 2025
723af62
Apply suggestions from code review
qiancai Oct 21, 2025
e9cb1b7
Apply suggestions from code review
qiancai Oct 21, 2025
45dae5b
Update tidb-cloud/set-up-sink-private-endpoint.md
qiancai Oct 21, 2025
408472a
Merge branch 'release-8.5' into pr/21929
qiancai Oct 22, 2025
7211c4b
Merge branch 'dm-11841-cdc-private-link-improve' of https://github.co…
qiancai Oct 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 35 additions & 46 deletions tidb-cloud/changefeed-sink-to-apache-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,20 +54,17 @@ TiDB Cloud currently supports Private Connect only for self-hosted Kafka. It doe

- If your Apache Kafka service is hosted in AWS, follow [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md) to ensure that the network connection is properly configured. After setup, provide the following information in the TiDB Cloud console to create the changefeed:

- The ID in Kafka Advertised Listener Pattern
- The Endpoint Service Name
- The Sink Private Link, follow [Setup Private Endpoint for Changefeed](/tidb-cloud/set-up-sink-private-endpoint.md) to create one.
- The Bootstrap Ports

- If your Apache Kafka service is hosted in Google Cloud, follow [Set Up Self-Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md) to ensure that the network connection is properly configured. After setup, provide the following information in the TiDB Cloud console to create the changefeed:

- The ID in Kafka Advertised Listener Pattern
- The Service Attachment
- The Sink Private Link, follow [Setup Private Endpoint for Changefeed](/tidb-cloud/set-up-sink-private-endpoint.md) to create one.
- The Bootstrap Ports

- If your Apache Kafka service is hosted in Azure, follow [Set Up Self-Hosted Kafka Private Link Service in Azure](/tidb-cloud/setup-azure-self-hosted-kafka-private-link-service.md) to ensure that the network connection is properly configured. After setup, provide the following information in the TiDB Cloud console to create the changefeed:

- The ID in Kafka Advertised Listener Pattern
- The Alias of Private Link Service
- The Sink Private Link, follow [Setup Private Endpoint for Changefeed](/tidb-cloud/set-up-sink-private-endpoint.md) to create one.
- The Bootstrap Ports

</div>
Expand Down Expand Up @@ -139,63 +136,55 @@ The steps vary depending on the connectivity method you select.
<div label="Private Link (AWS)">

1. In **Connectivity Method**, select **Private Link**.
2. Authorize the [AWS Principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html#principal-accounts) of TiDB Cloud to create an endpoint for your endpoint service. The AWS Principal is provided in the tip on the web page.
3. Make sure you select the same **Number of AZs** and **AZ IDs of Kafka Deployment**, and fill the same unique ID in **Kafka Advertised Listener Pattern** when you [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md) in the **Network** section.
4. Fill in the **Endpoint Service Name** which is configured in [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md).
5. Fill in the **Bootstrap Ports**. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
6. Select an **Authentication** option according to your Kafka authentication configuration.
2. Select the **Sink Private Endpoint** which is prepared in the [Network](#network) section. Make sure the Sink Private Endpoint's AZs match the same AZ of the Kafka Deployment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

To follow the style guide's preference for active voice, it's better to rephrase this sentence.1 Also, 'match the same AZ' is slightly redundant and can be simplified.

Suggested change
2. Select the **Sink Private Endpoint** which is prepared in the [Network](#network) section. Make sure the Sink Private Endpoint's AZs match the same AZ of the Kafka Deployment.
2. Select the **Sink Private Endpoint** that you prepared in the [Network](#network) section. Make sure the Sink Private Endpoint's AZs match the AZs of your Kafka deployment.

Style Guide References

Footnotes

  1. Repository Style Guide, lines 22, 43-45: Write in second person ("you") and avoid passive voice.

3. Fill in the **Bootstrap Ports**. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
4. Select an **Authentication** option according to your Kafka authentication configuration.

- If your Kafka does not require authentication, keep the default option **Disable**.
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the **user name** and **password** of your Kafka account for authentication.

7. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
8. Select a **Compression** type for the data in this changefeed.
9. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
10. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
11. TiDB Cloud creates the endpoint for **Private Link**, which might take several minutes.
12. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
13. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
5. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
6. Select a **Compression** type for the data in this changefeed.
7. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
8. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
9. TiDB Cloud creates the endpoint for **Private Link**, which might take several minutes.
10. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
11. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.

</div>
<div label="Private Service Connect (Google Cloud)">

1. In **Connectivity Method**, select **Private Service Connect**.
2. Ensure that you fill in the same unique ID in **Kafka Advertised Listener Pattern** when you [Set Up Self-Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md) in the **Network** section.
3. Fill in the **Service Attachment** that you have configured in [Setup Self Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md)
4. Fill in the **Bootstrap Ports**. It is recommended that you provide more than one port. You can use commas `,` to separate multiple ports.
5. Select an **Authentication** option according to your Kafka authentication configuration.
2. Select the **Sink Private Endpoint** which is prepared in the [Network](#network) section.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

To follow the style guide's preference for active voice, it's better to rephrase 'which is prepared'.1

Suggested change
2. Select the **Sink Private Endpoint** which is prepared in the [Network](#network) section.
2. Select the **Sink Private Endpoint** that you prepared in the [Network](#network) section.

Style Guide References

Footnotes

  1. Repository Style Guide, lines 22, 43-45: Write in second person ("you") and avoid passive voice.

3. Fill in the **Bootstrap Ports**. It is recommended that you provide more than one port. You can use commas `,` to separate multiple ports.
4. Select an **Authentication** option according to your Kafka authentication configuration.

- If your Kafka does not require authentication, keep the default option **Disable**.
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the **user name** and **password** of your Kafka account for authentication.

6. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
7. Select a **Compression** type for the data in this changefeed.
8. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
9. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
10. TiDB Cloud creates the endpoint for **Private Service Connect**, which might take several minutes.
11. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
12. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
5. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
6. Select a **Compression** type for the data in this changefeed.
7. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
8. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
9. TiDB Cloud creates the endpoint for **Private Service Connect**, which might take several minutes.
10. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
11. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.

</div>
<div label="Private Link (Azure)">

1. In **Connectivity Method**, select **Private Link**.
2. Authorize the Azure subscription of TiDB Cloud or allow anyone with your alias to access your Private Link service before creating the changefeed. The Azure subscription is provided in the **Reminders before proceeding** tip on the web page. For more information about the visibility of Private Link service, see [Control service exposure](https://learn.microsoft.com/en-us/azure/private-link/private-link-service-overview#control-service-exposure) in Azure documentation.
3. Make sure you fill in the same unique ID in **Kafka Advertised Listener Pattern** when you [Set Up Self-Hosted Kafka Private Link Service in Azure](/tidb-cloud/setup-azure-self-hosted-kafka-private-link-service.md) in the **Network** section.
4. Fill in the **Alias of Private Link Service** which is configured in [Set Up Self-Hosted Kafka Private Link Service in Azure](/tidb-cloud/setup-azure-self-hosted-kafka-private-link-service.md).
5. Fill in the **Bootstrap Ports**. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
6. Select an **Authentication** option according to your Kafka authentication configuration.
2. Select the **Sink Private Endpoint** which is prepared in the [Network](#network) section.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

To follow the style guide's preference for active voice, it's better to rephrase 'which is prepared'.1

Suggested change
2. Select the **Sink Private Endpoint** which is prepared in the [Network](#network) section.
2. Select the **Sink Private Endpoint** that you prepared in the [Network](#network) section.

Style Guide References

Footnotes

  1. Repository Style Guide, lines 22, 43-45: Write in second person ("you") and avoid passive voice.

3. Fill in the **Bootstrap Ports**. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
4. Select an **Authentication** option according to your Kafka authentication configuration.

- If your Kafka does not require authentication, keep the default option **Disable**.
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the **user name** and **password** of your Kafka account for authentication.

7. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
8. Select a **Compression** type for the data in this changefeed.
9. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
10. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
11. TiDB Cloud creates the endpoint for **Private Link**, which might take several minutes.
12. Once the endpoint is created, log in to the [Azure portal](https://portal.azure.com/) and accept the connection request.
13. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
5. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
6. Select a **Compression** type for the data in this changefeed.
7. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
8. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
9. TiDB Cloud creates the endpoint for **Private Link**, which might take several minutes.
10. Once the endpoint is created, log in to the [Azure portal](https://portal.azure.com/) and accept the connection request.
11. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.

</div>
</SimpleTab>
Expand Down
40 changes: 30 additions & 10 deletions tidb-cloud/changefeed-sink-to-mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ Before creating a changefeed, you need to complete the following prerequisites:

Make sure that your TiDB Cluster can connect to the MySQL service.

<SimpleTab>
<div label="VPC Peering">

If your MySQL service is in an AWS VPC that has no public internet access, take the following steps:

1. [Set up a VPC peering connection](/tidb-cloud/set-up-vpc-peering-connections.md) between the VPC of the MySQL service and your TiDB cluster.
Expand All @@ -49,6 +52,18 @@ If your MySQL service is in a Google Cloud VPC that has no public internet acces
3. Modify the ingress firewall rules of the VPC where MySQL is located.

You must add [the CIDR of the region where your TiDB Cloud cluster is located](/tidb-cloud/set-up-vpc-peering-connections.md#prerequisite-set-a-cidr-for-a-region) to the ingress firewall rules. Doing so allows the traffic to flow from your TiDB Cluster to the MySQL endpoint.
</div>

<div label="Private Endpoint">
Private Connect leverages **Private Link** or **Private Service Connect** technologies from cloud providers to enable resources in your VPC to connect to services in other VPCs using private IP addresses, as if those services were hosted directly within your VPC.

- Ensure your MySQL service is connective though private endpoint. Provide the following information in the TiDB Cloud console to create the changefeed:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There's a typo here. 'connective though' should be 'connectable through'. Also, the sentence can be rephrased to be more direct and use active voice, as recommended by the style guide.1

Suggested change
- Ensure your MySQL service is connective though private endpoint. Provide the following information in the TiDB Cloud console to create the changefeed:
- To connect through a private endpoint, provide the following information in the TiDB Cloud console to create the changefeed:

Style Guide References

Footnotes

  1. Repository Style Guide, lines 22, 43-45: Write in second person ("you") and avoid passive voice.


- The Sink Private Link, follow [Setup Private Endpoint for Changefeed](/tidb-cloud/set-up-sink-private-endpoint.md) to create one.
- The MySQL Ports
</div>

</SimpleTab>

### Load existing data (optional)

Expand Down Expand Up @@ -95,21 +110,26 @@ After completing the prerequisites, you can sink your data to MySQL.

2. Click **Create Changefeed**, and select **MySQL** as **Destination**.

3. Fill in the MySQL endpoints, user name, and password in **MySQL Connection**.
3. In **Connectivity Method**:

* If using **VPC Peering** or **Public IP**, fill in your MySQL endpoint.
* If using **Private Endpoint**, select the **Sink Private Endpoint** which is prepared in the [Network](#network) section.

4. Fill in the MySQL endpoints, user name, and password in **MySQL Connection**.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Since the MySQL endpoint is already provided in step 3 for VPC Peering/Public IP, and a Sink Private Endpoint is used for the other option, asking for 'MySQL endpoints' again here is redundant and confusing. It's better to only ask for the credentials.

Suggested change
4. Fill in the MySQL endpoints, user name, and password in **MySQL Connection**.
4. Fill in the user name and password in **MySQL Connection**.


4. Click **Next** to test whether TiDB can connect to MySQL successfully:
5. Click **Next** to test whether TiDB can connect to MySQL successfully:

- If yes, you are directed to the next step of configuration.
- If not, a connectivity error is displayed, and you need to handle the error. After the error is resolved, click **Next** again.

5. Customize **Table Filter** to filter the tables that you want to replicate. For the rule syntax, refer to [table filter rules](/table-filter.md).
6. Customize **Table Filter** to filter the tables that you want to replicate. For the rule syntax, refer to [table filter rules](/table-filter.md).

- **Case Sensitive**: you can set whether the matching of database and table names in filter rules is case-sensitive. By default, matching is case-insensitive.
- **Filter Rules**: you can set filter rules in this column. By default, there is a rule `*.*`, which stands for replicating all tables. When you add a new rule, TiDB Cloud queries all the tables in TiDB and displays only the tables that match the rules in the box on the right. You can add up to 100 filter rules.
- **Tables with valid keys**: this column displays the tables that have valid keys, including primary keys or unique indexes.
- **Tables without valid keys**: this column shows tables that lack primary keys or unique keys. These tables present a challenge during replication because the absence of a unique identifier can result in inconsistent data when the downstream handles duplicate events. To ensure data consistency, it is recommended to add unique keys or primary keys to these tables before initiating the replication. Alternatively, you can add filter rules to exclude these tables. For example, you can exclude the table `test.tbl1` by using the rule `"!test.tbl1"`.

6. Customize **Event Filter** to filter the events that you want to replicate.
7. Customize **Event Filter** to filter the events that you want to replicate.

- **Tables matching**: you can set which tables the event filter will be applied to in this column. The rule syntax is the same as that used for the preceding **Table Filter** area. You can add up to 10 event filter rules per changefeed.
- **Event Filter**: you can use the following event filters to exclude specific events from the changefeed:
Expand All @@ -120,28 +140,28 @@ After completing the prerequisites, you can sink your data to MySQL.
- **Ignore update old value expression**: excludes `UPDATE` statements where the old value matches a specified condition. For example, `age < 18` excludes updates where the old value of `age` is less than 18.
- **Ignore delete value expression**: excludes `DELETE` statements that meet a specified condition. For example, `name = 'john'` excludes `DELETE` statements where `name` is `'john'`.

7. In **Start Replication Position**, configure the starting position for your MySQL sink.
8. In **Start Replication Position**, configure the starting position for your MySQL sink.

- If you have [loaded the existing data](#load-existing-data-optional) using Dumpling, select **Start replication from a specific TSO** and fill in the TSO that you get from Dumpling exported metadata files.
- If you do not have any data in the upstream TiDB cluster, select **Start replication from now on**.
- Otherwise, you can customize the start time point by choosing **Start replication from a specific time**.

8. Click **Next** to configure your changefeed specification.
9. Click **Next** to configure your changefeed specification.

- In the **Changefeed Specification** area, specify the number of Replication Capacity Units (RCUs) to be used by the changefeed.
- In the **Changefeed Name** area, specify a name for the changefeed.

9. Click **Next** to review the changefeed configuration.
10. Click **Next** to review the changefeed configuration.

If you confirm that all configurations are correct, check the compliance of cross-region replication, and click **Create**.

If you want to modify some configurations, click **Previous** to go back to the previous configuration page.

10. The sink starts soon, and you can see the status of the sink changes from **Creating** to **Running**.
11. The sink starts soon, and you can see the status of the sink changes from **Creating** to **Running**.

Click the changefeed name, and you can see more details about the changefeed, such as the checkpoint, replication latency, and other metrics.
Click the changefeed name, and you can see more details about the changefeed, such as the checkpoint, replication latency, and other metrics.

11. If you have [loaded the existing data](#load-existing-data-optional) using Dumpling, you need to restore the GC time to its original value (the default value is `10m`) after the sink is created:
12. If you have [loaded the existing data](#load-existing-data-optional) using Dumpling, you need to restore the GC time to its original value (the default value is `10m`) after the sink is created:

{{< copyable "sql" >}}

Expand Down
Loading
Loading