Skip to content

Add cdc improved private link document. #21929

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Oct 22, 2025
Merged

Add cdc improved private link document. #21929

Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
c6cf863
DM-11841: Add cdc improved private link document.
ginkgoch Oct 17, 2025
cb217fc
DM-11841: Add sink endpoint to TOC.
ginkgoch Oct 17, 2025
e7df3ce
DM-11841: Document polish.
ginkgoch Oct 17, 2025
c8373cb
Update tidb-cloud/changefeed-sink-to-mysql.md
ginkgoch Oct 17, 2025
6176f8a
Update tidb-cloud/changefeed-sink-to-mysql.md
ginkgoch Oct 17, 2025
95129f8
Update tidb-cloud/changefeed-sink-to-mysql.md
ginkgoch Oct 17, 2025
000fc79
Update tidb-cloud/set-up-sink-private-endpoint.md
ginkgoch Oct 17, 2025
0c941dd
Update tidb-cloud/set-up-sink-private-endpoint.md
ginkgoch Oct 17, 2025
d4cddd1
DM-1184: Correct a role restriction.
ginkgoch Oct 17, 2025
d95c973
Add more info for domain.
ginkgoch Oct 17, 2025
00816bc
revise tidb-cloud/set-up-sink-private-endpoint.md
qiancai Oct 20, 2025
cafc654
Apply suggestions from code review
qiancai Oct 20, 2025
b33b50e
tidb-cloud/set-up-sink-private-endpoint.md: fix format issues
qiancai Oct 20, 2025
9ac73e9
tidb-cloud/set-up-sink-private-endpoint.md: revise wording
qiancai Oct 20, 2025
0c9d84e
tidb-cloud/set-up-sink-private-endpoint.md: remove unnecessary empty …
qiancai Oct 20, 2025
30b224c
tidb-cloud/changefeed-sink-to-apache-kafka.md: refine wording
qiancai Oct 20, 2025
bed99d1
tidb-cloud/changefeed-sink-to-mysql.md: revise changes
qiancai Oct 20, 2025
c37d346
update UI text according to UI
qiancai Oct 21, 2025
ffa3906
move the "Create Private Endpoint" line to step 2
qiancai Oct 21, 2025
6fedc1e
Apply suggestions from code review
qiancai Oct 21, 2025
5e31121
add an empty line
qiancai Oct 21, 2025
723af62
Apply suggestions from code review
qiancai Oct 21, 2025
e9cb1b7
Apply suggestions from code review
qiancai Oct 21, 2025
45dae5b
Update tidb-cloud/set-up-sink-private-endpoint.md
qiancai Oct 21, 2025
408472a
Merge branch 'release-8.5' into pr/21929
qiancai Oct 22, 2025
7211c4b
Merge branch 'dm-11841-cdc-private-link-improve' of https://github.co…
qiancai Oct 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions TOC-tidb-cloud.md
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doc preview: https://pr.pingcap-docsite-preview.pages.dev/tidbcloud/set-up-sink-private-endpoint/

Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,7 @@
- [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md)
- [Set Up Self-Hosted Kafka Private Link Service in Azure](/tidb-cloud/setup-azure-self-hosted-kafka-private-link-service.md)
- [Set Up Self-Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md)
- [Set Up Private Endpoint for Changefeeds](/tidb-cloud/set-up-sink-private-endpoint.md)
- Disaster Recovery
- [Recovery Group Overview](/tidb-cloud/recovery-group-overview.md)
- [Get Started](/tidb-cloud/recovery-group-get-started.md)
Expand Down
88 changes: 33 additions & 55 deletions tidb-cloud/changefeed-sink-to-apache-kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,23 +52,9 @@ Private Connect leverages **Private Link** or **Private Service Connect** techno

TiDB Cloud currently supports Private Connect only for self-hosted Kafka. It does not support direct integration with MSK, Confluent Kafka, or other Kafka SaaS services. To connect to these Kafka SaaS services via Private Connect, you can deploy a [kafka-proxy](https://github.com/grepplabs/kafka-proxy) as an intermediary, effectively exposing the Kafka service as self-hosted Kafka. For a detailed example, see [Set Up Self-Hosted Kafka Private Service Connect by Kafka-proxy in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md#set-up-self-hosted-kafka-private-service-connect-by-kafka-proxy). This setup is similar across all Kafka SaaS services.

- If your Apache Kafka service is hosted in AWS, follow [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md) to ensure that the network connection is properly configured. After setup, provide the following information in the TiDB Cloud console to create the changefeed:

- The ID in Kafka Advertised Listener Pattern
- The Endpoint Service Name
- The Bootstrap Ports

- If your Apache Kafka service is hosted in Google Cloud, follow [Set Up Self-Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md) to ensure that the network connection is properly configured. After setup, provide the following information in the TiDB Cloud console to create the changefeed:

- The ID in Kafka Advertised Listener Pattern
- The Service Attachment
- The Bootstrap Ports

- If your Apache Kafka service is hosted in Azure, follow [Set Up Self-Hosted Kafka Private Link Service in Azure](/tidb-cloud/setup-azure-self-hosted-kafka-private-link-service.md) to ensure that the network connection is properly configured. After setup, provide the following information in the TiDB Cloud console to create the changefeed:

- The ID in Kafka Advertised Listener Pattern
- The Alias of Private Link Service
- The Bootstrap Ports
- If your Apache Kafka service is hosted on AWS, follow [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md) to configure the network connection and obtain the **Bootstrap Ports** information, and then follow [Set Up Private Endpoint for Changefeeds](/tidb-cloud/set-up-sink-private-endpoint.md) to create a private endpoint.
- If your Apache Kafka service is hosted on Google Cloud, follow [Set Up Self-Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md) to configure the network connection and obtain the **Bootstrap Ports** information, and then follow [Set Up Private Endpoint for Changefeeds](/tidb-cloud/set-up-sink-private-endpoint.md) to create a private endpoint.
- If your Apache Kafka service is hosted on Azure, follow [Set Up Self-Hosted Kafka Private Link Service in Azure](/tidb-cloud/setup-azure-self-hosted-kafka-private-link-service.md) to configure the network connection and obtain the **Bootstrap Ports** information, and then follow [Set Up Private Endpoint for Changefeeds](/tidb-cloud/set-up-sink-private-endpoint.md) to create a private endpoint.

</div>
<div label="VPC Peering">
Expand Down Expand Up @@ -139,63 +125,55 @@ The steps vary depending on the connectivity method you select.
<div label="Private Link (AWS)">

1. In **Connectivity Method**, select **Private Link**.
2. Authorize the [AWS Principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html#principal-accounts) of TiDB Cloud to create an endpoint for your endpoint service. The AWS Principal is provided in the tip on the web page.
3. Make sure you select the same **Number of AZs** and **AZ IDs of Kafka Deployment**, and fill the same unique ID in **Kafka Advertised Listener Pattern** when you [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md) in the **Network** section.
4. Fill in the **Endpoint Service Name** which is configured in [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md).
5. Fill in the **Bootstrap Ports**. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
6. Select an **Authentication** option according to your Kafka authentication configuration.
2. In **Private Endpoint**, select the private endpoint that you created in the [Network](#network) section. Make sure the AZs of the private endpoint match the AZs of the Kafka deployment.
3. Fill in the **Bootstrap Ports** that you obtained from the [Network](#network) section. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
4. Select an **Authentication** option according to your Kafka authentication configuration.

- If your Kafka does not require authentication, keep the default option **Disable**.
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the **user name** and **password** of your Kafka account for authentication.

7. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
8. Select a **Compression** type for the data in this changefeed.
9. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
10. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
11. TiDB Cloud creates the endpoint for **Private Link**, which might take several minutes.
12. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
13. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
5. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
6. Select a **Compression** type for the data in this changefeed.
7. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
8. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
9. TiDB Cloud creates the endpoint for **Private Link**, which might take several minutes.
10. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
11. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.

</div>
<div label="Private Service Connect (Google Cloud)">

1. In **Connectivity Method**, select **Private Service Connect**.
2. Ensure that you fill in the same unique ID in **Kafka Advertised Listener Pattern** when you [Set Up Self-Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md) in the **Network** section.
3. Fill in the **Service Attachment** that you have configured in [Setup Self Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md)
4. Fill in the **Bootstrap Ports**. It is recommended that you provide more than one port. You can use commas `,` to separate multiple ports.
5. Select an **Authentication** option according to your Kafka authentication configuration.
2. In **Private Endpoint**, select the private endpoint that you created in the [Network](#network) section.
3. Fill in the **Bootstrap Ports** that you obtained from the [Network](#network) section. It is recommended that you provide more than one port. You can use commas `,` to separate multiple ports.
4. Select an **Authentication** option according to your Kafka authentication configuration.

- If your Kafka does not require authentication, keep the default option **Disable**.
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the **user name** and **password** of your Kafka account for authentication.

6. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
7. Select a **Compression** type for the data in this changefeed.
8. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
9. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
10. TiDB Cloud creates the endpoint for **Private Service Connect**, which might take several minutes.
11. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
12. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
5. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
6. Select a **Compression** type for the data in this changefeed.
7. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
8. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
9. TiDB Cloud creates the endpoint for **Private Service Connect**, which might take several minutes.
10. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
11. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.

</div>
<div label="Private Link (Azure)">

1. In **Connectivity Method**, select **Private Link**.
2. Authorize the Azure subscription of TiDB Cloud or allow anyone with your alias to access your Private Link service before creating the changefeed. The Azure subscription is provided in the **Reminders before proceeding** tip on the web page. For more information about the visibility of Private Link service, see [Control service exposure](https://learn.microsoft.com/en-us/azure/private-link/private-link-service-overview#control-service-exposure) in Azure documentation.
3. Make sure you fill in the same unique ID in **Kafka Advertised Listener Pattern** when you [Set Up Self-Hosted Kafka Private Link Service in Azure](/tidb-cloud/setup-azure-self-hosted-kafka-private-link-service.md) in the **Network** section.
4. Fill in the **Alias of Private Link Service** which is configured in [Set Up Self-Hosted Kafka Private Link Service in Azure](/tidb-cloud/setup-azure-self-hosted-kafka-private-link-service.md).
5. Fill in the **Bootstrap Ports**. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
6. Select an **Authentication** option according to your Kafka authentication configuration.
2. In **Private Endpoint**, select the private endpoint that you created in the [Network](#network) section.
3. Fill in the **Bootstrap Ports** that you obtained in the [Network](#network) section. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
4. Select an **Authentication** option according to your Kafka authentication configuration.

- If your Kafka does not require authentication, keep the default option **Disable**.
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the **user name** and **password** of your Kafka account for authentication.

7. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
8. Select a **Compression** type for the data in this changefeed.
9. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
10. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
11. TiDB Cloud creates the endpoint for **Private Link**, which might take several minutes.
12. Once the endpoint is created, log in to the [Azure portal](https://portal.azure.com/) and accept the connection request.
13. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
5. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
6. Select a **Compression** type for the data in this changefeed.
7. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
8. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
9. TiDB Cloud creates the endpoint for **Private Link**, which might take several minutes.
10. Once the endpoint is created, log in to the [Azure portal](https://portal.azure.com/) and accept the connection request.
11. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.

</div>
</SimpleTab>
Expand Down
Loading
Loading