Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Zero-ETL for PG #199

Merged
merged 24 commits into from
Nov 26, 2024
Merged

feat: Zero-ETL for PG #199

merged 24 commits into from
Nov 26, 2024

Conversation

TianyuZhang1214
Copy link
Contributor

@TianyuZhang1214 TianyuZhang1214 commented Nov 22, 2024

Resolves: #176, #119

Features Added:

1. CREATE SUBSCRIPTION Support

This PR introduces support for the CREATE SUBSCRIPTION SQL command in MyDuck Server. Users can now set up replication from a source PostgreSQL instance with the following SQL statement:

CREATE SUBSCRIPTION mysub CONNECTION 'dbname=foo host=bar user=repuser' PUBLICATION mypub;

Automatic Slot and Publication Creation:
If the required replication slot and publication are not already present on the source PostgreSQL server, they will be created automatically during the execution of the CREATE SUBSCRIPTION command.


2. Replica Mode with Docker

Added support for running MyDuck Server in replica mode, enabling seamless replication from a source PostgreSQL database at launch.

Usage Example:
The following docker run command demonstrates how to launch MyDuck Server in replica mode:

docker run \
  -p 3306:3306 \
  -p 5432:5432 \
  --privileged \
  --detach \
  --env SETUP_MODE=REPLICA \
  --env SOURCE_DSN="postgres://postgres:[email protected]:15432" \
  apecloud/myduckserver:latest
  • SETUP_MODE=REPLICA: Configures MyDuck Server to function as a replica.
  • SOURCE_DSN: Specifies the source PostgreSQL connection string.

This simplifies the process of setting up replication directly from a PostgreSQL database.

@TianyuZhang1214 TianyuZhang1214 marked this pull request as ready for review November 25, 2024 10:57
Copy link
Collaborator

@fanyang01 fanyang01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks exciting! This PR greatly simplifies the experience of Postgres replica setup.

A few comments:

README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
devtools/replica-setup-mysql/checker.sh Outdated Show resolved Hide resolved
pgserver/subscription_handler.go Outdated Show resolved Hide resolved
pgserver/subscription_handler.go Show resolved Hide resolved
Copy link
Contributor

@VWagen1989 VWagen1989 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really an exciting feature that provisions an easy way to create a replication stream to PostgreSQL. It makes things simple, and as simple as possible is a good taste!
Here's my review comments.

pgserver/connection_handler.go Outdated Show resolved Hide resolved
pgserver/subscription_handler.go Outdated Show resolved Hide resolved
pgserver/subscription_handler.go Outdated Show resolved Hide resolved
pgserver/subscription_handler.go Show resolved Hide resolved
Copy link
Collaborator

@fanyang01 fanyang01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! LGTM

Copy link
Contributor

@VWagen1989 VWagen1989 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@TianyuZhang1214 TianyuZhang1214 merged commit 95af164 into main Nov 26, 2024
5 of 6 checks passed
@TianyuZhang1214 TianyuZhang1214 deleted the 119-zero-etl-pg branch November 26, 2024 03:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants