-
Notifications
You must be signed in to change notification settings - Fork 152
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(electric): Add a new write-to-pg mode that applies changes directly to Postgres #698
Conversation
8ec69a7
to
c19fb5d
Compare
Curious what's the goal here? Lower latency? Higher throughput? |
The key thing this unlocks is being able to run with lower permissions. Currently we use inbound logical replication for writing data from Electric into Postgres. This adds an alternative direct writes mode that uses a standard interactive Postgres client. This reduces the permissions needed and, for example, means we can run on vanilla hosted Supabase. |
Both can be improved later on with this change in place. Direct writes allow us to batch multiple statements into one, keep a cache of prepared statements and generally be more aware of when client updates get committed in Postgres. We also get the freedom to "spread" client updates over a pool of connections, such that multiple updates are committed in parallel, whereas up until now we had to merged them into a single stream of transactions to be fed to Postgres as a stream logical replication messages. |
Interesting -- I was thinking under the assumption that direct writes would be slower, because you would wait for one transaction to be confirmed before sending the next? Is that true and then is it the case that that therefore reduces throughput but that this reduction may be mitigated and actually outweighed by the performance improvements from the techniques you listed? Is there something here in knowing when transactions can be written in parallel and is this something we already have enough metadata for? |
🔥 nice set of optimizations that are coming! |
124f0ca
to
5b29141
Compare
5b29141
to
0ac721e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks pretty good! I'm glad this functionality could be achieved in a relatively small code change.
I have multiple questions for particular lines, but also some general questions:
- Why are you calling it
inbound
replication everywhere? We're configuring the system from the POV of Electric, to which streaming to PG is outbound. I found this confusing and would appreciate this changed as I can see myself stumbling over this later on - Why are we keeping direct streaming at all? Perfomance comparison? Direct writes seem like a more generic approach from POV of deployment (also less configuration in general)
Great job all around!
...nsion/migrations/20230512000000_conflict_resolution_triggers/trigger_function_installers.sql
Outdated
Show resolved
Hide resolved
components/electric/lib/electric/replication/postgres/writer.ex
Outdated
Show resolved
Hide resolved
components/electric/lib/electric/replication/postgres/writer.ex
Outdated
Show resolved
Hide resolved
components/electric/lib/electric/replication/postgres/writer.ex
Outdated
Show resolved
Hide resolved
...onents/electric/lib/electric/postgres/extension/functions/__session_replication_role.sql.eex
Show resolved
Hide resolved
Hey, thanks for the review! @thruflo can answer those two general questions better than me but I'll give it a go:
|
Yup, as @alco says. Inbound derives from @balegas who I remember referenced MySQL naming. Which is database centric. Where @icehaunter is understandably thinking from Electric’s POV. I discussed with @balegas because I wanted us to be intentional about the naming and I didn’t think “immediate” made sense. If there’s a better term than inbound great. But I’d lean towards the Postgres-centric view if discussing inbound vs outbound. |
@thruflo @balegas @alco I would like to argue against using "inbound" here - we're configuring electric with this, not the Postgres, nor the entire stack. I believe it's bound to lead to more confusion, especially explicitly prefixed with |
I'm good with |
I'm happy with any decision. |
0ac721e
to
bf1b0d5
Compare
I believe that providing direct write performance comparable to logical replication is an easy baseline to achieve. Postgres supports query pipelining which means that even if we feed a single stream of changes, we don't have to wait for Postgres to response to every change before sending the next one. This already looks similar to the logical replication approach. On top of that we can add caching of prepared statements which implies encoding parameters using the binary format. This will result in fewer bytes sent over the wire and lower latency.
I may be speaking out of ignorance, but I think we have enough room for parallelization. At a very basic level, updates to disjoint sets of tables can be parallized. But even if two updates would conflict with each other, Postgres's MVCC should already be providing deterministic ordering of such concurrent updates, and our conflict-resolution logic will keep things consistent. |
bf1b0d5
to
34474d8
Compare
34474d8
to
b6fe6f9
Compare
The upsert_acknowledged_client_lsn trigger won't fire in the immediate write mode.
…ic writes This gives us control over which triggers fire regardless of whether writes are streamed to Postgres over a logical replication connection or applied directly as DML statements.
…cation_role option
b6fe6f9
to
93b2ba2
Compare
…BLE ALWAYS TRIGGER
Depending on whether proxy_opts are passed to it, we either open a regular connection to the singleton test db (current behaviour without proxy) or create a new temporary database and open a proxied connection to it. One of the reasons being the latest internal migration failing with a SyntaxError when applied via the proxy.
Using conditionals inside a loop may look weird but it actually provides more straightforward control flow compared to the previous version.
This follows the same approach that was implemented earlier in the conflict resolution trigger migration
…untime This is necessary for the new trigger templates to override the old ones in databases that already have the conflict resolution trigger migration applied.
This change removes the word "error" from the logs which was causing E2E tests to fail.
Note: I haven't implemented any specific error handling for the case where a client's update is rejected by Postgres for any reason. Opened VAX-1416 to address that at a later time. |
When Electric is started in this write mode, instead of creating a subscription in Postgres and streaming client writes to it over a logical replication connection, it will use a regular connection to apply client writes as regular DML statements.
The
direct_writes
mode can be enabled as follows when running Electric from source code:Alternatively, you can check out my branch
alco/direct-writes-to-pg-dev
, start the development database withmake start_dev_env
and start Electric with: