feat(electric): Add a new write-to-pg mode that applies changes directly to Postgres

[alco]

When Electric is started in this write mode, instead of creating a subscription in Postgres and streaming client writes to it over a logical replication connection, it will use a regular connection to apply client writes as regular DML statements.

The direct_writes mode can be enabled as follows when running Electric from source code:

MIX_ENV=prod \
ELECTRIC_WRITE_TO_PG_MODE=direct_writes \
PG_PROXY_PASSWORD=<pwd> \
DATABASE_URL=<url> \
iex -S mix

Alternatively, you can check out my branch alco/direct-writes-to-pg-dev, start the development database with make start_dev_env and start Electric with:

AUTH_MODE=insecure iex -S mix

[KyleAMathews]

Curious what's the goal here? Lower latency? Higher throughput?

[thruflo]

Curious what's the goal here? Lower latency? Higher throughput?

The key thing this unlocks is being able to run with lower permissions. Currently we use inbound logical replication for writing data from Electric into Postgres. This adds an alternative direct writes mode that uses a standard interactive Postgres client. This reduces the permissions needed and, for example, means we can run on vanilla hosted Supabase.

[alco]

Curious what's the goal here? Lower latency? Higher throughput?

Both can be improved later on with this change in place. Direct writes allow us to batch multiple statements into one, keep a cache of prepared statements and generally be more aware of when client updates get committed in Postgres. We also get the freedom to "spread" client updates over a pool of connections, such that multiple updates are committed in parallel, whereas up until now we had to merged them into a single stream of transactions to be fed to Postgres as a stream logical replication messages.

[thruflo]

Both can be improved later on with this change in place.

Interesting -- I was thinking under the assumption that direct writes would be slower, because you would wait for one transaction to be confirmed before sending the next? Is that true and then is it the case that that therefore reduces throughput but that this reduction may be mitigated and actually outweighed by the performance improvements from the techniques you listed?

Is there something here in knowing when transactions can be written in parallel and is this something we already have enough metadata for?

[KyleAMathews]

🔥 nice set of optimizations that are coming!

[icehaunter]

Looks pretty good! I'm glad this functionality could be achieved in a relatively small code change.

I have multiple questions for particular lines, but also some general questions:

1. Why are you calling it inbound replication everywhere? We're configuring the system from the POV of Electric, to which streaming to PG is outbound. I found this confusing and would appreciate this changed as I can see myself stumbling over this later on
2. Why are we keeping direct streaming at all? Perfomance comparison? Direct writes seem like a more generic approach from POV of deployment (also less configuration in general)

Great job all around!

[alco]

I have multiple questions for particular lines, but also some general questions:

Hey, thanks for the review! @thruflo can answer those two general questions better than me but I'll give it a go:

1. You can see from the commit history that I started with ELECTRIC_WRITE_MODE=streaming|immediate. The ELECTRIC_INBOUND_MODE name was James' idea, I believe. It's a reference to the act of applying inbound client updates via logical replication or direct writes to Postgres.
2. Again, not my idea, but it does let us keep existing users unaffected in case the new mode has edge cases we haven't uncovered yet. The new mode is operationally simpler and doesn't require a superuser role, so we will be making it the default mode soon enough, IMO. Then we'll be able to retire the code that deals with logical replication.

[thruflo]

Yup, as @alco says.

Inbound derives from @balegas who I remember referenced MySQL naming. Which is database centric. Where @icehaunter is understandably thinking from Electric’s POV.

I discussed with @balegas because I wanted us to be intentional about the naming and I didn’t think “immediate” made sense. If there’s a better term than inbound great. But I’d lean towards the Postgres-centric view if discussing inbound vs outbound.

[icehaunter]

@thruflo @balegas @alco I would like to argue against using "inbound" here - we're configuring electric with this, not the Postgres, nor the entire stack. I believe it's bound to lead to more confusion, especially explicitly prefixed with ELECTRIC_. Trying to "hide" Electric from the configuration seems like a decision that would lead to operational confusion and thus complexity in deployment. "Inbound" here is an ambiguous word. In my head it's definitely "outbound", but if that's also ambiguous, then maybe very explicitly ELECTRIC_WRITE_TO_PG_MODE?

[thruflo]

I'm good with ELECTRIC_WRITE_TO_PG_MODE.

[balegas]

I'm happy with any decision.
With regards to supporting both modes, let's keep things around until we learn more.

[alco]

Both can be improved later on with this change in place.

Interesting -- I was thinking under the assumption that direct writes would be slower, because you would wait for one transaction to be confirmed before sending the next? Is that true and then is it the case that that therefore reduces throughput but that this reduction may be mitigated and actually outweighed by the performance improvements from the techniques you listed?

I believe that providing direct write performance comparable to logical replication is an easy baseline to achieve. Postgres supports query pipelining which means that even if we feed a single stream of changes, we don't have to wait for Postgres to response to every change before sending the next one. This already looks similar to the logical replication approach. On top of that we can add caching of prepared statements which implies encoding parameters using the binary format. This will result in fewer bytes sent over the wire and lower latency.

Is there something here in knowing when transactions can be written in parallel and is this something we already have enough metadata for?

I may be speaking out of ignorance, but I think we have enough room for parallelization. At a very basic level, updates to disjoint sets of tables can be parallized. But even if two updates would conflict with each other, Postgres's MVCC should already be providing deterministic ordering of such concurrent updates, and our conflict-resolution logic will keep things consistent.

[alco]

Note: I haven't implemented any specific error handling for the case where a client's update is rejected by Postgres for any reason. Opened VAX-1416 to address that at a later time.