Skip to content

Commit

Permalink
docs: remove mentions of logical replication change streaming
Browse files Browse the repository at this point in the history
  • Loading branch information
icehaunter committed Jun 7, 2024
1 parent f2d97f4 commit 5aeb2d4
Show file tree
Hide file tree
Showing 14 changed files with 7 additions and 132 deletions.
83 changes: 0 additions & 83 deletions docs/api/service.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ Configuration options are passed as environment variables, e.g.:
```shell
docker run \
-e "DATABASE_URL=postgresql://..." \
-e "LOGICAL_PUBLISHER_HOST=..." \
-e "PG_PROXY_PASSWORD=..." \
-e "AUTH_JWT_ALG=HS512" \
-e "AUTH_JWT_KEY=..." \
Expand Down Expand Up @@ -132,88 +131,6 @@ Configure how Electric connects to Postgres and exposes its HTTP/WebSocket API.
<LogLevel />
</EnvVarConfig>

## Write-to-PG mode

Electric writes data to Postgres using one of two modes:

1. [logical replication](#logical-replication-mode)
2. [direct writes](#direct-writes-mode)

:::caution
The mode you choose affects your networking config and [database user permissions](#database-user-permissions).
:::

import ElectricWriteToPgMode from './_ELECTRIC_WRITE_TO_PG_MODE.md'

#### ELECTRIC_WRITE_TO_PG_MODE

<EnvVarConfig
name="ELECTRIC_WRITE_TO_PG_MODE"
defaultValue="logical_replication"
example="direct_writes">
<ElectricWriteToPgMode />
</EnvVarConfig>

### Logical replication mode

In `logical_replication` mode, Electric exposes a logical replication publisher service over TCP that speaks the [Logical Streaming Replication Protocol](https://www.postgresql.org/docs/current/protocol-logical-replication.html).

Postgres connects to Electric on `LOGICAL_PUBLISHER_HOST:LOGICAL_PUBLISHER_PORT` and establishes a logical replication subscription to this publisher service. Writes are then streamed in and applied using the logical replication subscription.

```
| <----------- DATABASE_URL ----------- |
Postgres | | Electric
| ---- LOGICAL_PUBLISHER_HOST:PORT ---> |
```

:::caution Caution
In logical replication mode:

1. the [database user](#database-user-permissions) that Electric connects to Postgres as must have the [`SUPERUSER` role attribute](https://www.postgresql.org/docs/16/role-attributes.html#ROLE-ATTRIBUTES)
2. Postgres must be able to connect to Electric (i.e.: must be able to establish an outbound TCP connection) on the host and port that you configure

As a result, you must make sure (in terms of networking / firewalls) not only that Postgres is reachable from Electric but also that Electric is reachable from Postgres. And Electric must know its own address, in order to provide it to Postgres when setting up the logical replication publication that allows writes to be replicated into Postgres.
:::

import LogicalPublisherHost from './_LOGICAL_PUBLISHER_HOST.md'
import LogicalPublisherPort from './_LOGICAL_PUBLISHER_PORT.md'

#### LOGICAL_PUBLISHER_HOST

<EnvVarConfig
name="LOGICAL_PUBLISHER_HOST"
required={true}
example="example.com">
<LogicalPublisherHost />
</EnvVarConfig>

#### LOGICAL_PUBLISHER_PORT

<EnvVarConfig
name="LOGICAL_PUBLISHER_PORT"
defaultValue="5433"
example="65433">
<LogicalPublisherPort />
</EnvVarConfig>

### Direct writes mode

In `direct_writes` mode, Electric writes data to Postgres using a standard interactive client connection. This avoids the need for Postgres to be able to connect to Electric and reduces the permissions required for the database user that Electric connects to Postgres as.

```
Postgres | <----------- DATABASE_URL ----------- | Electric
```

No additional configuration is required for direct writes mode. The `LOGICAL_PUBLISHER_HOST` and `LOGICAL_PUBLISHER_PORT` variables are not required and are ignored.

:::info Why are there two modes?
Originally (prior to v0.8) all writes to Postgres were made using logical replication.

In [version 0.8](/blog/2023/12/13/electricsql-v0.8-released#from-superuser-to-supabase), Electric added direct writes mode to reduce the database user permissions required and increase compatibility with Postgres hosting providers.

In future, we may deprecate logical replication and consolidate on direct writes. However, logical replication may have performance advantages and, for now, we've kept both modes available while direct writes mode gains maturity and we learn more about the operational characteristics of both modes.
:::

## Database user permissions

The Electric sync service connects to Postgres using the [`DATABASE_URL`](#database_url) connection string, in the format `postgresql://[userspec@][hostspec][/dbname]`.
Expand Down
10 changes: 0 additions & 10 deletions docs/deployment/_render.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,16 +51,6 @@ services:
value: "..."
- key: DATABASE_URL
value: "postgresql://..."
- key: LOGICAL_PUBLISHER_HOST
fromService:
type: web
name: tcp-proxy
property: host
- key: LOGICAL_PUBLISHER_PORT
fromService:
type: web
name: tcp-proxy
property: port
- type: web
name: http-proxy
runtime: image
Expand Down
13 changes: 3 additions & 10 deletions docs/deployment/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,18 +65,11 @@ Electric connects to Postgres using a `DATABASE_URL` connection string. In addit

Electric actually connects to Postgres in three ways:

1. it uses a normal interactive client connection to bootstrap resources and make database queries (for example, when you subscribe to a [Shape](../usage/data-access/shapes.md), Electric uses the interactive connection to query for all the rows in the shape)
2. it establishes a logical replication subscription to consume changes from Postgres
1. it uses a normal interactive client connection to bootstrap resources and make database queries (for example, when you subscribe to a [Shape](../usage/data-access/shapes.md), Electric uses the interactive connection to query for all the rows in the shape),
2. it establishes a logical replication subscription to consume changes from Postgres,
3. it writes changes to Postgres that it receives from clients over the [Satellite replication protocol](../api/satellite.md)

This last one, writes changes to Postgres, can be done in two ways:

- you can set `ELECTRIC_WRITE_TO_PG_MODE=direct_writes` to write data to Postgres using a normal interactive client connection
- you can set `ELECTRIC_WRITE_TO_PG_MODE=logical_replication` to stream data into Postgres using a logical replication publication

Direct writes mode works out of the box and is the easiest mode to configure, with no additional network connectivity required. Logical replication mode requires exposing a TCP host and port to Postgres, so that Postgres can establish a logical replication subscription. This can be configured using the `LOGICAL_PUBLISHER_HOST` and `LOGICAL_PUBLISHER_PORT` env vars.

See <DocPageLink path="api/service" /> for more information.
Writes are made over a regular connection with some additional configuration to distinguish client writes and direct-to-postgres writes in conflict resolution. See <DocPageLink path="api/service" /> for more information.

### 2. Migrations <-> Proxy

Expand Down
1 change: 0 additions & 1 deletion docs/deployment/digital-ocean.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,6 @@ services:
DATABASE_URL: "postgresql://postgres:******@example.com/postgres"
PG_PROXY_PASSWORD: "******"
AUTH_MODE: "insecure"
ELECTRIC_WRITE_TO_PG_MODE: "direct_writes"
ports:
- 80:5133
- 65432:65432
Expand Down
1 change: 0 additions & 1 deletion docs/deployment/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ See <DocPageLink path="usage/installation/service" /> to get familiar with confi
```shell
docker run \
-e "DATABASE_URL=postgresql://..." \
-e "LOGICAL_PUBLISHER_HOST=..." \
-e "PG_PROXY_PASSWORD=..." \
-e "AUTH_JWT_ALG=HS512" \
-e "AUTH_JWT_KEY=..." \
Expand Down
1 change: 0 additions & 1 deletion docs/deployment/fly.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ app = "electric-on-fly-test-app"
[env]
AUTH_MODE = "insecure"
DATABASE_URL = "postgresql://..."
ELECTRIC_WRITE_TO_PG_MODE = "direct_writes"
PG_PROXY_PASSWORD = "proxy_password"

# The main Internet-facing service of Electric
Expand Down
4 changes: 0 additions & 4 deletions docs/deployment/k8s.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,6 @@ spec:
env:
- name: DATABASE_URL
value: "postgresql://..."
- name: LOGICAL_PUBLISHER_HOST
value: "..."
- name: PG_PROXY_PASSWORD
value: "..."
- name: AUTH_JWT_ALG
Expand All @@ -46,8 +44,6 @@ spec:
ports:
- name: satellite-http
containerPort: 5133
- name: logical-pub-tcp
containerPort: 5433
- name: pg-proxy-tcp
containerPort: 65432
```
Expand Down
2 changes: 0 additions & 2 deletions docs/deployment/supabase.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,6 @@ Run your [Electric sync service](../api/service), either locally or [via one of
- set `AUTH_JWT_KEY` to the "JWT Secret" value you retrieved in step 3 above
- set `DATABASE_URL` to the connection string you retrieved in step 2 above
- set `DATABASE_USE_IPV6` to `true` unless you've purchased the IPv4 add-on for Supabase
- set `ELECTRIC_WRITE_TO_PG_MODE` to `direct_writes`
- set `PG_PROXY_PASSWORD` to a secure password value and note it down

Depending on how you run Electric these could be passed as arguments to Docker, set in a ENV file or entered into a managed host's dashboard. For example, to run Electric locally using Docker (replacing the `...` placeholder values with your specific values):
Expand All @@ -95,7 +94,6 @@ docker run \
-e "AUTH_JWT_KEY=..." \
-e "DATABASE_URL=..." \
-e "DATABASE_USE_IPV6=true" \
-e "ELECTRIC_WRITE_TO_PG_MODE=direct_writes" \
-e "PG_PROXY_PASSWORD=..." \
-e "PG_PROXY_PORT=65432" \
-p 5133:5133 \
Expand Down
8 changes: 2 additions & 6 deletions docs/examples/notes/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,10 @@ npm run dev

### Connecting Electric to Postgres

The Electric sync service connects to Postgres using the `DATABASE_URL` environment variable. Depending on your choice of write mode, Postgres may also need to connect to Electric to consume a logical replication publication.

This is configured using the `LOGICAL_PUBLISHER_HOST` (and `LOGICAL_PUBLISHER_PORT`) environment variables:
The Electric sync service connects to Postgres using the `DATABASE_URL` environment variable. Postgres is configured to use `logical` replication to stream changes to Electric.

```
|<--------DATABASE_URL----------|
Postgres | | Electric
|-----LOGICAL_PUBLISHER_HOST--->|
Postgres |<--------DATABASE_URL----------| Electric
```

See <DocPageLink path="api/service" /> for more information.
1 change: 0 additions & 1 deletion docs/quickstart/_setup_manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ docker pull electricsql/electric:latest
docker run \
-e "DATABASE_URL=postgresql://..." \
-e "DATABASE_REQUIRE_SSL=false \
-e "LOGICAL_PUBLISHER_HOST=..." \
-e "PG_PROXY_PASSWORD=..." \
-e "AUTH_MODE=insecure" \
-p 5133:5133 \
Expand Down
6 changes: 1 addition & 5 deletions docs/reference/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,11 +95,7 @@ Whilst local writes are final, they are still subject to concurrent merge semant

### Streaming into Electric

When the local database migrations are generated from the Postgres DDL changes, triggers are added that automatically copy insert, update and delete operations on the tables to the "oplog" table. The satellite process then processes these operations by sending them to the Electric server over the Satellite protocol. Electric then applies server-side validation and authorisation before sending on to Postgres over the incoming logical-replication stream.

:::note
Electric acts as a [logical replication publisher](https://www.postgresql.org/docs/current/logical-replication.html). This is why you configure a `LOGICAL_PUBLISHER_HOST` when deploying the Electric sync service -- so that Postgres can connect to consume inbound logical replication.
:::
When the local database migrations are generated from the Postgres DDL changes, triggers are added that automatically copy insert, update and delete operations on the tables to the "oplog" table. The satellite process then processes these operations by sending them to the Electric server over the Satellite protocol. Electric then applies server-side validation and authorisation before executing the writes on Postgres.

### Streaming into Postgres

Expand Down
7 changes: 1 addition & 6 deletions docs/usage/installation/service.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,13 @@ You can run the [pre-packaged Docker images](#images) published on Docker Hub, o

## Configuration

The Electric sync service is configured using environment variables. The three required variables are:
The Electric sync service is configured using environment variables. The two required variables are:

- `DATABASE_URL` in the format of a Postgres [Connection URI](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING-URIS)
- `LOGICAL_PUBLISHER_HOST` that the sync service is running on (must be accessible from the Postgres instance to establish an inbound replication subscription)
- `PG_PROXY_PASSWORD` to safe-guard access to the [Migrations proxy](../data-modelling/migrations.md#migrations-proxy)

```shell
DATABASE_URL="postgresql://user:password@localhost:5432/electric"
LOGICAL_PUBLISHER_HOST="localhost"
PG_PROXY_PASSWORD="..."
```

Expand All @@ -34,7 +32,6 @@ Pre-packaged images are available on Docker Hub at [electricsql/electric](https:
```shell
docker run \
-e "DATABASE_URL=postgresql://..." \
-e "LOGICAL_PUBLISHER_HOST=..." \
-e "PG_PROXY_PASSWORD=..." \
-e "AUTH_MODE=insecure" \
-p 5133:5133 \
Expand Down Expand Up @@ -75,7 +72,6 @@ services:
environment:
DATABASE_URL: postgresql://postgres:pg_password@pg/postgres
DATABASE_REQUIRE_SSL: false
LOGICAL_PUBLISHER_HOST: electric
PG_PROXY_PASSWORD: proxy_password
AUTH_MODE: insecure
ports:
Expand Down Expand Up @@ -103,7 +99,6 @@ Then run:
```shell
docker run \
-e "DATABASE_URL=postgresql://..." \
-e "LOGICAL_PUBLISHER_HOST=..." \
-e "PG_PROXY_PASSWORD=..." \
-e "AUTH_MODE=insecure" \
-p 5133:5133 \
Expand Down
1 change: 0 additions & 1 deletion examples/checkout/electric/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ services:
environment:
DATABASE_URL: postgresql://postgres:[email protected]:5432/postgres
PG_PROXY_PASSWORD: proxy_password
ELECTRIC_WRITE_TO_PG_MODE: direct_writes
AUTH_MODE: secure
AUTH_JWT_ALG: HS256
AUTH_JWT_KEY: XXXXXXXXXXX
Expand Down
1 change: 0 additions & 1 deletion examples/recipes/backend/docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ services:
environment:
DATABASE_URL: postgresql://postgres:pg_password@postgres:${PG_PORT:-5432}/${APP_NAME:-electric}
PG_PROXY_PASSWORD: proxy_password
LOGICAL_PUBLISHER_HOST: electric
AUTH_MODE: insecure
ports:
- ${ELECTRIC_PORT:-5133}:5133
Expand Down

0 comments on commit 5aeb2d4

Please sign in to comment.