Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: remove mentions of logical replication change streaming #1343

Open
wants to merge 1 commit into
base: ilia/vax-1920-direct-writes-only
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 0 additions & 22 deletions components/electric/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,28 +149,6 @@ COMMIT;

Note that this procedure **MUST** be called within the same transaction as the migration.

## OSX

Note that if, when running on OSX, you get errors like:

```
could not connect to the publisher: connection to server at \"host.docker.internal\" (192.168.65.2), port 5433 failed
```

You may need to adjust your docker networking or run Electric within docker. To run within Docker, you can build the docker image locally:

```sh
make docker-build
```

And then run with the right env vars, e.g.:

```sh
docker run -it -p "5433:5433" -p "5133:5133" \
-e "DATABASE_URL=postgresql://electric:[email protected]:54321/electric" \
electric:local-build
```

## Contributing

See the [Community Guidelines](https://github.com/electric-sql/meta) including the [Guide to Contributing](https://github.com/electric-sql/meta/blob/main/CONTRIBUTING.md) and [Contributor License Agreement](https://github.com/electric-sql/meta/blob/main/CLA.md).
Expand Down
86 changes: 1 addition & 85 deletions docs/api/service.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,10 @@ Configuration options are passed as environment variables, e.g.:
```shell
docker run \
-e "DATABASE_URL=postgresql://..." \
-e "LOGICAL_PUBLISHER_HOST=..." \
-e "PG_PROXY_PASSWORD=..." \
-e "AUTH_JWT_ALG=HS512" \
-e "AUTH_JWT_KEY=..." \
-p 5133:5133 \
-p 5433:5433 \
electricsql/electric
```

Expand All @@ -38,7 +36,7 @@ For a longer form description of how to successfully deploy the sync service and
:::caution A note on ports
Your configuration options affect the number and type of ports that need to be exposed. You must always expose the [`HTTP_PORT`](#http_port). Your [write-to-PG mode](#write-to-pg-mode) and [migrations proxy](#migrations-proxy) config then determines whether you need to expose up-to two TCP ports: the [`LOGICAL_PUBLISHER_PORT`](#logical_publisher_port) and [`PG_PROXY_PORT`](#pg_proxy_port) respectively.

In development using Docker you usually want to map all the necessary ports to your host network (`-p 5133:5133` and `-p 5433:5433` in the example above).
In development using Docker you usually want to map all the necessary ports to your host network (`-p 5133:5133` in the example above).

In production you must make sure your hosting infrastructure exposes the necessary ports and protocols. If not, you can use the workarounds provided in the form of [direct writes mode](#direct-writes-mode) and the [proxy tunnel](#migrations-proxy).

Expand Down Expand Up @@ -132,88 +130,6 @@ Configure how Electric connects to Postgres and exposes its HTTP/WebSocket API.
<LogLevel />
</EnvVarConfig>

## Write-to-PG mode

Electric writes data to Postgres using one of two modes:

1. [logical replication](#logical-replication-mode)
2. [direct writes](#direct-writes-mode)

:::caution
The mode you choose affects your networking config and [database user permissions](#database-user-permissions).
:::

import ElectricWriteToPgMode from './_ELECTRIC_WRITE_TO_PG_MODE.md'

#### ELECTRIC_WRITE_TO_PG_MODE

<EnvVarConfig
name="ELECTRIC_WRITE_TO_PG_MODE"
defaultValue="logical_replication"
example="direct_writes">
<ElectricWriteToPgMode />
</EnvVarConfig>

### Logical replication mode

In `logical_replication` mode, Electric exposes a logical replication publisher service over TCP that speaks the [Logical Streaming Replication Protocol](https://www.postgresql.org/docs/current/protocol-logical-replication.html).

Postgres connects to Electric on `LOGICAL_PUBLISHER_HOST:LOGICAL_PUBLISHER_PORT` and establishes a logical replication subscription to this publisher service. Writes are then streamed in and applied using the logical replication subscription.

```
| <----------- DATABASE_URL ----------- |
Postgres | | Electric
| ---- LOGICAL_PUBLISHER_HOST:PORT ---> |
```

:::caution Caution
In logical replication mode:

1. the [database user](#database-user-permissions) that Electric connects to Postgres as must have the [`SUPERUSER` role attribute](https://www.postgresql.org/docs/16/role-attributes.html#ROLE-ATTRIBUTES)
2. Postgres must be able to connect to Electric (i.e.: must be able to establish an outbound TCP connection) on the host and port that you configure

As a result, you must make sure (in terms of networking / firewalls) not only that Postgres is reachable from Electric but also that Electric is reachable from Postgres. And Electric must know its own address, in order to provide it to Postgres when setting up the logical replication publication that allows writes to be replicated into Postgres.
:::

import LogicalPublisherHost from './_LOGICAL_PUBLISHER_HOST.md'
import LogicalPublisherPort from './_LOGICAL_PUBLISHER_PORT.md'

#### LOGICAL_PUBLISHER_HOST

<EnvVarConfig
name="LOGICAL_PUBLISHER_HOST"
required={true}
example="example.com">
<LogicalPublisherHost />
</EnvVarConfig>

#### LOGICAL_PUBLISHER_PORT

<EnvVarConfig
name="LOGICAL_PUBLISHER_PORT"
defaultValue="5433"
example="65433">
<LogicalPublisherPort />
</EnvVarConfig>

### Direct writes mode

In `direct_writes` mode, Electric writes data to Postgres using a standard interactive client connection. This avoids the need for Postgres to be able to connect to Electric and reduces the permissions required for the database user that Electric connects to Postgres as.

```
Postgres | <----------- DATABASE_URL ----------- | Electric
```

No additional configuration is required for direct writes mode. The `LOGICAL_PUBLISHER_HOST` and `LOGICAL_PUBLISHER_PORT` variables are not required and are ignored.

:::info Why are there two modes?
Originally (prior to v0.8) all writes to Postgres were made using logical replication.

In [version 0.8](/blog/2023/12/13/electricsql-v0.8-released#from-superuser-to-supabase), Electric added direct writes mode to reduce the database user permissions required and increase compatibility with Postgres hosting providers.

In future, we may deprecate logical replication and consolidate on direct writes. However, logical replication may have performance advantages and, for now, we've kept both modes available while direct writes mode gains maturity and we learn more about the operational characteristics of both modes.
:::

## Database user permissions

The Electric sync service connects to Postgres using the [`DATABASE_URL`](#database_url) connection string, in the format `postgresql://[userspec@][hostspec][/dbname]`.
Expand Down
10 changes: 0 additions & 10 deletions docs/deployment/_render.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,16 +51,6 @@ services:
value: "..."
- key: DATABASE_URL
value: "postgresql://..."
- key: LOGICAL_PUBLISHER_HOST
fromService:
type: web
name: tcp-proxy
property: host
- key: LOGICAL_PUBLISHER_PORT
fromService:
type: web
name: tcp-proxy
property: port
- type: web
name: http-proxy
runtime: image
Expand Down
13 changes: 3 additions & 10 deletions docs/deployment/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,18 +65,11 @@ Electric connects to Postgres using a `DATABASE_URL` connection string. In addit

Electric actually connects to Postgres in three ways:

1. it uses a normal interactive client connection to bootstrap resources and make database queries (for example, when you subscribe to a [Shape](../usage/data-access/shapes.md), Electric uses the interactive connection to query for all the rows in the shape)
2. it establishes a logical replication subscription to consume changes from Postgres
1. it uses a normal interactive client connection to bootstrap resources and make database queries (for example, when you subscribe to a [Shape](../usage/data-access/shapes.md), Electric uses the interactive connection to query for all the rows in the shape),
2. it establishes a logical replication subscription to consume changes from Postgres,
3. it writes changes to Postgres that it receives from clients over the [Satellite replication protocol](../api/satellite.md)

This last one, writes changes to Postgres, can be done in two ways:

- you can set `ELECTRIC_WRITE_TO_PG_MODE=direct_writes` to write data to Postgres using a normal interactive client connection
- you can set `ELECTRIC_WRITE_TO_PG_MODE=logical_replication` to stream data into Postgres using a logical replication publication

Direct writes mode works out of the box and is the easiest mode to configure, with no additional network connectivity required. Logical replication mode requires exposing a TCP host and port to Postgres, so that Postgres can establish a logical replication subscription. This can be configured using the `LOGICAL_PUBLISHER_HOST` and `LOGICAL_PUBLISHER_PORT` env vars.

See <DocPageLink path="api/service" /> for more information.
Writes are made over a regular connection with some additional configuration to distinguish client writes and direct-to-postgres writes in conflict resolution. See <DocPageLink path="api/service" /> for more information.

### 2. Migrations <-> Proxy

Expand Down
1 change: 0 additions & 1 deletion docs/deployment/digital-ocean.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,6 @@ services:
DATABASE_URL: "postgresql://postgres:******@example.com/postgres"
PG_PROXY_PASSWORD: "******"
AUTH_MODE: "insecure"
ELECTRIC_WRITE_TO_PG_MODE: "direct_writes"
ports:
- 80:5133
- 65432:65432
Expand Down
2 changes: 0 additions & 2 deletions docs/deployment/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,10 @@ See <DocPageLink path="usage/installation/service" /> to get familiar with confi
```shell
docker run \
-e "DATABASE_URL=postgresql://..." \
-e "LOGICAL_PUBLISHER_HOST=..." \
-e "PG_PROXY_PASSWORD=..." \
-e "AUTH_JWT_ALG=HS512" \
-e "AUTH_JWT_KEY=..." \
-p 5133:5133 \
-p 5433:5433 \
-p 65432:65432 \
-t \
electricsql/electric
Expand Down
1 change: 0 additions & 1 deletion docs/deployment/fly.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ app = "electric-on-fly-test-app"
[env]
AUTH_MODE = "insecure"
DATABASE_URL = "postgresql://..."
ELECTRIC_WRITE_TO_PG_MODE = "direct_writes"
PG_PROXY_PASSWORD = "proxy_password"

# The main Internet-facing service of Electric
Expand Down
6 changes: 1 addition & 5 deletions docs/deployment/k8s.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar_position: 50

To run the [Electric sync service](../api/service.md) in [Kubernetes](https://kubernetes.io) deploy the [Docker image](./docker.md) within a [Pod](https://kubernetes.io/docs/concepts/workloads/controllers/pod), usually via a [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment).

The container image needs ports `5133`, `5433` and `65432` exposed and the environment variables described in <DocPageLink path="api/service" /> configured.
The container image needs ports `5133`, and `65432` exposed and the environment variables described in <DocPageLink path="api/service" /> configured.

For example:

Expand Down Expand Up @@ -35,8 +35,6 @@ spec:
env:
- name: DATABASE_URL
value: "postgresql://..."
- name: LOGICAL_PUBLISHER_HOST
value: "..."
- name: PG_PROXY_PASSWORD
value: "..."
- name: AUTH_JWT_ALG
Expand All @@ -46,8 +44,6 @@ spec:
ports:
- name: satellite-http
containerPort: 5133
- name: logical-pub-tcp
containerPort: 5433
- name: pg-proxy-tcp
containerPort: 65432
```
Expand Down
2 changes: 0 additions & 2 deletions docs/deployment/supabase.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,6 @@ Run your [Electric sync service](../api/service), either locally or [via one of
- set `AUTH_JWT_KEY` to the "JWT Secret" value you retrieved in step 3 above
- set `DATABASE_URL` to the connection string you retrieved in step 2 above
- set `DATABASE_USE_IPV6` to `true` unless you've purchased the IPv4 add-on for Supabase
- set `ELECTRIC_WRITE_TO_PG_MODE` to `direct_writes`
- set `PG_PROXY_PASSWORD` to a secure password value and note it down

Depending on how you run Electric these could be passed as arguments to Docker, set in a ENV file or entered into a managed host's dashboard. For example, to run Electric locally using Docker (replacing the `...` placeholder values with your specific values):
Expand All @@ -95,7 +94,6 @@ docker run \
-e "AUTH_JWT_KEY=..." \
-e "DATABASE_URL=..." \
-e "DATABASE_USE_IPV6=true" \
-e "ELECTRIC_WRITE_TO_PG_MODE=direct_writes" \
-e "PG_PROXY_PASSWORD=..." \
-e "PG_PROXY_PORT=65432" \
-p 5133:5133 \
Expand Down
8 changes: 2 additions & 6 deletions docs/examples/notes/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,10 @@ npm run dev

### Connecting Electric to Postgres

The Electric sync service connects to Postgres using the `DATABASE_URL` environment variable. Depending on your choice of write mode, Postgres may also need to connect to Electric to consume a logical replication publication.

This is configured using the `LOGICAL_PUBLISHER_HOST` (and `LOGICAL_PUBLISHER_PORT`) environment variables:
The Electric sync service connects to Postgres using the `DATABASE_URL` environment variable. Postgres is configured to use `logical` replication to stream changes to Electric.

```
|<--------DATABASE_URL----------|
Postgres | | Electric
|-----LOGICAL_PUBLISHER_HOST--->|
Postgres |<--------DATABASE_URL----------| Electric
```

See <DocPageLink path="api/service" /> for more information.
2 changes: 0 additions & 2 deletions docs/quickstart/_setup_manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,9 @@ docker pull electricsql/electric:latest
docker run \
-e "DATABASE_URL=postgresql://..." \
-e "DATABASE_REQUIRE_SSL=false \
-e "LOGICAL_PUBLISHER_HOST=..." \
-e "PG_PROXY_PASSWORD=..." \
-e "AUTH_MODE=insecure" \
-p 5133:5133 \
-p 5433:5433 \
-p 65432:65432 \
electricsql/electric
```
Expand Down
6 changes: 1 addition & 5 deletions docs/reference/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,11 +95,7 @@ Whilst local writes are final, they are still subject to concurrent merge semant

### Streaming into Electric

When the local database migrations are generated from the Postgres DDL changes, triggers are added that automatically copy insert, update and delete operations on the tables to the "oplog" table. The satellite process then processes these operations by sending them to the Electric server over the Satellite protocol. Electric then applies server-side validation and authorisation before sending on to Postgres over the incoming logical-replication stream.

:::note
Electric acts as a [logical replication publisher](https://www.postgresql.org/docs/current/logical-replication.html). This is why you configure a `LOGICAL_PUBLISHER_HOST` when deploying the Electric sync service -- so that Postgres can connect to consume inbound logical replication.
:::
When the local database migrations are generated from the Postgres DDL changes, triggers are added that automatically copy insert, update and delete operations on the tables to the "oplog" table. The satellite process then processes these operations by sending them to the Electric server over the Satellite protocol. Electric then applies server-side validation and authorisation before executing the writes on Postgres.

### Streaming into Postgres

Expand Down
7 changes: 1 addition & 6 deletions docs/usage/installation/service.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,13 @@ You can run the [pre-packaged Docker images](#images) published on Docker Hub, o

## Configuration

The Electric sync service is configured using environment variables. The three required variables are:
The Electric sync service is configured using environment variables. The two required variables are:

- `DATABASE_URL` in the format of a Postgres [Connection URI](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING-URIS)
- `LOGICAL_PUBLISHER_HOST` that the sync service is running on (must be accessible from the Postgres instance to establish an inbound replication subscription)
- `PG_PROXY_PASSWORD` to safe-guard access to the [Migrations proxy](../data-modelling/migrations.md#migrations-proxy)

```shell
DATABASE_URL="postgresql://user:password@localhost:5432/electric"
LOGICAL_PUBLISHER_HOST="localhost"
PG_PROXY_PASSWORD="..."
```

Expand All @@ -34,7 +32,6 @@ Pre-packaged images are available on Docker Hub at [electricsql/electric](https:
```shell
docker run \
-e "DATABASE_URL=postgresql://..." \
-e "LOGICAL_PUBLISHER_HOST=..." \
-e "PG_PROXY_PASSWORD=..." \
-e "AUTH_MODE=insecure" \
-p 5133:5133 \
Expand Down Expand Up @@ -75,7 +72,6 @@ services:
environment:
DATABASE_URL: postgresql://postgres:pg_password@pg/postgres
DATABASE_REQUIRE_SSL: false
LOGICAL_PUBLISHER_HOST: electric
PG_PROXY_PASSWORD: proxy_password
AUTH_MODE: insecure
ports:
Expand Down Expand Up @@ -103,7 +99,6 @@ Then run:
```shell
docker run \
-e "DATABASE_URL=postgresql://..." \
-e "LOGICAL_PUBLISHER_HOST=..." \
-e "PG_PROXY_PASSWORD=..." \
-e "AUTH_MODE=insecure" \
-p 5133:5133 \
Expand Down
1 change: 0 additions & 1 deletion examples/checkout/backend/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -410,7 +410,6 @@ services:
environment:
DATABASE_URL: postgresql://supabase_admin:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}
PG_PROXY_PASSWORD: proxy_password
LOGICAL_PUBLISHER_HOST: electric
AUTH_MODE: secure
AUTH_JWT_ALG: HS256
AUTH_JWT_KEY: ${JWT_SECRET} # Use the same JWT_SECRET as the rest of Supabase
Expand Down
1 change: 0 additions & 1 deletion examples/checkout/electric/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ services:
environment:
DATABASE_URL: postgresql://postgres:[email protected]:5432/postgres
PG_PROXY_PASSWORD: proxy_password
ELECTRIC_WRITE_TO_PG_MODE: direct_writes
AUTH_MODE: secure
AUTH_JWT_ALG: HS256
AUTH_JWT_KEY: XXXXXXXXXXX
Expand Down
1 change: 0 additions & 1 deletion examples/recipes/backend/docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ services:
environment:
DATABASE_URL: postgresql://postgres:pg_password@postgres:${PG_PORT:-5432}/${APP_NAME:-electric}
PG_PROXY_PASSWORD: proxy_password
LOGICAL_PUBLISHER_HOST: electric
AUTH_MODE: insecure
ports:
- ${ELECTRIC_PORT:-5133}:5133
Expand Down
Loading