-
Notifications
You must be signed in to change notification settings - Fork 135
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix(electric): Advance the active replication slot even when there ar…
…e no writes to electrified tables (#1402) We've had the logic for advancing the main replication slot for some time now. That slot is used as a way to retain WAL records, allowing the sync service to maintain a window within which it can go back in time and replay old transactions. A separate replication slot is used for the logical replication connection that the sync service maintains with Postgres when it's running. This slot is also known as the active replication slot. The issue this PR addresses is caused by the fact that the active replication slot is never advanced when electrified tables aren't seeing any writes. This manifests as an endlessly growing replication lag and disk usage when the database sees a constant rate of writes to non-electrified tables: ![Screenshot from 2024-03-23 18-19-05](https://github.com/electric-sql/electric/assets/207748/2047f530-4945-4d03-a6fd-e6e61fbe5c9a) With the fix in this PR in place, the sync service can now maintain constant replication lag that we can further adjust, for example, to prevent DigitalOcean from raising alerts about high replication lag. On the chart below you can see how disk usage goes down when the sync service is upgraded to a patched version: it never goes up the same way it used to: ![Screenshot from 2024-06-25 13-00-58](https://github.com/electric-sql/electric/assets/207748/a427fbe0-f188-474a-b319-36b0a7d31062) And here's zoomed-in view of a 6-hour time span with the patched version running: ![Screenshot from 2024-06-25 13-01-09](https://github.com/electric-sql/electric/assets/207748/9834ca7b-2a2c-4a1a-8140-d355cdeb4739) Fixes #1285.
- Loading branch information
Showing
3 changed files
with
133 additions
and
13 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
47 changes: 47 additions & 0 deletions
47
e2e/tests/01.08_electric_acknowledges_logical_message_lsn.lux
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
[doc Electric acknowledges logical message's LSN] | ||
[include _shared.luxinc] | ||
|
||
[invoke setup] | ||
|
||
[shell pg_1] | ||
!SELECT pg_logical_emit_message(false, '', 'hello from PG'); | ||
?? pg_logical_emit_message | ||
??------------------------- | ||
?(\d+/[0-9a-fA-F]+) | ||
[my logical_msg_lsn=$1] | ||
??(1 row) | ||
|
||
[shell electric] | ||
??Got a message from PG via logical replication: \ | ||
%Electric.Postgres.LogicalReplication.Messages.Message{\ | ||
transactional?: false, \ | ||
lsn: #Lsn<$logical_msg_lsn>, \ | ||
prefix: "", \ | ||
content: "hello from PG"} | ||
??Acknowledging $logical_msg_lsn | ||
|
||
[shell pg_1] | ||
!BEGIN; | ||
??BEGIN | ||
|
||
!SELECT pg_logical_emit_message(true, '', 'hello from PG transaction'); | ||
?? pg_logical_emit_message | ||
??------------------------- | ||
?(\d+/[0-9a-fA-F]+) | ||
[my logical_msg_lsn=$1] | ||
??(1 row) | ||
|
||
!COMMIT; | ||
??COMMIT | ||
|
||
[shell electric] | ||
??Got a message from PG via logical replication: \ | ||
%Electric.Postgres.LogicalReplication.Messages.Message{\ | ||
transactional?: true, \ | ||
lsn: #Lsn<$logical_msg_lsn>, \ | ||
prefix: "", \ | ||
content: "hello from PG transaction"} | ||
??Acknowledging $logical_msg_lsn | ||
|
||
[cleanup] | ||
[invoke teardown] |