fix(electric): Advance the active replication slot even when there are no writes to electrified tables #1402
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is needed to prevent the active replication slot from retaining too much WAL when there's no write activity in electrified tables.
alco
force-pushed
the
alco/logical-message-ack
branch
from
edc15f0
to
4fd8520
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We've had the logic for advancing the main replication slot for some time now. That slot is used as a way to retain WAL records, allowing the sync service to maintain a window within which it can go back in time and replay old transactions.
A separate replication slot is used for the logical replication connection that the sync service maintains with Postgres when it's running. This slot is also known as the active replication slot.
The issue this PR addresses is caused by the fact that the active replication slot is never advanced when electrified tables aren't seeing any writes. This manifests as an endlessly growing replication lag and disk usage when the database sees a constant rate of writes to non-electrified tables:
With the fix in this PR in place, the sync service can now maintain constant replication lag that we can further adjust, for example, to prevent DigitalOcean from raising alerts about high replication lag. On the chart below you can see how disk usage goes down when the sync service is upgraded to a patched version: it never goes up the same way it used to:
And here's zoomed-in view of a 6-hour time span with the patched version running:
Fixes #1285.