Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimise successive imports (for quick removal, re-add of tables) #31

Open
lawrencejones opened this issue May 4, 2020 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@lawrencejones
Copy link
Owner

At present, removal of a table from the publication causes us to consider that table totally unreliable. In order to re-add the table to our stream, we'd schedule an import of all existing data, in order to ensure we haven't missed anything.

Given users should be able to remove tables without too much worry, especially if one table has changes that are incompatible with the sink, imports should attempt to be cheap. If we were to track the LSN at which we ceased to stream changes, we might be able to filter which rows are included in a subsequent import by using the xmin field on each row.

@lawrencejones lawrencejones added the enhancement New feature or request label Jan 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant