Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

publish: build checkpoint files during ledger close #4446

Merged
merged 10 commits into from
Oct 18, 2024

Conversation

marta-lokhova
Copy link
Contributor

@marta-lokhova marta-lokhova commented Aug 29, 2024

Resolves #4212
Resolves #4469

The change removes publish dependency on txhistory and ledgerheaders SQL tables, and instead constructs ledger headers/tx sets/tx results checkpoints at ledger close. This allows us to deprecate and drop support for txhistory entirely, which is beneficial for performance. Additionally, the code became simpler, and the xdr->sql->xdr roundtrip processing has been removed (as part of this, txsethistory table was dropped as well). The tradeoff in this approach is that we lose atomicity we had with SQL transactions. As a result, core now manually has to handle crashes. This is done by always constructing "tmp" checkpoints first, and only promoting them after a checkpoint ledger has been committed to the database. In the event that tmp file has been written, but core crashed in apply and rolled back the transaction, there's automatic file cleanup on startup, that trims any uncommitted ledgers from the temp file.

This change also drops publishqueue SQL table completely in favor of checkpoint files containing HAS. A similar recovery mechanism is used to cleanup files in case of a crash, except it's simpler, since we can just delete whole checkpoint files instead of truncating them.

Note: now that we drop txhistory, we can also deprecate the --force-back flag.

@marta-lokhova
Copy link
Contributor Author

Note: the current version of this PR drops txhistory table for simplicity, however depending on who depends on it, we might need to still support it, while also building checkpoint files directly.

@marta-lokhova marta-lokhova changed the title Build checkpoint files during ledger close publish: build checkpoint files during ledger close Sep 5, 2024
@marta-lokhova marta-lokhova force-pushed the publishSQlPath branch 5 times, most recently from ae408dc to ccf6b27 Compare September 12, 2024 01:33
@marta-lokhova marta-lokhova force-pushed the publishSQlPath branch 4 times, most recently from f630856 to 63140a6 Compare October 14, 2024 21:43
Copy link
Contributor

@graydon graydon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Broadly speaking: great work! Handful of nits, handful of clarifying questions / requests for documentation about what's going on, and a couple places I'm uncertain about the recovery logic that I'd like to tighten up a bit (or understand what I'm currently failing to). But overall happy to see, and happy to collaborate on wrapping it up tomorrow.

src/main/test/ApplicationUtilsTests.cpp Show resolved Hide resolved
src/history/test/HistoryTests.cpp Outdated Show resolved Hide resolved
src/history/StateSnapshot.cpp Show resolved Hide resolved
src/history/StateSnapshot.cpp Outdated Show resolved Hide resolved
src/history/HistoryManagerImpl.cpp Outdated Show resolved Hide resolved
src/history/HistoryManagerImpl.cpp Show resolved Hide resolved
src/history/HistoryManagerImpl.cpp Outdated Show resolved Hide resolved
src/ledger/LedgerManagerImpl.cpp Outdated Show resolved Hide resolved
src/ledger/LedgerManagerImpl.h Outdated Show resolved Hide resolved
src/history/CheckpointBuilder.cpp Show resolved Hide resolved
graydon
graydon previously approved these changes Oct 17, 2024
@graydon graydon added this pull request to the merge queue Oct 18, 2024
Merged via the queue into stellar:master with commit d60775c Oct 18, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants