-
Notifications
You must be signed in to change notification settings - Fork 970
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
publish: build checkpoint files during ledger close #4446
Conversation
62d8121
to
ca1f928
Compare
Note: the current version of this PR drops |
ca1f928
to
c479ba8
Compare
c479ba8
to
3cbf7a2
Compare
ae408dc
to
ccf6b27
Compare
f630856
to
63140a6
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Broadly speaking: great work! Handful of nits, handful of clarifying questions / requests for documentation about what's going on, and a couple places I'm uncertain about the recovery logic that I'd like to tighten up a bit (or understand what I'm currently failing to). But overall happy to see, and happy to collaborate on wrapping it up tomorrow.
5750285
to
7c23875
Compare
7c23875
to
4c39bf5
Compare
4c39bf5
to
c910a9d
Compare
Resolves #4212
Resolves #4469
The change removes publish dependency on
txhistory
andledgerheaders
SQL tables, and instead constructs ledger headers/tx sets/tx results checkpoints at ledger close. This allows us to deprecate and drop support fortxhistory
entirely, which is beneficial for performance. Additionally, the code became simpler, and the xdr->sql->xdr roundtrip processing has been removed (as part of this,txsethistory
table was dropped as well). The tradeoff in this approach is that we lose atomicity we had with SQL transactions. As a result, core now manually has to handle crashes. This is done by always constructing "tmp" checkpoints first, and only promoting them after a checkpoint ledger has been committed to the database. In the event thattmp
file has been written, but core crashed in apply and rolled back the transaction, there's automatic file cleanup on startup, that trims any uncommitted ledgers from the temp file.This change also drops
publishqueue
SQL table completely in favor of checkpoint files containing HAS. A similar recovery mechanism is used to cleanup files in case of a crash, except it's simpler, since we can just delete whole checkpoint files instead of truncating them.Note: now that we drop
txhistory
, we can also deprecate the--force-back
flag.