-
Notifications
You must be signed in to change notification settings - Fork 84
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make persistence incremental thread-safe for Network Reliability layer #1211
Make persistence incremental thread-safe for Network Reliability layer #1211
Conversation
Transactions CostsSizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using
Script summary
Cost of Init Transaction
Cost of Commit TransactionThis is using ada-only outputs for better comparability.
Cost of CollectCom Transaction
Cost of Close Transaction
Cost of Contest Transaction
Cost of Abort TransactionSome variation because of random mixture of still initial and already committed outputs.
Cost of FanOut TransactionInvolves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.
End-To-End Benchmark ResultsThis page is intended to collect the latest end-to-end benchmarks results produced by Hydra's Continuous Integration system from the latest Please take those results with a grain of salt as they are currently produced from very limited cloud VMs and not controlled hardware. Instead of focusing on the absolute results, the emphasis should be on relative results, eg. how the timings for a scenario evolve as the code changes. Generated at 2023-12-15 10:50:09.935283054 UTC Baseline Scenario
Baseline Scenario
|
6bd29fa
to
a3e2aa6
Compare
a3e2aa6
to
588a7a3
Compare
588a7a3
to
63a1a83
Compare
8853335
to
77d749b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should fix the bug and I added some minor nitpicks myself.
(forever $ threadDelay 0.01 >> loadAll) | ||
(forM_ moreItems $ \item -> append item >> threadDelay 0.01) | ||
`shouldThrow` \case | ||
IncorrectAccessException{} -> True |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Funnily, I had something like that initially, but I found giving a proper name to the race_ ...
block more legible and stating explicitly what it's supposed to do instead of requiring the user to unpack the whole thing.
This small experiment demonstrates how to detect "race conditions" when some atomic result is expected, the idea is to use a TVar to simulate files operations and show the consequences of non-atomic writes and reads on messages resending.
This test is not very satisfying but for now I cannot come up with a better idea. It shows that with large messages (~10kB) there's exception thrown but the peers can get stuck, which is a different problem than the one observed in production.
We try to improve naming the log messages' fields, make it more consistent between different messages, and replace field accessor function and positional construction with proper field name punning.
We used to do that but then moved to always reloading from file, which lead to race conditions when reading/writing concurrently and incorrect deserialisation of messages. It seems both safer and faster to use a local cache, and only reload messages at start of the network layer.
77d749b
to
1b07b81
Compare
This PR attempts to fix #1202 by removing concurrent loading and appending of messages, and using an in-memory cache to resend messages. This is achieved by preventing loading from a different thread in
PersistenceIncremental
once we have startedappend
ing to it.This is a bit lame, and there are actually more problems lurking in this persistence layer as we cannot guarantee the vector clock indices are always in sync with the messages we persist. We should start thinking of moving to a persistence backend where we can guarantee atomicity of updates to vector clock and persisted message.