Twom database format and first implementation #5157
Conversation
brong
force-pushed
the
twom
branch
2 times, most recently
from
2a4c0a9 to
74d9e42
Compare
|
I've removed include-in-fm while tests are failing. |
brong
force-pushed
the
twom
branch
4 times, most recently
from
7d3b303 to
5ffbf60
Compare
brong
requested review from
dilyanpalauzov,
robmueller,
rjbs,
elliefm,
wolfsage and
rsto
and removed request for
dilyanpalauzov
brong
force-pushed
the
twom
branch
2 times, most recently
from
981f5c7 to
a0615b2
Compare
|
Bah, CI failed because |
brong
requested review from
elliefm,
dilyanpalauzov,
wolfsage,
robmueller,
ksmurchison and
rjbs
and removed request for
dilyanpalauzov
|
Ok I've rewritten to have DELETE just be an ancestor pointer in the level0 chain (no key or value) and the most recent ADD or REPLACE always be in the full multi-level links except for the DELETE being immediately before it if present. This added a 'deleted_offset' to struct tm_loc, and a bunch of extra logic (slightly more code overall) - but partly because I just open-coded it everywhere, it could do with some code refactoring but it's working and the structure is what I want, so I figured it's time to ask for reviews again! I also wrote up a bunch of extra doc about it which I'll pop in the repo. |
| return r; | ||
| } | ||
|
|
||
| int twom_db_fetch(struct twom_db *db, |
There was a problem hiding this comment.
Can the public functions in this file be marked as HIDDEN ⇔ __attribute__((__visibility__("hidden"))), as they are not called from outside libcyrus.so?
There was a problem hiding this comment.
The intention is to have this as a completely separate library eventually, though the code is in Cyrus, so I don't want to do this.
|
Based on robmueller comments elsewhere, I have implemented two-stage locking and separate repack locking (also based on wolfsage review up above about the repack) |
brong
force-pushed
the
twom
branch
5 times, most recently
from
471dfc1 to
a02584a
Compare
This contains the full implementation of the xxhash algorithm. It's only being used by twom database format, so it's being compiled straight into that file. This is an unmodified copy of xxhash.h from: commit dd11140c2dc5d53a3c0a949d67af7f40f546878e in the repository at [email protected]:Cyan4973/xxHash.git
A cyrusdb wrapper around twom library
see: https://stackoverflow.com/questions/27625597/how-to-implement-a-writer-preferring-read-write-lock-for-nix-processes Two stage locking; 1) "headlock" - locks the first 16 bytes of the message (the twom magic) 2) "datalock" - locks the DUMMY record And there's also a third lock: 3) "repacklock" - locks the GENERATION field (8 bytes at offset 40) I'll document this in a separate commit
Throwing this open for wide reviews now! It's been rewritten a BUNCH of times, and optimised a fair bit - it's got a simpler internal structure and basically everything aligned with the right datastructures I think. It does MVCC reads nicely, can keep its transaction in foreach unless the callback writes, in which case it STILL does the right thing and re-locks and re-finds itself.
On disk format of records is:
