RocksDB Python API

[joe-iddon]

Reference Issues/PRs

Closes #913
Extends #945 (C++ tests) and #961 (Conda mac problems)

What does this implement or fix?

Implements the RocksDB Python API. Update: We will not advertise this yet, since conda support is not there, and the documentation still needs writing.

Since the C++ code for RocksDB is excluded on conda, a static readonly property is attached to LibraryManager in the python_bindings.cpp there so that the Python code knows whether to offer up the rocksdb_library_adapter.py, or not.

I.e.

>>> arcticdb_ext.storage.LibraryManager.rocksdb_support
False # On conda. On PyPi (windows + Linux), this will give True

Still to do:

• Add a warning if ever opened twice, like in Warning for LMDB when library reopened #1000 . Unlike LMDB, RocksDB does not support multi-process access.
• Add a proper cleanup method, as was done with LMDB. See code of rocksdb_library_adapter.py. Should be similar to Small LMDB Fixes: 2GiB map size for Windows, Validation before delete #918
• See below about switching from Slice to PinnableSlice.

Any other comments?

Removed the std::make_unique anti-pattern that was introduced to storage_factory.cpp in #625 which required Storage implementations to be move-constructable. This was highlighted to me since I defined a custom destructor for the RocksDBStorage class which destroyed its default move constructor, so gave compiler errors.

Ideas for Profiling + Improving Performance / Configuring RocksDB

Profiling ideas:

• Need to compile in release not debug mode, since RocksDB should behave better in this mode
• Flamegraph+perf
• Enable performance tracing with #define. Note might want to add more ARCTICDB_SAMPLE macros for this to rocksdb_storage.cpp.
• Use py-spy with PID input and speedscope output using options --native to profile C++ code, and --idle to include time spent waiting on I/O operations etc
• Edit: We now have a Wiki page to explain our profiling options.
• Ideas for graphs/parameters to vary in comparing LMDB and RocksDB:
  • write+read for different data sizes, varying segment size (rows+columns) since RocksDB's LSM tree should out-perform LMDB's BTree when there are a large number of key/value pairs.
  • batch_write and batch_read operations to compare parallel performance
  • append and update as these will produce smaller segments, so may favor RocksDB as more key/value pairs

Performance ideas

• As noted in the code, should switch from Slice to PinnableSlice. This is only possible once Add a mechanism to extend storage transaction lifetime to lifetime of… #975 has been merged, but will enable us to eliminate a copy in creating the Segment from the bytes during a read.
• Writes cannot be optimised in this way. There is a necessary copy, which unlike in LMDB, cannot be avoided.
• Optimise the BloomFilter (already enabled, but could be tuned).
• Add block caching
• Configure number of SST files in L0, L1, etc.
• Configure how many I/O threads to give RocksDB. Currently set to 16 from default 1 using IncreaseParallelism, but don't want too many, because then might try too hard to distribute work and harm performance from context switching.
• Implement WriteBatch when deleting/putting multiple key/values, rather than Put and Delete one-by-one.
• Add a WaitForCompact before closing as this will mean the DB can be opened faster next time.

Documentation and reference material on RocksDB:

• Documentation of RocksDB is on Facebook's GitHub Wiki.
• The Basic Operations section is a good starting point.
• A good guide to the internals can be found here.

Random other idea:

• The C++ test parameterization in test_embedded.cpp could be made neater in two ways: either just spread the list definition over multiple lines, then can literally #define out the one RocksDB if we are on conda; OR make the BackendGenerator class have a constexpr constructor and then should be able to use a constexpr version of emplace_back ... except not on C++20, so would have to write a constexpr version of emplace_back.

Checklist

Checklist for code changes...

• Have you updated the relevant docstrings, documentation and copyright notice?
• Is this contribution tested against all ArcticDB's features?
• Do all exceptions introduced raise appropriate error messages?
• Are API changes highlighted in the PR description?
• Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?

[poodlewars]

We can just hang thing off the module like here

ArcticDB/cpp/arcticdb/storage/python_bindings.cpp

Line 38 in 263e843

storage.attr("CONFIG_LIBRARY_NAME") = py::str(arcticdb::storage::CONFIG_LIBRARY_NAME);

can't we, no need for this to be tied to the library_manager?

[poodlewars]

Suggested change

	// the consturction of the segment
	// the construction of the segment

[poodlewars]

Ah wow, nice cleanup, much better.

[poodlewars]

rm

[poodlewars]

Can you document in here that this is a work in progress, beta API and should not be relied upon?
And also log some warnings if anyone does actually use the RocksDB adapter so they see at runtime we're not making any promises about it.