feat: Initial MVP implementation of RocksDbStorage #623

jsantell · 2023-09-06T22:02:40Z

First pass on a RocksDB backend.

Built on #622 with a few more changes to make storage agnostic in the remaining parts (e.g. helpers and noosphere::platform). See the patch on top of #622.

Feature Flags

There's an additive flag for using RocksDB in multithreaded mode (rational: target features using base feature, cleaner especially given a default assumption of "sled" in a world where we could also have more backends).

Flags for noosphere and noosphere_cli just propagate to noosphere_storage. A different name (experimental-rocksdb vs rocksdb) used at the top level to denote experimentality, and ensure Cargo feature flags propagate correctly when using a workspace rather than applying everywhere applicable. With more RocksDB experience, we can most likely just use one (single or multi threaded).

noosphere:experimental-rocksdb
noosphere:experimental-rocksdb-mt
noosphere_cli:experimental-rocksdb
noosphere_cli:experimental-rocksdb-mt
noosphere_storage:rocksdb
noosphere_storage:rocksdb-mt

There are other ways of managing these backends, no strong preference.

Performance

Definitely room for improvement here, but currently the RocksDB implementation is half as slow as sled. No noticeable difference between single and multithreaded RocksDB implementations.

Running full test suite (with inconsistent doc tests)

55s-70s RocksDB
28s-40s Sled

Running noosphere-cli tests only

5.9s RocksDB
2.5s Sled

jsantell · 2023-09-08T21:15:16Z

Added PerformanceStorage<S: Storage> for tracking perf/space (could be rolled into TrackingStorage), a Space trait (with Space::get_space_usage) implemented for storage providers.

The benchmarks are in noosphere_storage/examples/bench. This work is gated behind performance flag, though even with features defined in the example manifest, it still needs passed in the command line. Updated CI to add these flags, but there's gotta be a better solution.

jsantell · 2023-09-08T21:43:35Z

Initial takeaways of running some high-level comparisons in these benches:

edit: updated with crude impl of sqlite

DB	Sm Read	Lg Read	Sm Write	Lg Write	Space Amp
Sled	10us	30us	30us	140us	2
Rocks	4us	10us	30us	100us	1.1
Sqlite	86us	84us	19000us	17000us	1.1
IDB (Firefox)	700us	1200us	900us	1300us	1.2
IDB (Chrome)	1200us	1200us	7700us	9700us	1.2

Not calling RocksDB flush results in similar read/write performance between Rocks and Sled (with Rocks slightly faster). The WAL should mean we do not need to manually call flush.
Firefox write performance is faster than Chrome by an order of magnitude (1-1.5ms vs 8-10ms)
RocksDB and IndexedDb implementations have pretty good space amplification (~1.25x). Sled is closer to 2x.
IndexedDb and Sqlite have worse space amplification for small databases, not particularly relevant IMO
Suspicious that small reads/writes take longer than large read/writes for Sqlite

jsantell · 2023-09-08T22:08:28Z

Adding --experimental-rocksdb to apple builds to test the generated artifact

jsantell · 2023-09-11T19:06:05Z

Merged in SqliteStorage work from #549 (stats updated in #623 (comment)), fixed up glue to hopefully generate a usable iOS simulator artifact

cdata

Awesome work on this.

rust/noosphere-cli/Cargo.toml

.github/workflows/noosphere_apple_build.yaml

rust/noosphere-cli/tests/helpers/random.rs

rust/noosphere-core/src/bench.rs

rust/noosphere-storage/src/implementation/rocks_db.rs

rust/noosphere-storage/src/implementation/sqlite.rs

jsantell · 2023-09-11T23:35:40Z

Updated from feedback (not yet accounting for rebase shuffling of some helpers)

rust/noosphere-storage/src/implementation/rocks_db.rs

rust/noosphere/build.rs

rust/noosphere/src/platform.rs

rust/noosphere-storage/src/implementation/rocks_db.rs

cdata · 2023-09-12T17:48:37Z

LMK when you are at a good point and I'll give the rebase a go now that #624 is landed.

rust/noosphere-storage/src/implementation/rocks_db.rs

jsantell · 2023-09-12T20:58:24Z

.github/workflows/noosphere_apple_build.yaml

@@ -72,7 +72,7 @@ jobs:
        with:
          brew: protobuf cmake
      - name: 'Cargo Build'
-        run: cargo build --package noosphere --release --locked --target ${{ matrix.target }}
+        run: cargo build --package noosphere --release --locked --target ${{ matrix.target }} --features rocksdb


This is the only thing that should be backed out before landing.

Why do we want to back this out? We want this to be the default for iOS, right?

Thinking about this a little more: we probably want to make this part of the matrix that gets generated in this step. We should use rocksdb for iOS and the default (presumably sled) everywhere else so that we don't have to migrate orb-created desktop replicas just yet.

Just decoupling the implementation from the "on" switch -- we could make it a part of the build matrix, or configure it in noosphere::platform

IDK if it's possible to distinguish e.g., M2/iPad from M2/MacBook Air in this way. We may need to talk it over a little bit to figure out the safe path forward (or face coming up with a migration playbook for orb users).

cdata · 2023-09-13T16:24:24Z

rust/noosphere-common/src/channel.rs

-/// Represents a request to be processed in `MessageProcessor`,
-/// sent from the associated `MessageClient`.
+/// Represents a request to be processed in [MessageProcessor],
+/// sent from the associated [MessageClient].


A doc example might be a nice stand-in for tests, just to demonstrate intended usage.

Ah, nevermind, I see there is a test!

cdata · 2023-09-13T16:29:50Z

rust/noosphere/src/ffi/noosphere.rs

@@ -94,6 +94,9 @@ impl Drop for NsNoosphere {
 /// You can also initialize the ns_noosphere_t with an optional third
 /// argument: a URL string that refers to a Noosphere Gateway API somewhere on
 /// the network that one or more local spheres may have access to.
+///
+/// Note that storages (`global_storage_path`, `sphere_storage_path`) can only
+/// be opened by a single `ns_noosphere` at a time.


rust/noosphere-storage/src/implementation/rocks_db.rs

cdata · 2023-09-13T16:45:50Z

rust/noosphere-storage/src/implementation/rocks_db.rs

+}
+
+impl RocksDbStorage {
+    pub fn new(path: PathBuf) -> Result<Self> {


What happens if the storage is initialized multiple times over the same location? Idempotent/safe to do?

Same restrictions as sled e.g. only one new() per path allowed at a time, e.g. this fails:

async fn it_can_open_concurrent_dbs() -> Result<()> { let tempdir = tempfile::TempDir::new()?; let storage_path = tempdir.path().to_owned(); let storage_1 = RocksDbStorage::new(storage_path.clone())?; let mut store_1 = storage_1.get_key_value_store("links").await?; let storage_2 = RocksDbStorage::new(storage_path.clone())?; let mut store_2 = storage_2.get_key_value_store("links").await?; let bytes_1 = vec![0, 1, 2, 3]; let bytes_2 = vec![4, 5, 6, 7]; store_1.write(b"from 1", &bytes_1).await?; store_2.write(b"from 2", &bytes_2).await?; assert_eq!(store_1.read(b"from 2").await?.unwrap(), bytes_2); assert_eq!(store_2.read(b"from 1").await?.unwrap(), bytes_1); Ok(()) }

* Introduce `RocksDbStorage` * Rename NativeStorage -> SledStorage, WebStorage -> IndexedDbStorage. Make storage generic over gateway. * Added `PerformanceStorage<S: Storage>`, `trait noosphere_storage::Space` for benchmarking storage backends.

cdata

LGTM, excellent change!

This test started failing while testing subconsciousnetwork/noosphere#623, leading us to question why it ever passed in the first place. Due to Swift compiler trivia the deinit for `Noosphere` is not guaranteed to be called at the time we overwrite the variable, meaning two instances of `Noosphere` co-exist pointing at the same data. We should not ever be trying to open multiple instances of `Noosphere` pointing at the same database. After team discussion we decided to drop this test case and make this an explicit failure case on the Noosphere side.

jsantell requested a review from cdata September 6, 2023 22:02

jsantell force-pushed the rocksdb branch from ecd9d2b to 04fbf25 Compare September 6, 2023 22:10

jsantell mentioned this pull request Sep 11, 2023

chore: Genercize storage across gateway and rename "default" storages #622

Closed

cdata reviewed Sep 11, 2023

View reviewed changes

cdata reviewed Sep 12, 2023

View reviewed changes

rust/noosphere-storage/src/implementation/rocks_db.rs Outdated Show resolved Hide resolved

cdata reviewed Sep 12, 2023

View reviewed changes

rust/noosphere/build.rs Show resolved Hide resolved

cdata reviewed Sep 12, 2023

View reviewed changes

rust/noosphere/build.rs Show resolved Hide resolved

cdata reviewed Sep 12, 2023

View reviewed changes

rust/noosphere/src/platform.rs Outdated Show resolved Hide resolved

jsantell commented Sep 12, 2023

View reviewed changes

rust/noosphere-storage/src/implementation/rocks_db.rs Outdated Show resolved Hide resolved

cdata reviewed Sep 12, 2023

View reviewed changes

rust/noosphere-storage/src/implementation/rocks_db.rs Show resolved Hide resolved

jsantell force-pushed the rocksdb branch 3 times, most recently from 27b26df to 7edf0b0 Compare September 12, 2023 20:57

jsantell commented Sep 12, 2023

View reviewed changes

jsantell requested a review from cdata September 12, 2023 21:05

jsantell force-pushed the rocksdb branch from 7edf0b0 to f9dca50 Compare September 12, 2023 21:56

bfollington mentioned this pull request Sep 13, 2023

Remove testWritesThenCloseThenReopenWithVars subconsciousnetwork/subconscious#891

Merged

cdata reviewed Sep 13, 2023

View reviewed changes

rust/noosphere-storage/src/implementation/rocks_db.rs Outdated Show resolved Hide resolved

cdata reviewed Sep 13, 2023

View reviewed changes

jsantell force-pushed the rocksdb branch from f9dca50 to e11b072 Compare September 13, 2023 17:04

jsantell force-pushed the rocksdb branch from e11b072 to f0a27dd Compare September 13, 2023 17:05

cdata approved these changes Sep 13, 2023

View reviewed changes

cdata merged commit 7155f86 into main Sep 13, 2023

cdata deleted the rocksdb branch September 13, 2023 18:15

github-actions bot mentioned this pull request Sep 19, 2023

chore: release main #616

Merged

cdata mentioned this pull request Sep 21, 2023

feat: Introduce a streaming way to encode BodyChunkIpld. Fixes #498 #586

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Initial MVP implementation of RocksDbStorage #623

feat: Initial MVP implementation of RocksDbStorage #623

jsantell commented Sep 6, 2023

jsantell commented Sep 8, 2023 •

edited

Loading

jsantell commented Sep 8, 2023 •

edited

Loading

jsantell commented Sep 8, 2023

jsantell commented Sep 11, 2023

cdata left a comment

jsantell commented Sep 11, 2023

cdata commented Sep 12, 2023

jsantell Sep 12, 2023

cdata Sep 13, 2023

jsantell Sep 13, 2023

cdata Sep 13, 2023

cdata Sep 13, 2023

cdata Sep 13, 2023

cdata Sep 13, 2023

cdata Sep 13, 2023

jsantell Sep 13, 2023

cdata left a comment

feat: Initial MVP implementation of RocksDbStorage #623

feat: Initial MVP implementation of RocksDbStorage #623

Conversation

jsantell commented Sep 6, 2023

Feature Flags

Performance

jsantell commented Sep 8, 2023 • edited Loading

jsantell commented Sep 8, 2023 • edited Loading

jsantell commented Sep 8, 2023

jsantell commented Sep 11, 2023

cdata left a comment

Choose a reason for hiding this comment

jsantell commented Sep 11, 2023

cdata commented Sep 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdata left a comment

Choose a reason for hiding this comment

jsantell commented Sep 8, 2023 •

edited

Loading

jsantell commented Sep 8, 2023 •

edited

Loading