Skip to content

Conversation

@drahnr
Copy link
Contributor

@drahnr drahnr commented Nov 3, 2025

Currently we only use LargeSmt<Memory> with the Memory backend. This is not very useful in practice, the entire purpose of LargeSmt is having a disk backed Smt implementation.

This PR will ensure we have the rocksdb feature enabled in miden-crypto until it's extracted per #1218 .

How it's done

We use the dependency feature unification of cargo to ensure we depend on miden-crypto with the rocksdb feature enabled. However, this incurs both a hefty penalty on compile times by default, compiling rocksdb-sys and additional CI dependencies, which leads to disabling it by default.

CI

Since rocksdb comes with some overhead / CI time / cache we only want to run it whenever we change OR link against system level libraries statically.
NOT part of this PR since I am unsure of the best path.

Extras

Some additional type aliases are used to avoid lengthy types.

@drahnr drahnr requested a review from sergerad November 3, 2025 17:55
@drahnr drahnr changed the title expose account optin rocksdb support feat: expose rocksdb feature as opt-in Nov 3, 2025
@drahnr drahnr added the blocked Something prevents working on this label Nov 4, 2025
@drahnr
Copy link
Contributor Author

drahnr commented Nov 4, 2025

Requires 0xMiden/crypto#613 to be merged, otherwise we might get a panic.

Base automatically changed from bernhard-1227-account-tree-with-history to next November 4, 2025 09:44
@drahnr drahnr force-pushed the bernhard-1227-add-optin-rocksdb-support branch from a06e53f to 397aaa9 Compare November 4, 2025 15:09
@drahnr drahnr removed the blocked Something prevents working on this label Jan 14, 2026
Makefile Outdated

WARNINGS=RUSTDOCFLAGS="-D warnings"
BUILD_PROTO=BUILD_PROTO=1
ALL_FEATURES="--features=tracing-forest,concurrent,tracing-forest,tx-prover,batch-prover,block-prover,std"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a comment on why this is necessary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This plays into CI and not compiling rocksdb by default on CI (see absence of it in the list), but only if relevant files changed, i.e. anything in accounts. That was the rough idea to limit the CI roundtrip time to those PRs where it matters. I am aware this is far from optimal and I'd prefer to do it differently (always compile).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not loving the CI complexity.

Is the goal not to always have this enabled? When would we run a node without this once its working? I would imagine that CI caching should keep the compile times in check or no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really hate the CI crap that's part of the PR.
The CI caching should keep the CI times in check, but right now rocksdb also compiles it's C/CXX dependency, which takes a while.

The rationale: It takes a while to compile rocksdb. If it's cached, it re-claims a good chunk of cache size. I don't necessarily want to cache this at all. I'd preferably link statically against host libraries (at least in CI), but this has proven itself to be fragile (llvm compiler version fun).

I am very open to suggestions how to handle this.

Also: Not a fan of GHA, it's a constrained dumpsterfire.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I simplified it to use host libraries. As said earlier, no good solutions, only trade-offs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Host libraries are too old on debian 🫠

Comment on lines 88 to 92
/// The storage backend for account trees.
#[cfg(feature = "rocksdb")]
pub type AccountTreeStorage = RocksDbStorage;
#[cfg(not(feature = "rocksdb"))]
pub type AccountTreeStorage = MemoryStorage;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a roadmap or point-in-time when we would always have this enabled?

@drahnr drahnr force-pushed the bernhard-1227-add-optin-rocksdb-support branch from e42fc7e to 164f712 Compare January 15, 2026 16:22
@drahnr drahnr changed the title feat: expose rocksdb feature as opt-in feat: expose rocksdb feature as ~opt-in~ default Jan 15, 2026
@drahnr drahnr changed the title feat: expose rocksdb feature as ~opt-in~ default feat: expose rocksdb feature as default storage for AccountTree and NullifierTree Jan 15, 2026
@drahnr drahnr requested a review from sergerad January 15, 2026 19:38
@drahnr drahnr force-pushed the bernhard-1227-add-optin-rocksdb-support branch from 07c4f17 to a29ed14 Compare January 16, 2026 10:47
@drahnr drahnr changed the title feat: expose rocksdb feature as default storage for AccountTree and NullifierTree feat: rocksdb storage for AccountTree and NullifierTree by default Jan 16, 2026
@drahnr drahnr merged commit cc56cd1 into next Jan 16, 2026
18 checks passed
@drahnr drahnr deleted the bernhard-1227-add-optin-rocksdb-support branch January 16, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants