feat: sqlite vfs fdb implementation

[NathanFlurry]

Changes

[NathanFlurry]

• chore: add compression #2345
• feat: sqlite vfs fdb implementation #2282 👈 (View in Graphite)
• main

---

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[cloudflare-workers-and-pages]

Deploying rivet with    Cloudflare Pages

Latest commit:	3d9c1c5
Status:	✅  Deploy successful!
Preview URL:	https://05b1cfbe.rivet.pages.dev
Branch Preview URL:	https://03-29-feat-sqlite-vfs-fdb-im.rivet.pages.dev

View logs

[greptile-apps]

PR Summary

This PR implements a SQLite Virtual File System (VFS) for FoundationDB, allowing SQLite databases to be stored in FoundationDB's distributed key-value store.

• Implements a complete SQLite VFS with page-based storage in FoundationDB, including file operations (read/write/truncate) and proper transaction handling
• Adds comprehensive Prometheus metrics for monitoring performance, errors, and resource usage
• Includes benchmarks comparing FDB VFS against filesystem-based SQLite with different journal modes (DELETE, WAL, WAL2)
• Creates a test framework with helper utilities for testing SQLite VFS implementations
• Uses a keyspace approach with prefixes to ensure key uniqueness and proper organization in FoundationDB

_{24 file(s) reviewed, 27 comment(s)
Edit PR Review Bot Settings | Greptile}

[greptile-apps]

style: lazy_static is duplicated in both dependencies and dev-dependencies sections

Suggested change

lazy_static = "1.4.0"

[greptile-apps]

style: This contains a hardcoded absolute path specific to a developer's environment. Consider replacing with a placeholder like path/to/your/test/binary or instructions on how to find the correct path from the cargo output.

Suggested change

	# Run test with lldb (for example, test_file_create_metadata)
	RUST_BACKTRACE=1 lldb /Users/nathan/rivet/ee/oss/./target/debug/deps/file_ops-d47459581e41a236 test_file_create_metadata
	# Run test with lldb (for example, test_file_create_metadata)
	RUST_BACKTRACE=1 lldb path/to/test/binary test_file_create_metadata

[greptile-apps]

logic: VFS is registered here in main() but also in setup_sqlite() on line 102, which could lead to multiple registrations of the same VFS.

[greptile-apps]

logic: query_count() is used to fetch a text value, but this function is designed to return a count. Consider using a more appropriate function for retrieving text data.

[greptile-apps]

style: There's an extra space after the comma before the new package entry that should be removed for consistency with other entries.

[greptile-apps]

logic: Missing libc dependency in the imports. The tests use libc::malloc and other C functions, but libc isn't imported here.

Suggested change

	use std::ffi::{c_void, CString};
	use libsqlite3_sys::{sqlite3_file, SQLITE_OK, SQLITE_OPEN_CREATE, SQLITE_OPEN_READWRITE};
	use uuid::Uuid;
	use std::ffi::{c_void, CString};
	use libsqlite3_sys::{sqlite3_file, SQLITE_OK, SQLITE_OPEN_CREATE, SQLITE_OPEN_READWRITE};
	use uuid::Uuid;
	use libc;

[greptile-apps]

style: These imports should be grouped more logically. Consider organizing them with std imports first, then external crates, and finally internal modules. Also, tracing and tracing_subscriber should be imported together.

[greptile-apps]

style: The unsafe block is necessary here, but it would be better to add a comment explaining why it's safe in this context - foundationdb::boot() initializes the FDB network and is designed to be called only once.

[greptile-apps]

style: The test doesn't verify the value returned from the get operation. Consider adding an assertion to check that the retrieved value matches the expected 'test_value'.

Suggested change

	tx.get(b"test_key", false).await?;
	let value = tx.get(b"test_key", false).await?;
	assert_eq!(value.as_ref().map(|v| v.as_ref()), Some(b"test_value".as_ref()));

[greptile-apps]

logic: Using a deprecated method with #[allow(deprecated)] is acknowledged in the comment, but this creates a potential memory leak since the VFS instance will be dropped after this function returns while SQLite still holds a pointer to it.