feat: Validator database #1614

sergerad · 2026-01-29T06:47:20Z

Context

The Validator currently only stores validated transactions in-memory. We want to persist validated transactions for resilience and potential future use cases around debugging / audit-ability w.r.t public transactions.

The Validator will be run as a separate instance to the node so it needs its own database, schema, etc.

Relates to #1316.

Changes

Add data directory CLI arg for Validator command.
Make various types and functions from Db crate pub for use in Validator crate.
Add sqlite, diesel scaffolding to Validator crate alongside new schema.
Fix standalone Validator setup with bundled components.
Rename insecure key variables to just validator key.
Merge Validator command args into ValidatorConfig struct.

crates/validator/src/db/migrations/2025062000000_setup/up.sql

…lidator-db

sergerad · 2026-02-01T23:01:26Z

@bobbinth I have validated these behaviours on this branch:

Validator as separate process works
Validator going down does not stop Node, can serve read requests
Node resumes normal functioning when Validator is brought back up

…lidator-db

crates/validator/Cargo.toml

crates/validator/src/server/mod.rs

crates/store/src/db/mod.rs

crates/validator/src/db/models.rs

bin/node/src/commands/mod.rs

bin/node/src/commands/validator.rs

crates/validator/src/block_validation/mod.rs

Mirko-von-Leipzig · 2026-02-02T07:52:43Z

crates/validator/src/db/mod.rs

+#[instrument(target = COMPONENT, skip_all)]
+pub async fn load(database_filepath: PathBuf) -> Result<miden_node_store::Db, DatabaseSetupError> {
+    let manager = ConnectionManager::new(database_filepath.to_str().unwrap());
+    let pool = deadpool_diesel::Pool::builder(manager).max_size(16).build()?;


Does max_size mean the capacity, or is the actual capacity chosen differently and this is just the upper bound to that algorithm?

An upper bound but I don't know what goes on under the hood. @drahnr do you know why we wouldn't just use default?

/// Maximum size of the [`Pool`]. /// /// Default: `cpu_count * 4` /// /// [`Pool`]: super::Pool pub max_size: usize,

No, I don't remember when or who introduced it

crates/validator/src/db/mod.rs

drahnr · 2026-02-02T13:30:39Z

crates/validator/src/db/mod.rs

+#[instrument(target = COMPONENT, skip_all)]
+pub async fn load(database_filepath: PathBuf) -> Result<miden_node_store::Db, DatabaseSetupError> {
+    let manager = ConnectionManager::new(database_filepath.to_str().unwrap());
+    let pool = deadpool_diesel::Pool::builder(manager).max_size(16).build()?;


I'd prefer to have our own builder pattern, with any options we use in practice.

Unsure which builder we would be adding here. Is it something that would be used across both store and validator? Could roll it up into this issue #1653

crates/validator/src/db/migrations.rs

crates/validator/src/db/schema.rs

…lidator-db

CHANGELOG.md

crates/validator/src/tx_validation/info.rs

Mirko-von-Leipzig · 2026-02-12T08:09:02Z

crates/validator/src/server/mod.rs

+        // Store the validated transaction.
+        self.db
+            .transact("insert_transaction", move |conn| insert_transaction(conn, &tx_info))
+            .await?;


We should ensure that this is instrumented (it should be, but async + closures is tricky)

2026-02-12T22:43:49.934198Z INFO rpc:validate_block:find_unvalidated_transactions: miden-validator: crates/validator/src/db/mod.rs:68: close, time.busy: 11.1ms, time.idle: 449µs rpc.service: "validator.Api", rpc.method: "SignBlock", otel.name: "validator.Api/SignBlock" tx_ids: [0x18f8ad1c9312257fe192507336a4b6590d2f0a92127506eb47738c77b18c258b, 0x7c9fed92d7bd767796a7d51f8e9a368bbf2b17a601ff471aa07dfc88bee7dcd1]

2026-02-12T22:55:33.672764Z INFO rpc:submit_proven_transaction:insert_transaction: miden-validator: crates/validator/src/db/mod.rs:39: close, time.busy: 6.56ms, time.idle: 35.0µs rpc.service: "validator.Api", rpc.method: "SubmitProvenTransaction", otel.name: "validator.Api/SubmitProvenTransaction" tx_id: 0x99eeffc42a2b6150e6e7fed88f86350e926fd52992911b02bd4162abec0e8670

Mirko-von-Leipzig · 2026-02-12T08:11:45Z

crates/validator/src/db/models.rs

+    pub fn new(info: &ValidatedTransactionInfo) -> Self {
+        Self {
+            id: info.tx_id().to_bytes(),
+            block_num: info.block_num().to_raw_sql(),
+            account_id: info.account_id().to_bytes(),
+            info: info.to_bytes(),
+        }


I think we should handle all the serialization here, instead of serializing in VAlidatedTransactionInfo as I allude to in this comment

LMK if my latest changes reflect your thinking here

crates/validator/src/block_validation/mod.rs

Mirko-von-Leipzig · 2026-02-12T08:32:04Z

crates/validator/src/db/mod.rs

+    // Convert TransactionIds to bytes for query.
+    let tx_id_bytes: Vec<Vec<u8>> = tx_ids.iter().map(TransactionId::to_bytes).collect();
+
+    // Query the database for matching transactions ids.
+    let raw_transaction_ids = schema::validated_transactions::table
+        .select(schema::validated_transactions::id)
+        .filter(schema::validated_transactions::id.eq_any(tx_id_bytes))
+        .order(schema::validated_transactions::id.asc())
+        .load::<Vec<u8>>(conn)
+        .map_err(DatabaseError::from)?;


This is probably not important at the moment, but since this will likely always remain a sqlite local database, I think we should just do an exists check in a loop? This would simplify the query, return values and output construction.

Though whether diesel supports this is unclear to me.

Unsure how many transactions we expect to have in the average block, but feels counterintuitive to scale # of db queries with # of transactions.

crates/validator/src/block_validation/mod.rs

drahnr · 2026-02-12T14:10:40Z

bin/node/src/commands/bundled.rs

        }

+        // Start the Validator if we have bound a socket.
+        if let Some(address) = validator_socket_address {


The configuration decision based on the validator_socket_address is a bit obfuscating the fact that the user did not provide a validator URL but a key

Not sure that part of the stack should be considering the key. Key presence is considered elsewhere. This part of the stack just cares about whether to run the local instance or not

…lidator-db

sergerad · 2026-02-13T02:04:53Z

crates/validator/src/db/mod.rs

+/// Open a connection to the DB and apply any pending migrations.
+#[instrument(target = COMPONENT, skip_all)]
+pub async fn load(database_filepath: PathBuf) -> Result<miden_node_store::Db, DatabaseSetupError> {
+    let manager = ConnectionManager::new(database_filepath.to_str().unwrap());


@Mirko-von-Leipzig @drahnr despite this (and the PRAGMA queries it makes for WAL etc) I still get db locked errors when running the integration tests.

ERROR rpc:submit_proven_transaction: miden-validator: crates/validator/src/server/mod.rs:126: error: status: 'Internal error', self: "database is locked" rpc.service: "validator.Api", rpc.method: "SubmitProvenTransaction", otel.name: "validator.Api/SubmitProvenTransaction"

Validator could receive parallel requests for submit_proven_transaction endpoint so I do think its something we need to solve (not just integration tests' fault).

Adding this does avoid it. ATM I'm reusing the store's connection manager code for validator.

pub(crate) fn configure_connection_on_creation( conn: &mut SqliteConnection, ) -> Result<(), ConnectionManagerError> { // Wait up to 5 seconds for writer locks before erroring. diesel::sql_query("PRAGMA busy_timeout=5000") .execute(conn) .map_err(ConnectionManagerError::ConnectionParamSetup)?;

sergerad added 2 commits January 29, 2026 14:10

Initial validator db impl

9ed9c7a

Add basic validation logic

560c79a

Mirko-von-Leipzig reviewed Jan 29, 2026

View reviewed changes

crates/validator/src/db/migrations/2025062000000_setup/up.sql Outdated Show resolved Hide resolved

sergerad added 5 commits January 30, 2026 10:47

Simplify schema

77b1616

Changelog, toml

0636937

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-va…

6f4b0d3

…lidator-db

Fix comment

71c90b8

Add error case

7bdd615

sergerad marked this pull request as ready for review January 29, 2026 22:47

sergerad requested review from Mirko-von-Leipzig, bobbinth and drahnr January 29, 2026 22:47

sergerad mentioned this pull request Jan 29, 2026

Guardrails: Tx and Block Validation #1316

Open

12 tasks

Fix standalone validator setup alongside bundled components

4e76c2e

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-va…

c01448e

…lidator-db

Mirko-von-Leipzig reviewed Feb 2, 2026

View reviewed changes

drahnr reviewed Feb 2, 2026

View reviewed changes

crates/validator/src/db/migrations.rs Show resolved Hide resolved

drahnr reviewed Feb 2, 2026

View reviewed changes

crates/validator/src/db/schema.rs Outdated Show resolved Hide resolved

sergerad added 10 commits February 3, 2026 11:21

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-va…

2b8008f

…lidator-db

RM row affected check

4fe9890

Fix comment

f3aaeba

Add UnvalidatedTransactions err

405737e

Rename insecure key

2d7a3ba

Move to kv store

d3c8770

Misc cleanup

1ec03ef

Undo pub changes to store crate

4735bd1

Changelog

147b0aa

Missing file

1596c50

sergerad added 6 commits February 9, 2026 11:33

Rm pub

025cc93

Add ValidatedTransactionInfo

6fa8f46

Missing file

3af9c0a

Fix comment

a6bf55d

Fix var name

a7a2455

Fix changelog

e60ab9b

sergerad requested review from Mirko-von-Leipzig and bobbinth and removed request for bobbinth February 9, 2026 00:03

sergerad added 6 commits February 11, 2026 07:50

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-va…

28ca16f

…lidator-db

Toml

adb60b3

Revert dockerfile

00e45be

Add block_num col and index + ValidatedTransactionInfoBlob

5ca7001

Fix comment

0081637

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-va…

c341458

…lidator-db

Mirko-von-Leipzig reviewed Feb 12, 2026

View reviewed changes

drahnr reviewed Feb 12, 2026

View reviewed changes

sergerad added 7 commits February 13, 2026 08:35

Update changelog wording

f3d1cfd

Refactor validated tx info

7937a7f

Source errors

65b5af0

Missing file

61a9548

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-va…

d032732

…lidator-db

Instrument for errors, on conflict do nothing, WAL and timeout

806ec05

Instrument db fns

959f7e0

sergerad requested review from Mirko-von-Leipzig and drahnr February 12, 2026 23:17

RM unnecessary src

528a265

sergerad commented Feb 13, 2026

View reviewed changes

Add 5s pragma

4f46477

feat: Validator database #1614

Are you sure you want to change the base?

feat: Validator database #1614

Uh oh!

Conversation

sergerad commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Changes

Uh oh!

Uh oh!

sergerad commented Feb 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sergerad commented Jan 29, 2026 •

edited

Loading