Skip to content

Bounded wall-clock fill timeout for WitnessSigner's BatchTail (nightly research) - #1165

Draft
ruvnet wants to merge 5 commits into
mainfrom
claude/focused-darwin-rxgets
Draft

ruvnet wants to merge 5 commits into
mainfrom
claude/focused-darwin-rxgets

Conversation

@ruvnet

@ruvnet ruvnet commented Oct 11, 2026

Copy link
Copy Markdown
Owner

Summary

Nightly research run (2026-10-11) that implements the specific gap the previous nightly (ADR-347) deferred: ruvector-agent-memory's SignedWitnessSink::SigningStrategy::BatchTail has no bound on how long a partially-filled signing span can stay unauthenticated if arrivals stop. ADR-347's own "Alternatives Considered" ("Wall-clock BatchTail fill timeout. Deferred.") and Open Question 1 named this directly.

  • Adds SigningStrategy::BatchTailTimeout { batch_size, max_wait_ns }, which closes a pending span at batch_size records or after max_wait_ns since the oldest unsealed record, whichever comes first — porting the hybrid count-or-timeout policy ruvector-retrieval-receipt::batch_fill already validated (ADR-343), adapted natively to SignedWitnessSink rather than added as a new dependency.
  • Purely additive: PerRecord/BatchTail are byte-for-byte unchanged; no new Cargo dependency, no new crate.
  • The adversarial-review pass caught a real ordering bug before any benchmark ran (checking the timeout after merging a new arrival into a stale span, instead of before) — fixed and covered by a dedicated regression test plus a randomized property test.

Architecture

TransactionalLedger.add/accept
        -> WitnessSink::emit_batch
              -> SignedWitnessSink::emit_batch_at(records, now_ns)
                    -> [BatchTailTimeout] check_timeout_at(now_ns)  (seal stale span FIRST)
                    -> merge into pending span, count-close if full
        -> inner: MemoryWitnessLog
periodic tick (ruFlo workflow / host scheduler) -> check_timeout()

Files changed

  • crates/ruvector-agent-memory/src/witness_signing.rs — core implementation (new strategy variant, emit_batch_at/check_timeout_at/check_timeout/next_timeout_deadline_ns/strategy, WitnessSignerError::ZeroMaxWait).
  • crates/ruvector-agent-memory/src/witness_signing_tests.rs — 10 new tests (45 → 55 in the crate), including a randomized bounded-latency property test.
  • crates/ruvector-agent-memory/examples/witness_signing_timeout_bench.rs — new dedicated benchmark (dense-burst overhead + sparse-arrival bounded-latency, both real wall-clock measurements).
  • docs/adr/ADR-348-bounded-witness-batch-fill-timeout.md + docs/adr/INDEX.md — design record.
  • docs/research/nightly/2026-10-11-bounded-witness-batch-timeout/{README.md,gist.md} — full nightly report (capability discovery, benchmark evidence, rejected alternatives, Darwin-style lineage, applications, falsification criteria).

Benchmark command

cargo run --release -p ruvector-agent-memory --example witness_signing_timeout_bench

Real benchmark results (3 runs)

Dense-burst overhead (20,000 add+accept pairs, batch_size=64, no idle gaps; threshold ±10%):

Run BatchTail{64} mean BatchTailTimeout{64,5ms} mean Overhead Signatures
1 3.387µs 3.340µs -1.39% 625 / 625
2 3.448µs 3.441µs -0.19% 625 / 625
3 3.635µs 3.377µs -7.11% 625 / 625

Sparse-arrival worst-case latency (20-record burst, real 15ms idle gap, max_wait_ns=5ms, 1ms poll; bound = 6ms):

Run BatchTail pending after gap BatchTailTimeout sealed after
1 20 (unchanged, no timeout concept) 5.585ms
2 20 (unchanged, no timeout concept) 5.527ms
3 20 (unchanged, no timeout concept) 5.653ms

Every run printed an explicit Acceptance: ACCEPT from the benchmark binary itself.

Full regression: cargo test -p ruvector-agent-memory --lib → 55/55 passing. cargo clippy -p ruvector-agent-memory --all-targets -- -D warnings → clean. cargo fmt -p ruvector-agent-memory -- --check → clean.

Acceptance result

ACCEPT

Darwin / Flywheel / MetaHarness

Checked before assuming availability: npx metaharness --help resolves (a scaffolding generator, not a runtime for this repo), npx ruvector harness doctor/darwin/flywheel does not resolve an executable in this checkout. Documented honestly in the nightly README rather than fabricating tool output; this run used a manual equivalent (explicit hypothesis, real baseline, bounded one-candidate evolution, pre-declared fitness gates, retained rejected alternatives in the ADR).

Security review

No new cryptographic surface — a timeout-forced seal produces the same SignPurpose::BatchTail-tagged span a count-triggered seal would, indistinguishable to a verifier. Wall clock is Instant-based (monotonic, immune to system-clock adjustment). "No witness, no mutation" ordering preserved. Full detail in ADR-348's Security/Governance/Failure Modes sections.

Main limitations

  • The bound is caller-enforced: a periodic check_timeout tick must actually be wired up in production, or BatchTailTimeout silently degrades to BatchTail's original behavior. This is documented as an operational obligation, not hidden.
  • No durable (non-MemoryWitnessLog) sink, multi-issuer, or WASM/edge measurement — explicitly out of scope, listed as open items.
  • Benchmark figures are from one shared, containerized workstation (3 repetitions); treat as indicative.

Production recommendation

Land as-is (additive, opt-in, zero behavior change for existing callers). Before relying on the bound in a real deployment: pick a max_wait_ns default, build the described ruFlo periodic-tick workflow, and extend verification to a durable sink.


Generated by Claude Code

claude and others added 5 commits October 11, 2026 07:38
Add SigningStrategy::BatchTailTimeout { batch_size, max_wait_ns }, which
closes a pending signing span at batch_size records or after max_wait_ns
since the oldest unsealed record, whichever comes first - ported from
ruvector-retrieval-receipt::batch_fill's hybrid policy (ADR-343), not
reused as a dependency, since SignedWitnessSink commits spans through
the WitnessSink trait rather than a standalone scheduler.

Resolves the gap ADR-347 deferred ("No wall-clock fill timeout in this
pass") and its Open Question 1: under plain BatchTail, a partially
filled span has no bound on how long it can stay unauthenticated if
arrivals stop.

Adds emit_batch_at/check_timeout_at/check_timeout/
next_timeout_deadline_ns/strategy. Checks for an already-stale pending
span before merging a new arrival into it, not after - merging first
would silently stretch a late arrival's own wait past max_wait_ns.
Purely additive: PerRecord and BatchTail are unchanged.

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_015NaMrxfn7UW7J5ps5Uf2TV
…ions

10 new tests (45 -> 55 in the crate): construction errors, dense-arrival
equivalence with plain BatchTail, deadline boundary cases, the
opportunistic-seal-on-late-arrival ordering fix, honest-chain
verification with forced partial seals, and a randomized property test
(8 trials x 20 steps, xorshift64-seeded) asserting observed pending age
never exceeds max_wait_ns + poll_tick while the signed chain still
verifies end to end.

Trial/step counts are kept modest: Ed25519 signing in this sandbox's
unoptimized debug build runs far slower than release mode, and a larger
sweep turned this from a unit test into a multi-minute run; the
dedicated benchmark example covers the release-mode, larger-N regime.

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_015NaMrxfn7UW7J5ps5Uf2TV
New cargo example (witness_signing_timeout_bench) with two real,
wall-clock measurements, each with an explicit numeric acceptance gate
defined before running:

1. Dense-burst overhead: 20,000 add+accept pairs, no idle gaps,
   BatchTail{64} vs BatchTailTimeout{64, 5ms}. Gate: mean-latency and
   signature-count delta within +/-10%.
2. Sparse-arrival worst-case latency: a 20-record burst well under
   batch_size, then a real 15ms idle gap. Gate: BatchTail leaves the
   burst permanently pending with no timeout concept at all;
   BatchTailTimeout force-seals within max_wait_ns + poll_interval.

Three runs: both gates PASS on every run (dense overhead -1.4%/-0.2%/
-7.1%, i.e. no regression; sparse force-seal at 5.53-5.65ms against a
6ms bound). Prints an explicit ACCEPT/REJECT line.

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_015NaMrxfn7UW7J5ps5Uf2TV
Records the design, guarantees, limits, alternatives considered,
security/governance analysis, migration/rollback path, and rejection
criteria for SigningStrategy::BatchTailTimeout. Answers ADR-347's Open
Question 1 and its deferred "wall-clock BatchTail fill timeout"
alternative directly.

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_015NaMrxfn7UW7J5ps5Uf2TV
Hypothesis, capability discovery (MetaHarness/Darwin/Flywheel checked
and found not wired into this checkout - documented honestly rather
than assumed), architecture diagram, raw benchmark evidence across
three runs, rejected alternatives, manual Darwin-style lineage, ruFlo
production-path workflow, practical and long-horizon applications, and
falsification criteria for the 2026-10-11 nightly run.

Acceptance: ACCEPT.

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_015NaMrxfn7UW7J5ps5Uf2TV

ruvnet commented Oct 11, 2026

Copy link
Copy Markdown
Owner Author

CI: npm audit (npm/ workspace) is failing, but not because of this PR

The supply-chain workflow's npm audit (npm/ workspace) job (run) fails on this PR's head commit with 9 vulnerabilities (5 moderate, 3 high, 1 critical) in transitive node_modules dependencies: @modelcontextprotocol/sdk, axios, compression, fast-copy, ip-address, proxy-addr, and qs (via body-parser/express).

This PR's diff is Rust (crates/ruvector-agent-memory) and Markdown docs only — it touches no package.json, lockfile, or node_modules content. I confirmed the same job fails identically on main itself at its current head (6191f8f66cdb132cbb676fa94571f87c1e8de308, run), with the same 9 advisories — so this is a pre-existing finding on the base branch, not something introduced here.

I checked for an existing fix PR to port in (there have been several past PRs fixing npm audit findings, e.g. #739, #682, #474, #346, all closed/merged already) but none currently open addresses this specific set of advisories, so there's nothing to port. Resolving it would mean running npm audit fix / bumping dependency versions across the npm/ workspace — out of scope for this Rust-focused research PR, and I'd rather not widen this diff with an unrelated dependency upgrade.

Not re-running the job: this is a deterministic audit finding (reproducible on main), not a flake, so a re-run wouldn't change the result. Leaving this as a known, pre-existing, unrelated failure unless maintainers want it addressed separately.


Generated by Claude Code

ruvnet commented Oct 11, 2026

Copy link
Copy Markdown
Owner Author

CI: Tests (core-and-rest) is failing, also not caused by this PR

Tests (core-and-rest) fails at the link step (run) building ruvector-typesafe-train's smoke_train test binary: rust-lld: error: undefined symbol: __isoc23_strtol / __isoc23_strtoull / __isoc23_strtoll plus a libstdc++ _M_replace_cold symbol, linking the prebuilt libort_sys (ONNX Runtime) archive. This is a glibc/libstdc++ ABI mismatch between the ONNX Runtime binary and the Ubuntu 22.04 runner's toolchain — nothing to do with ruvector-agent-memory or this PR's diff.

Confirmed pre-existing: the identical job (run) fails the same way on main's current head (6191f8f66cdb132cbb676fa94571f87c1e8de308), with byte-for-byte the same undefined symbols in the same libort_sys archive.

A fix exists but hasn't landed on main yet: open PR #1114 diagnosed this exact failure (citing the same symbols) and validated that moving the core-and-rest job to an ubuntu-24.04 runner resolves it (all 3,827 tests passed after that change, per that PR's own description). That fix is bundled inside #1114's much larger, unrelated, unmerged change (a tenant-bound MicroLoRA runtime feature) as a shared-workflow-file edit, not available as a small standalone patch to cherry-pick. Changing a shared CI workflow's runner OS for all PRs is a disproportionate, repo-wide infrastructure change for this narrow Rust-only research PR to carry unilaterally, so I'm not porting it here — recommend it land via #1114 or a dedicated CI-infra PR instead.

Not re-running: deterministic linker failure, reproduces identically on main, not a flake.

(Tests (core-platform) and Tests (core-and-rest-heavy) were still in progress as of this comment; per PR #1114's description they hit the identical ort_sys link failure on the same runner class, so if they complete red here too, same root cause and conclusion applies.)


Generated by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants