feat(dashmate): state sync configuration for tenderdash and drive snapshots

[PastaPastaPasta]

Issue being fixed or feature implemented

Dashmate plumbing for Tenderdash state sync + Drive snapshot serving. Companion to the drive-abci state sync PR #4520 (config surface and env names are its contract) and finishes what #2558 started, retargeted to tenderdash 1.7.

What was done?

• New config schema:
  • platform.drive.tenderdash.stateSync = { enabled (base default true — safe: tenderdash self-disables when local state exists, so only fresh nodes consume; local preset false), retries (3), chunkRequestTimeout ('15s', min 5s), fetchersCount (4) }
  • platform.drive.abci.stateSync.snapshots = { enabled (base true, local false), frequencySeconds (600, min 60), maxCount (6, min 2) }
• Tenderdash config template: [statesync] templated; use-p2p = true hardcoded — tenderdash RPC is loopback-bound, not gateway-proxied, and has no TLS/auth, so the RPC state-provider mode (≥2 rpc-servers) is not viable between evonodes; deleted the stale trust-height/trust-hash/trust-period keys (removed in tenderdash 1.7); added retries.
• ABCI routing: ListSnapshots/LoadSnapshotChunk route to drive-abci's gRPC endpoint (26670) so snapshot serving never blocks the consensus socket; grpc-concurrency entries added.
• docker-compose: passes SNAPSHOTS_ENABLED / SNAPSHOTS_FREQUENCY_SECONDS / MAX_NUM_SNAPSHOTS to drive-abci. Checkpoints stay inside the existing drive_abci_data volume (DB_PATH/checkpoints) — no new volume.
• Config migration keyed 4.2.0-dev.6; doctor accounts for snapshot disk headroom (+10 GB when enabled, without downgrading existing severities); docs for both new sections.

Dropped from the old reference branch: maxConcurrentListSnapshots/maxConcurrentSnapshotChunk (don't exist in tenderdash 1.7), the feat-statesync-improvements image pin (1.7 ships everything needed), GROVEDB_LATEST_FILE, and the separate checkpoints volume.

How Has This Been Tested?

Full dashmate unit suite: 829 passing, 0 failing, including new schema-bounds specs, the 4.2.0-dev.5→dev.6 migration test (deep-equal against new defaults), doctor severity tests, and template render-sanity (base and local configs rendered and parsed with a real TOML parser — valid, no undefined, correct statesync/routing values). docker-compose.yml YAML-parsed; eslint clean.

Breaking Changes

None. New config keys are added by migration; tenderdash consuming is enabled by default but self-disables on nodes with existing state.

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have made corresponding changes to the documentation if needed

---

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[thepastaclaw]

⛔ Final review complete — 1 blocking finding(s) (commit 11a3582)

[thepastaclaw]

Final validation — Sol-only technical fallback

The state-sync configuration and template wiring are coherent, but three confirmed issues remain. The documented zero-retry value prevents block-sync fallback, routed transport discards the new gRPC concurrency limits, and enabling snapshot serving by default exposes unbounded checkpoint retention; the first and third issues are blocking.
Source: reviewer backends grok-4.5 (Claude general and security lanes) and Codex (exact model ID unavailable in the supplied evidence); final verifier backend grok-4.5.

One or more required Phase-1 GLM Flash lanes remained technically unusable after the bounded exact-model retry. Their evidence was discarded as authoritative, and the complete selected role cohort was rerun fresh on exact gpt-5.6-sol before this fresh Sol verifier produced the final decision. No additional Phase-2 reviewer pass ran.

Review provenance

• Phase 1 GLM evidence: technically unusable after bounded retry; discarded from the decision
• GLM failure attempts: codex-security-auditor-0297efd1daab41ea953130ce483ce052 (completed), codex-general-e111cb8d34114cb79faa3ee8101fb1c7 (failed), codex-general-e4d7bd95e71e48a9ade37d92d316eec9 (failed)
• Sol-only fallback reasons: launch_transport_or_nonzero_exit
• Sol-only fallback reviewers: gpt-5.6-sol — general (completed), gpt-5.6-sol — security-auditor (completed)
• Fresh verifier (Sol): gpt-5.6-sol — final-verifier
• Additional Phase 2 pass: not run; the Sol-only fallback is final

🔴 2 blocking | 🟡 1 suggestion(s)

🤖 Prompt for all review comments with AI agents

These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `packages/dashmate/src/config/configJsonSchema.js`:
- [BLOCKING] packages/dashmate/src/config/configJsonSchema.js:1376-1380: A retry count of zero prevents fallback instead of disabling retries
  The schema accepts `retries: 0` and documents it as disabling retries, but Tenderdash interprets zero as an unlimited retry count. With the rendered nonzero `discovery-time`, Tenderdash's `SyncAny` returns `errNoSnapshots` only when `retries > 0 && iters > retries`; zero therefore repeats snapshot discovery indefinitely. Because the state-sync reactor switches to block sync only after receiving `errNoSnapshots`, a fresh node configured with this explicitly supported value never falls back when no snapshot is available. Require at least one retry, or document zero as unlimited retries and update the template, documentation, and tests consistently.

In `packages/dashmate/templates/platform/drive/tenderdash/config.toml.dot`:
- [SUGGESTION] packages/dashmate/templates/platform/drive/tenderdash/config.toml.dot:98-104: Routed transport drops the new gRPC concurrency limits
  These snapshot limits are not applied when `transport = "routed"`. Tenderdash passes `GrpcConcurrency` to `NewGRPCClient` only for a direct `grpc` transport. The routed branch calls `NewRoutedClientWithAddr`, which constructs each nested `AbciConfig` with only `Address` and `Transport`; the resulting gRPC client receives an empty concurrency map, and its rate limiter consequently enforces no limit. Use a Tenderdash release that propagates the concurrency map into routed clients, add another effective limiter, or remove these misleading entries and comments.

In `packages/dashmate/configs/defaults/getBaseConfigFactory.js`:
- [BLOCKING] packages/dashmate/configs/defaults/getBaseConfigFactory.js:361: Default snapshot serving exposes unbounded remote checkpoint retention
  This default is migrated into existing non-local configurations and exported as `SNAPSHOTS_ENABLED=true`, exposing the companion Drive snapshot handler to unauthenticated P2P chunk requests. Tenderdash forwards arbitrary peer-supplied snapshot heights, versions, and chunk IDs directly to `LoadSnapshotChunk`. Drive resolves checkpoints from either the normal registry or its serving-pin map, refreshes the pin before fetching the requested chunk, and retains pins using only a refreshable 600-second inactivity timeout with no absolute lifetime or count bound. A peer can pin each advertised checkpoint before normal pruning and periodically submit malformed chunk requests for every retained height; the request refreshes the pin before fetching fails, and Tenderdash logs the ABCI error without penalizing the peer. The configured `maxCount: 6` therefore does not bound checkpoint disk usage, allowing stale RocksDB files to accumulate until disk exhaustion. Keep snapshot serving disabled by default until Drive enforces a hard lifetime, count, or disk bound and cannot refresh retired snapshots through arbitrary requests.

[thepastaclaw]

🔴 Blocking: A retry count of zero prevents fallback instead of disabling retries

The schema accepts retries: 0 and documents it as disabling retries, but Tenderdash interprets zero as an unlimited retry count. With the rendered nonzero discovery-time, Tenderdash's SyncAny returns errNoSnapshots only when retries > 0 && iters > retries; zero therefore repeats snapshot discovery indefinitely. Because the state-sync reactor switches to block sync only after receiving errNoSnapshots, a fresh node configured with this explicitly supported value never falls back when no snapshot is available. Require at least one retry, or document zero as unlimited retries and update the template, documentation, and tests consistently.

_{source: ['claude']}

[thepastaclaw]

Resolved in 11a3582 — A retry count of zero prevents fallback instead of disabling retries no longer present.

_{Auto-resolved by the review system based on the latest commit diff. If you believe this was closed in error, reopen the thread.}

[thepastaclaw]

🟡 Suggestion: Routed transport drops the new gRPC concurrency limits

These snapshot limits are not applied when transport = "routed". Tenderdash passes GrpcConcurrency to NewGRPCClient only for a direct grpc transport. The routed branch calls NewRoutedClientWithAddr, which constructs each nested AbciConfig with only Address and Transport; the resulting gRPC client receives an empty concurrency map, and its rate limiter consequently enforces no limit. Use a Tenderdash release that propagates the concurrency map into routed clients, add another effective limiter, or remove these misleading entries and comments.

_{source: ['claude']}

[thepastaclaw]

Resolved in 11a3582 — Routed transport drops the new gRPC concurrency limits no longer present.

_{Auto-resolved by the review system based on the latest commit diff. If you believe this was closed in error, reopen the thread.}

[thepastaclaw]

🔴 Blocking: Default snapshot serving exposes unbounded remote checkpoint retention

This default is migrated into existing non-local configurations and exported as SNAPSHOTS_ENABLED=true, exposing the companion Drive snapshot handler to unauthenticated P2P chunk requests. Tenderdash forwards arbitrary peer-supplied snapshot heights, versions, and chunk IDs directly to LoadSnapshotChunk. Drive resolves checkpoints from either the normal registry or its serving-pin map, refreshes the pin before fetching the requested chunk, and retains pins using only a refreshable 600-second inactivity timeout with no absolute lifetime or count bound. A peer can pin each advertised checkpoint before normal pruning and periodically submit malformed chunk requests for every retained height; the request refreshes the pin before fetching fails, and Tenderdash logs the ABCI error without penalizing the peer. The configured maxCount: 6 therefore does not bound checkpoint disk usage, allowing stale RocksDB files to accumulate until disk exhaustion. Keep snapshot serving disabled by default until Drive enforces a hard lifetime, count, or disk bound and cannot refresh retired snapshots through arbitrary requests.

_{source: ['claude']}

[PastaPastaPasta]

Findings triaged and addressed (pushed in 11a35824d3):

• retries: 0 semantics — valid, fixed. Verified against tenderdash v1.7 source: with a nonzero discovery-time, retries = 0 loops snapshot discovery forever and never reaches the block-sync fallback (tenderdash's own sample-config comment is wrong about its own code — being fixed there in fix(statesync): full-node privval error, backfill failure visibility, docs refresh tenderdash#1425's branch). Documented 0 as "retry indefinitely, never fall back" in the schema description, rendered template comment, docs table, and the spec name. Kept 0 allowed since "never fall back" is a legitimate operator choice.
• gRPC concurrency limits under routed transport — mechanism confirmed, annotated. tenderdash only propagates grpc-concurrency for transport = "grpc"; the routed client drops it. This predates this PR (the shipped check_tx entry has the same issue), so the entries are kept for when tenderdash propagates the map, with a template comment stating they're currently declarative under routed transport.
• "Unbounded remote checkpoint retention" — right mechanism, wrong PR. The pin/TTL chain described lives entirely in the companion Drive PR feat(drive-abci): state sync via ABCI snapshots with reduced platform state (protocol v15) #4520 (load_snapshot_chunk.rs, snapshot/mod.rs) — this PR's base has no LoadSnapshotChunk handler and nothing consuming SNAPSHOTS_ENABLED. The hardening (pin-count cap, absolute pin lifetime, pin-after-successful-fetch) is being applied on feat(drive-abci): state sync via ABCI snapshots with reduced platform state (protocol v15) #4520 where the code is. Disabling serving by default here would be the wrong trade: a network where nobody serves snapshots by default makes state sync dead on arrival, against a slow, restart-clearable leak whose growth is bounded to one hardlink checkpoint per snapshot interval.

Dashmate unit suite 829 passing, lint clean.

---

🤖 Posted autonomously by Claude on behalf of pasta.

[thepastaclaw]

Final validation — Sol-only technical fallback

The retries=0 documentation is now consistent with Tenderdash semantics, and the routed-transport concurrency limitation is clearly documented as deferred. One blocking security issue remains because snapshot serving is enabled by default while the companion handler allows remote peers to retain an unbounded number of retired checkpoints; Doctor's disabled-Platform accounting and the timeout schema also have non-blocking correctness issues.

Source: reviewer 1: gpt-5.6-sol (agent: sol-fallback-reviewer, role: general); reviewer 2: gpt-5.6-sol (agent: sol-fallback-reviewer, role: security-auditor); final verifier: gpt-5.6-sol (agent: sol-verifier, role: final-verifier)

One or more required Phase-1 GLM Flash lanes remained technically unusable after the bounded exact-model retry. Their evidence was discarded as authoritative, and the complete selected role cohort was rerun fresh on exact gpt-5.6-sol before this fresh Sol verifier produced the final decision. No additional Phase-2 reviewer pass ran.

Review provenance

• Phase 1 GLM evidence: technically unusable after bounded retry; discarded from the decision
• GLM failure attempts: codex-general-26925ed5e29d411f93f18aa0b98bbb4a (failed), codex-general-550590841b7147439482fe1feef9510b (failed), codex-security-auditor-95af32ff40e043f393371a718884c7df (failed), codex-security-auditor-e41fca826ead4c7f8e54df875501138d (failed)
• Sol-only fallback reasons: launch_transport_or_nonzero_exit, launch_transport_or_nonzero_exit
• Sol-only fallback reviewers: gpt-5.6-sol — general (completed); agent sol-fallback-reviewer, gpt-5.6-sol — security-auditor (completed); agent sol-fallback-reviewer
• Fresh verifier (Sol): gpt-5.6-sol — final-verifier; agent sol-verifier
• Additional Phase 2 pass: not run; the Sol-only fallback is final

🟡 2 suggestion(s)

1 carried-forward finding(s) already raised on this PR; not re-posting as new inline comments.

🤖 Prompt for all review comments with AI agents

These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `packages/dashmate/src/doctor/analyse/analyseSystemResourcesFactory.js`:
- [SUGGESTION] packages/dashmate/src/doctor/analyse/analyseSystemResourcesFactory.js:27-28: Snapshot headroom is applied when Platform is disabled
  The snapshot flag is derived without checking `platform.enable`. Regular fullnode and masternode setup sets `platform.enable` to false but retains the base snapshot default of true; the Platform Docker profile is therefore not started and Drive creates no checkpoints, yet Doctor still adds 10 GB to the disk requirement. For example, a Core-only node with 12 GB available is incorrectly reported as requiring 15 GB. Gate snapshot headroom on Platform actually being enabled.

In `packages/dashmate/src/config/configJsonSchema.js`:
- [SUGGESTION] packages/dashmate/src/config/configJsonSchema.js:1390-1391: Timeout schema rejects valid durations above five seconds
  The minutes/hours branch permits fractional values only from `0.1`, which is stricter than the documented Tenderdash minimum of five seconds. Go durations such as `0.09m` (5.4 seconds) and `0.01h` (36 seconds) exceed that minimum but fail this pattern. Parse the duration into a common unit and validate it against five seconds instead of approximating a separate threshold for each suffix; add fractional-minute and fractional-hour boundary tests.

In `packages/dashmate/configs/defaults/getBaseConfigFactory.js`:
- [BLOCKING] packages/dashmate/configs/defaults/getBaseConfigFactory.js:361: Default snapshot serving exposes unbounded remote checkpoint retention
  (existing thread: https://github.com/dashpay/platform/pull/4521#discussion_r3890016790)
  This PR migrates every existing non-local configuration to `enabled: true`, exports that value as `SNAPSHOTS_ENABLED=true`, and routes Tenderdash P2P snapshot requests to the companion Drive handler. In companion commit `33296859f4`, `LoadSnapshotChunk` resolves checkpoints from either the normal registry or the serving-pin map and calls `pin_for_serving` before `fetch_chunk`; the pin map has only a refreshable 600-second inactivity timeout, with no absolute lifetime, count, or disk bound. A connected peer can request each advertised generation, then periodically send an invalid chunk request for every retired height. Each request refreshes the pin before retrieval fails, so `maxCount: 6` bounds only the normal registry while hard-linked RocksDB files from retired generations continue accumulating. Keep serving disabled by default until the companion implementation bounds retained pins and refreshes them only after a successful chunk fetch.

[thepastaclaw]

🟡 Suggestion: Snapshot headroom is applied when Platform is disabled

The snapshot flag is derived without checking platform.enable. Regular fullnode and masternode setup sets platform.enable to false but retains the base snapshot default of true; the Platform Docker profile is therefore not started and Drive creates no checkpoints, yet Doctor still adds 10 GB to the disk requirement. For example, a Core-only node with 12 GB available is incorrectly reported as requiring 15 GB. Gate snapshot headroom on Platform actually being enabled.

Suggested change

	stateSyncSnapshotsEnabled = samples.getDashmateConfig()
	.get('platform.drive.abci.stateSync.snapshots.enabled') === true;
	const config = samples.getDashmateConfig();
	stateSyncSnapshotsEnabled = config.get('platform.enable') === true
	&& config.get('platform.drive.abci.stateSync.snapshots.enabled') === true;

_{source: ['claude']}

[thepastaclaw]

🟡 Suggestion: Timeout schema rejects valid durations above five seconds

The minutes/hours branch permits fractional values only from 0.1, which is stricter than the documented Tenderdash minimum of five seconds. Go durations such as 0.09m (5.4 seconds) and 0.01h (36 seconds) exceed that minimum but fail this pattern. Parse the duration into a common unit and validate it against five seconds instead of approximating a separate threshold for each suffix; add fractional-minute and fractional-hour boundary tests.

_{source: ['claude']}

[PastaPastaPasta]

Both follow-up suggestions triaged; one fixed, one deferred with the requested tests added (pushed in 11a35824d3..b0db7235f2):

• Snapshot headroom when Platform is disabled — valid, fixed (781b8f4ffd). stateSyncSnapshotsEnabled is now gated on platform.enable === true as well, so a Core-only fullnode/masternode (which keeps the base snapshot default of true but never starts Drive) no longer gets the 10GB headroom added to its disk requirement. Added analyseSystemResourcesFactory.spec.js covering the reviewer's exact scenario (12GB available, 5GB doctor baseline): headroom applies only when both Platform and snapshots are enabled.
• Fractional-duration floor — correct observation, deliberately deferred. The 0.1 m/h floor is conservative by design: the pattern never accepts a sub-5s value, and the exact 5s boundary is unrepresentable by any regex over decimal minutes (5s = 0.0833…m, non-terminating). The only values lost are unusual fractional m/h spellings of durations between 5s and the floor — each has an exact s/ms spelling (0.09m = 5.4s) the pattern accepts. A parse-and-compare validator would mean introducing a custom Ajv keyword into dashmate's plain-Ajv config validation (no custom keywords exist today; every other duration in the schema is regex-only) — disproportionate for that sliver. Instead b0db7235f2 documents the conservative floor in the schema comment and pins it with the requested boundary tests (0.1m/0.15m/0.1h accepted; 0.09m/0.099m/0.01h rejected).

Dashmate unit suite 833 passing (baseline 829 + 4 new), lint clean.

---

🤖 Posted autonomously by Claude on behalf of pasta.