feat: featureExport / featureConsensusEntropy#693
Draft
sublimator wants to merge 254 commits intodevfrom
Draft
Conversation
Co-authored-by: tequ <git@tequ.dev>
Co-authored-by: tequ <git@tequ.dev>
Co-authored-by: tequ <git@tequ.dev>
Co-authored-by: tequ <git@tequ.dev>
Port the UNL Validator Transaction (UVTxn) pattern from the RNG feature to allow validators to submit signed ttEXPORT_SIGN transactions without requiring a funded account. Changes: - Add isUVTx() to identify UVTxn transaction types - Add inUNLReport() templates to check validator UNLReport membership - Add getValidationSecretKey() to Application for signing - Modify Transactor for UVTxn bypasses (fee, seq, signature checks) - Add makeExportSignTxns() to generate validator signatures - Hook into RCLConsensus to submit ttEXPORT_SIGN during accept - Update applySteps.cpp routing for ttEXPORT_SIGN - Remove direct ttEXPORT_SIGN injection from TxQ::accept Note: Currently uses Change transactor with UVTx branches. May refactor to dedicated ExportSign transactor class.
Move ttEXPORT_SIGN handling to dedicated ExportSign transactor class, following the same pattern as ttENTROPY/Entropy from the RNG feature. UVTxns (signed validator transactions) should not be mixed with pseudo-transactions in the Change transactor. - Create ExportSign.h/cpp with preflight, preclaim, doApply - Route ttEXPORT_SIGN through ExportSign in applySteps.cpp - Remove UVTx branches from Change transactor - Add documentation markers to View.h for inUNLReport functions
- Fix xport hook API whitelist to declare 4 args (I32, I32, I32, I32) instead of 2, matching the actual implementation signature - Fix TxQ.cpp to use emplace_back with STObject for sfExportedTxn instead of setFieldVL, since sfExportedTxn is OBJECT type not VL. The previous code would throw "Wrong field type" at runtime.
- fix Guard.h: add import_whitelist_2 to signature lookup chain (was causing "Function type is inconsistent" errors for xport APIs) - fix InvariantCheck.cpp: add ltEXPORTED_TXN to valid ledger entry types (was causing "invalid ledger entry type added" invariant failures) - add SetHook.cpp: TODO comment documenting API version confusion - add Export_test.cpp: comprehensive test suite for export feature - testBasicSetup: verify hook installation works - testEmitPayment: verify emit() flow works - testXportPayment: verify xport() creates ltEXPORTED_TXN - includes DebugLogs helper for per-partition log levels - parameterized runXportTest helper for future validator tests Note: validator signing flow (ttEXPORT_SIGN) still needs debugging - causes internal error on env.close() when validator config enabled.
adds step-by-step trace logging with [EXPORT-TRACE] prefix to track the complete export transaction lifecycle: - STEP-1: xport() creates ltEXPORTED_TXN - STEP-2a: rawTxInsert ttEXPORT_SIGN in callback - STEP-2b: doApply ttEXPORT_SIGN - STEP-3a: rawTxInsert ttEXPORT - STEP-4: doApply ttEXPORT (cleanup) filter with: grep '\[EXPORT-TRACE\]'
Replace on-ledger ttEXPORT_SIGN transactions with ephemeral signature collection via TMValidation messages. This eliminates O(n²) metadata bloat from accumulating signatures on-ledger. Changes: - Add ExportSignatureCollector for in-memory signature storage with quorum tracking (80% UNL threshold) - Extend TMValidation protobuf with exportSignatures field - Sign pending exports during validate() and broadcast via validation - Extract signatures from received TMValidation in PeerImp - TxQ checks quorum from memory instead of ledger - Inject ttEXPORT when quorum reached (can be ledger N+1 or N+2) - Clean up collector after ttEXPORT processed Includes [EXPORT-TIMING] debug logging for timing analysis.
Validators now sign ALL pending ltEXPORTED_TXN entries every ledger (not just those from the current ledger). Signatures are cached in ExportSignatureCollector and re-broadcast until the export is finalized. Changes: - Add hasSignatureFrom() and getSignatureFrom() to collector for checking/retrieving cached signatures - signPendingExports() now iterates ALL pending exports, uses cached signature if available, otherwise signs fresh - Signatures keep broadcasting until ltEXPORTED_TXN is deleted This ensures: - Late validators can contribute (sign when they come online) - Network partitions self-heal (signatures propagate on reconnect) - Node restarts recover (re-sign from ledger state) The ltEXPORTED_TXN acts as a "ticket" - signatures only valid while it exists. No explicit expiry check needed; ledger state is the gatekeeper.
- remove DBG_EXPORT macros and all usages - remove [EXPORT-TRACE] and [EXPORT-TIMING] debug prefixes - adjust log levels (verbose logs to trace, summaries to debug) - upgrade "quorum reached" to info level (important event) - standardize log prefixes to use "Export:" - re-enable relay loop in OpenLedger.cpp - remove reentrant call detection debug code
Add cryptographic verification of export signatures as they arrive: - stashTxnData() caches serialized txn for verification - verifyAndAddSignature() verifies against cached data, rejects invalid - isSignatureVerified() / verifySignature() for Transactor fallback - Cleanup methods updated to clear verification cache Also removes leftover debug std::cerr from OpenView, STObject, and tests.
- Remove makeExportSignTxns() function (signatures now via TMValidation) - Simplify ExportSign::doApply() to no-op (ttEXPORT_SIGN kept for protocol) - Remove sfSigners from ltEXPORTED_TXN format (collected in memory now) - Remove unused OpenView include and forward declaration - Remove vestigial comment in TxQ about makeExportSignTxns
- Remove accept_wasm and emit_wasm hooks (not export-related) - Remove testBasicSetup, testEmitPayment, testXportPayment - Keep only testXportPaymentWithValidator which tests the export flow
- Delete ExportSign.cpp/h transactor (ttEXPORT_SIGN no longer used) - Remove isUVTx() function and all UVTx checks from Transactor/TxQ - Remove ttEXPORT_SIGN from TxFormats enum and format definition - Remove jss::ExportSign - Move signPendingExports() to ExportSignatureCollector Export signatures are now collected ephemerally via TMValidation messages, not via ttEXPORT_SIGN transactions.
Introduce data structures for consensus-derived randomness using commit-reveal scheme: - Add ExtendedPosition struct with consensus targets (txSetHash, commitSetHash, entropySetHash) and pipelined leaves (myCommitment, myReveal) - operator== excludes leaves to allow convergence with unique leaves - add() includes ALL fields to prevent signature stripping attacks - Add EstablishState enum for sub-phases: ConvergingTx, ConvergingCommit, ConvergingReveal - Update Consensus template to use Adaptor::Position_t - Add Position_t typedef to RCLConsensus::Adaptor and test CSF Peer This is the foundational data structure work for the RNG implementation. The gating logic and entropy computation will follow.
- Serialize full ExtendedPosition in share() and propose() - Deserialize ExtendedPosition in PeerImp using fromSerialIter() - Add harvestRngData() to collect commits/reveals from peer proposals - Conditionally call harvest via if constexpr for test compatibility
- Add clearRngState() call in startRoundInternal - Reset estState_ in closeLedger when entering establish phase - Implement three-phase RNG checkpoint gating: - ConvergingTx: wait for quorum commits, build commitSet - ConvergingCommit: reveal entropy, transition immediately - ConvergingReveal: wait for reveals or timeout, build entropySet - Use if constexpr for test framework compatibility
…layer Add protocol definitions for consensus-derived entropy pseudo-transaction: - ttCONSENSUS_ENTROPY = 105 transaction type - ltCONSENSUS_ENTROPY = 0x0058 ledger entry type - keylet::consensusEntropy() singleton keylet (namespace 'X') - applyConsensusEntropy() handler in Change.cpp - Added to isPseudoTx() in STTx.cpp The entropy value is stored in sfDigest field of the singleton ledger object. This provides the protocol foundation for same-ledger entropy injection.
This does not introduce a new levelization cycle; the existing xrpld.app <-> xrpld.overlay loop now has equal aggregate include counts after the consensus-extension work. Treat this as essentially the same architectural situation, not a meaningful worsening by itself. TODO: if we want to fix the boundary properly, extract a small shared consensus-extension wire/interface layer below both app and overlay instead of shaving includes to change the generated ratio.
Count the local proposer when deciding whether the previous round had enough participants for RNG, since prevProposers only tracks peers. This avoids a 4/5 honest quorum being treated as below quorum after one validator diverges. Allow an already quorum-aligned entropySetHash to proceed despite below-quorum conflicting hashes, while retaining zero-entropy fallback when no entropy hash reaches quorum alignment. Add CSF coverage for a persistent single bogus entropy hash and for conflicting bogus hashes without quorum.
Document the consensus-extension invariants for RNG, sidecars, export sig convergence, validator quorum, zero-entropy fallback, and proposal signing. Link the note from the RCL consensus README so future changes have a durable checklist.
Remove the stale TMValidation exportSignatures field from the draft proto path now that export signatures ride signed proposal sidecars. Document that any future validation-carried ConsensusExtensions data must be covered by the signed validation payload and duplicate/replay identity, not an unsigned wrapper field.
Stamp export signatures learned from proposals, sidecar sets, and candidate tx-set upgrades with a ledger sequence so cleanupStale can age them out. Remove invalid unverified signatures after tx-local verification fails, with a buffer match check to avoid deleting newer replacements.
Limit outbound TMProposeSet export signature attachments to ExportLimits::maxPendingExports so honest proposals stay within the same bound enforced by inbound proposal validation. Extra exports remain unsigned for that proposal and rely on the existing retry/expiry path.
Cap pending ttEXPORT work in open/apply ledgers, including hook-emitted exports when TxQ drains the emitted directory into the open ledger. Enforce the same bound for per-account shadow tickets so durable pending imports cannot grow unbounded.
Enforce the pending export cap for hook-emitted ttEXPORT work before commit. Replace the non-present sfEmittedTxn template field when building ltEMITTED_TXN entries so in-flight ledger checks see the emitted wrapper. Overflowing xport emission now returns tecDIR_FULL and leaves the emitted backlog capped at ExportLimits::maxPendingExports.
Keep hook result/state finalization non-fatal while enforcing the hook-export backlog cap through the transaction-level ApplyContext guard. This avoids resetting non-success tec metadata and preserves hook_again weak execution behavior.
Export-only originally used unanimity as a conservative substitute for the CE/RNG sidecar machinery. That made sense before Export had its own signed ExtendedPosition field and exportSigSetHash convergence gate. Now Export sidecars are signed and converged independently of RNG, so a quorum-aligned exportSigSetHash plus verified active-view signature quorum is deterministic enough for Export-only mode. Keeping unanimity would let one active validator veto an otherwise converged export round. Update CSF and testnet coverage to treat Export-only the same way: one missing/conflicting signer in a 5-validator network succeeds at 4/5, while below-quorum still retries or expires.
Clarify that export sidecar publication is local verified material only, and fetched sidecar leaves must be active-view checked, candidate-tx verified, and promoted into ExportSigCollector before closed-ledger apply can use them.
When a candidate set contains ttEXPORT but a node has no local verified export sig material yet, give tx-converged peers one bounded opportunity to advertise an exportSigSetHash before closed-ledger apply. This is a safety coordination window, not a wait-for-Export-success mechanism. If no advertised sidecar arrives or fetched material cannot be merged by the deadline, Export convergence is marked failed and the transaction retries or expires through normal rules. Add CSF coverage for a peer that can only succeed by fetching peer-advertised export sidecars, plus a direct ConsensusExtensionsTick test for the pre-advertisement observation window. Document the consensus-extension priority order: safe, fast, works.
Require tx-converged peers to advertise sidecar hashes before accepting RNG entropy or export signature success from local quorum alignment. The RNG reveal fast path now publishes the entropy set and waits for peer observation instead of accepting in the same tick. On timeout, RNG clears the advertised entropy hash and falls back to deterministic zero. Add unit and CSF regression coverage for asymmetric peer observation.
# Conflicts: # src/test/app/SetHookTSH_test.cpp # src/xrpld/app/tx/detail/InvariantCheck.cpp # src/xrpld/app/tx/detail/InvariantCheck.h
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## dev #693 +/- ##
==========================================
- Coverage 66.52% 65.56% -0.96%
==========================================
Files 831 847 +16
Lines 78166 81702 +3536
Branches 44374 46591 +2217
==========================================
+ Hits 52000 53569 +1569
- Misses 17808 19441 +1633
- Partials 8358 8692 +334 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
# Conflicts: # hook/sfcodes.h # include/xrpl/protocol/Feature.h # include/xrpl/protocol/detail/sfields.macro
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Last updated: 2026-03-24 | branch: feature-export-rng | commit: 9562b45 (chore: remove stale Peer-level RNG forwarders)
featureExport + featureConsensusEntropy: Cross-Chain Exports + Decentralized Secure Randomness
This PR introduces two features under separate amendment flags:
featureExport— Cross-chain transaction export: hooks or users create transactions signed by the network's validators for use on external chainsfeatureConsensusEntropy— Consensus-derived randomness: deterministic, manipulation-resistant entropy available to hooks viadice()andrandom()Both share UNL infrastructure (quorum calculation, UNLReport trust model) but are independently amendment-gated.
Amendment Relationship
The two amendments are independent but complementary:
Why Export benefits from CE: Without CE, export uses ephemeral signature collection. Different validators may see different sig counts at ledger close. With 80% quorum this could cause validators to flip between success and retry across ledger boundaries — acceptable but not ideal. Unanimity avoids this churn entirely, at the cost of any missing validator blocking the export.
With CE enabled, Export piggybacks on CE's
ExtendedPositionserialization, sub-state machine, and SHAMap fetch/merge infrastructure to converge on a shared sig set before closing the ledger. All validators agree on exactly which exports have quorum, enabling the standard 80% threshold.Neither blocks the other: CE's RNG sub-states and Export's sig convergence run in parallel during the establish phase. Both check their own gates independently and fall back gracefully if they can't converge.
UNL Source and Fallback Behavior
Both features use UNLReport ActiveValidators (
UNLReport.sfActiveValidators) as the canonical validator set when available.For liveness on early ledgers and test/dev environments, both use a fallback model:
UNLReportis unavailable or empty, use the node's local validator configuration as a temporary proxy.Part 1: Cross-Chain Transaction Export
What This Feature Does
This feature enables cross-chain transaction export — allowing a hook or user on Xahau to create a transaction that Xahau's validators collectively sign. The resulting multisigned transaction is a normal, valid transaction on XRPL — no protocol changes or cooperation from XRPL are required.
How it works from XRPL's perspective
An account on XRPL is set up with a SignerList pointing to Xahau validator keys. When Xahau's validators sign an exported transaction, they're effectively acting as members of that SignerList. Once enough signatures are collected, the transaction can be submitted to XRPL as a standard multi-signed transaction. XRPL doesn't know or care that the signatures came from Xahau.
This design requires zero XRPL-side changes, no XLS specification, and no amendment on XRPL.
How it works from Xahau's perspective
A hook calls
xport()or a user submits attEXPORTtransaction (type 91). The transaction enters the open ledger (tesSUCCESSprovisional), and validators attach their multisign signatures to consensus proposals. When enough validators have signed (quorum met), the export succeeds withsfExportResultin the transaction metadata — containing the fully multisigned transaction assfExportedTxn(readable JSON, ready for raw submission to XRPL). If quorum isn't reached beforeLastLedgerSequence, the export expires withtecEXPORT_EXPIRED.In standalone mode (unit tests / dev), Export::doApply signs directly with the node's own validator keys — no consensus proposals needed.
Exported transactions must use
TicketSequence(withSequence=0) because a bounced transaction on the destination chain would jam sequential sequence numbers. ANetworkIDguard rejects exports targeting the local network or from unconfigured nodes.Design Evolution
The export signing mechanism went through three iterations:
V1 — On-ledger signature accumulation (built, abandoned): Each validator submitted a signing transaction that modified a shared ledger entry. Every modification produced Previous/Final metadata containing the entire growing signer array — O(n²) metadata explosion with 35 validators.
V2 — Validation-based ephemeral sigs (built, replaced): Moved signatures out of the ledger into memory. An
ltEXPORTED_TXNentry stored the pending export, validators signed at validation time and broadcast viaTMValidation, and the TxQ injected attEXPORT_FINALIZEpseudo-tx on quorum. This solved the metadata bloat but required multiple ledger closes and a complex lifecycle (separate ledger entry → signature collection → pseudo-tx injection → cleanup).V3 — Retriable proposal-based sigs (current): A single
ttEXPORTtransaction enters the open ledger, validators attach signatures to consensus proposals, and the transactor either succeeds (quorum) or retries (terRETRY_EXPORT). No intermediate ledger objects. Result in metadata. Same-ledger finalization.How It Works
Instead, exports use a retriable transaction pattern with proposal-based signature collection:
ttEXPORT→ enters the open ledger withtesSUCCESS(provisional, consumes sequence + fee)sfExportedTxn), computes its multisign signature (buildMultiSigningData+sign), and attachestxHash(32) + pubkey(33) + signature(~72)toTMProposeSet.exportSignatures(deduplicated per round viamarkSent())ExportSigCollector(validated against UNL)sfExportedTxninsidesfExportResultmetadata, creates shadow ticket keyed by the signed tx hash (getHash(HashPrefix::transactionID)— includes Signers)terRETRY_EXPORT(retained for next ledger, no fee/sequence consumed)tecEXPORT_EXPIRED(sequence consumed, clean failure — preventstefMAX_LEDGERsilently dropping the tx on the next ledger)Only the final result touches the ledger. No intermediate signature accumulation. The multisigned blob in metadata is ready for raw submission to XRPL. Typical latency: same ledger (99%+ on healthy network).
Tiered Quorum: Safety Without Complexity
Export quorum adapts based on whether ConsensusEntropy is also enabled:
ExportSigCollectorcalculateQuorumThreshold)ExtendedPosition, sub-state machine, and SHAMap fetch/merge for deterministic agreementSHAMap Convergence (with CE)
When CE is enabled, export sigs converge using the same infrastructure as RNG commit/reveal:
exportSigSetHashadded toExtendedPosition(flag 0x10)sha512Half(txHash, validatorPK)Signature Collection
Proposal Attachment
Each validator scans the open ledger for
ttEXPORTtxns, extracts the inner tx (sfExportedTxn), computes its multisign signature viabuildMultiSigningData(), and attachestxHash + pubkey + signatureto the proposal. Gated onfeatureExport, deduplicated per round viamarkSent().Proposal attachment code (ConsensusExtensions.cpp)
📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:1555-1666Sig Harvesting
Trusted proposals are harvested in
PeerImp.cpp. Each entry's pubkey is validated against the trusted validator set before acceptance. The wire format is variable-length:txHash(32) + pubkey(33) + signature(~72). Signatures are stored inExportSigCollectoralongside pubkeys.ExportSigCollector
Thread-safe collector (
ExportSigCollector.h) with stale cleanup (256-ledger timeout). Stores pubkeys for quorum counting and signature buffers for assembling the multisigned tx. Key methods:addSignature(txHash, pubkey)— pubkey-only (quorum counting)addSignature(txHash, pubkey, signature)— with multisign signaturesnapshotWithSigs()— returns all collected signatures for blob assemblymarkSent(txHash)— deduplicates per consensus roundcleanupStale(ledgerSeq)— removes entries older than 256 ledgersExport Transactor
The transactor has three outcomes on the closed ledger:
sfExportResultmetadataterRETRY_EXPORT(before LLS) ortecEXPORT_EXPIRED(on the LLS ledger — last chance, sequence consumed cleanly)Key detail: the shadow ticket stores
getHash(HashPrefix::transactionID)of the multisigned blob (which includes the Signers array), not the unsigned inner tx hash. This ensures the hash matches what ends up in the XPOP when the tx executes on XRPL.Export::doApply (full source)
📍
src/xrpld/app/tx/detail/Export.cpp:85-310Shadow Tickets and the Export Round-Trip
Export is a 3-way handshake, not fire-and-forget:
sfExportedTxninsidesfExportResultmetadata — readable JSON, ready for raw submissionltSHADOW_TICKET) is created, keyed by account + ticket sequence, storing the signed tx hash (getHash(HashPrefix::transactionID)— includes all fields including Signers)SignerListpoints to the Xahau validator keys, so XRPL validates it as a standard multisigned transaction. It executes (or bounces), producing an XPOP.ttIMPORT. Import checks the shadow ticket exists, verifies the XPOP inner tx hash matches the shadow ticket's stored hash, consumes the shadow ticket (frees reserve), and fires hooks.Shadow tickets are round-trip completion tokens:
xport_cancel()hook API orsfCancelTicketSequenceonttEXPORTttEXPORTtransactionWhen
ttIMPORTseessfTicketSequenceon the inner transaction, it takes the export callback path: verify shadow ticket hash match, consume the shadow ticket, fire hooks, done. The export callback path skips thesfOperationLimitand signing key match checks (the shadow ticket already proves the relationship). No B2M balance crediting. When there's nosfTicketSequence, it takes the existing Burn-to-Mint path unchanged.Hook Integration
Hooks call
xport()which internally constructs attEXPORTwrapper (withsfEmitDetails) and pushes it onto the emitted txn queue. The wrapper flows through the normal emitted txn path:xport_reserve(N)— reserves N export slots (also reserves emit slots)xport(inner_tx_blob)— validates inner tx → constructsttEXPORTwrapper → emits itttEXPORTenters the open ledger next round → proposal-based sig collection → retriable transactorThe hook receives the inner tx hash (the cross-chain transaction it built), while the
ttEXPORTwrapper handles the Xahau-side lifecycle.Export Protocol Additions
sfExportResult(OBJECT 98)sfExportedTxn+sfLedgerSequence+sfTransactionHash)sfCancelTicketSequence(UINT32 101)sfExportedTxn(OBJECT 90)ltSHADOW_TICKET(0x5374)ttEXPORT(91)terRETRY_EXPORTtecEXPORT_EXPIRED(200)TMProposeSet.exportSignatures(field 13)xport(),xport_reserve(),xport_cancel()Part 2: Decentralized Secure Randomness
Adding randomness to deterministic consensus sounds simple until you try to do it without breaking safety. This part implements Same-Ledger Usable Randomness: finalizing entropy after user intent is locked, but before normal execution in that same ledger.
🔎 Review Scope
featureConsensusEntropyisDefaultNo; behavior is inert until enabled by amendment vote.ConsensusEntropy_test(Hookdice()/random(), fallback semantics) andExtendedPosition_test(serialization compatibility and malformed wire cases).🛠 How It Works (The Final Solution)
The architecture centers on converging on signed input sets rather than voting on a derived output hash. This ensures that every node can independently verify and reconstruct the final result.
1. Transport: Piggybacked Proposals
The
ConsensusProposalwire format is extended viaExtendedPosition. Most entropy data (commitments and reveals) flows through existing proposal gossip with low incremental payload overhead on the fast path, while consensus latency cost comes from the added sub-state progression/timeouts.ExtendedPosition::operator==only compares thetxSetHash. RNG sub-state differences never stall the core consensus on user transactions.2. Pipelined Sub-states
RNG progression runs inside internal
establishsub-states. These are checkpoints within the existing consensus cadence:ConvergingTx: Normal transaction convergence while harvesting entropy commitments.ConvergingCommit: Locking thecommitSetonce an 80% quorum overUNLReport ActiveValidators(or fallback set ifUNLReportis unavailable) is reached.ConvergingReveal: Targets reveals from 100% of known committers, bounded by timeout/fallback paths (including the 1.5s reveal timeout) to preserve liveness.3. SHAMap Union Convergence
Harvested commitments and reveals are stored in ephemeral, unbacked SHAMaps.
InboundTransactionspipeline to fetch only the missing leaves from peers.4. Synthetic Injection & Same-Ledger Execution
Once reveals are collected, the final entropy is computed deterministically (
sha512Half(sorted_reveals)).buildLCL(Ledger Construction), the node locally synthesizes attCONSENSUS_ENTROPYpseudo-transaction.⚓ Hook API Integration
Provides two new deterministic WebAssembly APIs for Hook developers:
dice(sides): Returns a fair integer from0tosides-1.random(write_ptr, write_len): Fills a buffer with cryptographically secure consensus-derived randomness.🛡 Safety & Liveness
🛠 Infrastructure & Support Logic
Several non-obvious plumbing changes were required to make the RNG pipeline robust and testable:
1. Fast Polling during RNG Transitions
To reduce the latency impact of the extra sub-states, the heartbeat timer accelerates to 250ms (tunable via
XAHAU_RNG_POLL_MS) while in the RNG pipeline.📍
src/xrpld/app/misc/NetworkOPs.cpp:1031-10442. Local Testnet Resource Charging
Connections from
127.0.0.1normally share a single IP resource bucket. This change preserves the port for loopback addresses so that local multi-node testnets don't hit peer resource limits due to the increased RNG set traffic.📍
include/xrpl/resource/detail/Logic.h:113-1173. Test Environment Gating
featureConsensusEntropyis excluded from defaultjtx::Envtests to prevent its automatic pseudo-tx injection from breaking existing test suites that rely on specific transaction counts.📍
src/test/jtx/Env.h:86-894. Pseudo-transaction Filtering
Internal metadata (commits/reveals) is stored as pseudo-transactions in ephemeral SHAMaps for transport. This logic ensures they are filtered out and never submitted to the actual transaction processing engine.
📍
src/xrpld/app/ledger/ConsensusTransSetSF.cpp:70-74Guided Code Review (Projected Source)
This section follows runtime order so the code reads as a story, not a file dump.
1) Proposal payload:
ExtendedPositioncarries RNG + Export sidecar fieldsExtendedPositionadds commit/reveal set identities, export sig set hash, and per-validator leaves while keeping tx-set identity explicit.Non-obvious:
operator==compares onlytxSetHashon purpose. That decouples core tx-set convergence from RNG/Export sub-state drift.operator==(equality firewall):📍
src/xrpld/app/consensus/RCLCxPeerPos.h:110-145add()(signed serialization of all sidecar fields):📍
src/xrpld/app/consensus/RCLCxPeerPos.h:159-193fromSerialIter()(legacy + extended wire decode):📍
src/xrpld/app/consensus/RCLCxPeerPos.h:216-2612) Harvest stage: trust boundary + reveal verification
Incoming RNG data is rejected for senders outside
UNLReport ActiveValidators(or fallback set whenUNLReportis unavailable), and reveals are accepted only if they match prior commitments.📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:1155-12643) Quorum basis: expected proposers first, UNLReport/fallback set
📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:57-674) State-machine checkpoints:
ConvergingTx -> ConvergingCommit -> ConvergingReveal📍
src/xrpld/consensus/ConsensusExtensionsTick.h:22-6705) Export sig convergence gate (parallel with RNG)
📍
src/xrpld/consensus/ConsensusExtensionsTick.h:674-7456) SHAMap construction: commit/reveal sets with proof blobs
📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:315-371📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:379-4347) Injection stage (A): final entropy selection with deterministic fallback
📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:1021-10778) Injection stage (B): build and enqueue
ttCONSENSUS_ENTROPY📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:1081-11319) Build stage: entropy pseudo-tx executes before normal transactions
📍
src/xrpld/app/ledger/detail/BuildLedger.cpp:111-14810) Apply stage: write consensus entropy into the singleton ledger object
📍
src/xrpld/app/tx/detail/Change.cpp:248-26411) Wire anchor: proposal message carrying extended payload bytes
📍
include/xrpl/proto/ripple.proto:153-175[Architectural Retrospective]
The Road to Consensus-Native Randomness: A Retrospective
A narrative history of how the RNG architecture evolved from early
featureRNGexperiments into the finalfeatureConsensusEntropydesign.Adding randomness to deterministic consensus sounds simple until you try to do it without breaking safety.
Consensus requires determinism: every honest node must compute the same state transition.
Randomness requires unpredictability: nobody should know the final value early enough to game it.
The requirement that made this hard was not just "randomness," but same-ledger usable randomness: finalize entropy after user intent is locked, but before normal execution in that same ledger.
That path was not linear.
Part I: What the First Branch Taught Us (
featRNG)The initial branch was aggressively practical: reuse existing transaction paths, avoid deep consensus surgery, and move fast.
Experiment 1:
ttRNGlooked straightforward, then failed quicklyThe earliest model used a single transaction path (
ttRNG) with validator-generated entropy.It failed for a concrete reason: entropy bytes entered open-ledger transaction flow too early.
That made the randomness path mempool-observable and timing-sensitive, so sophisticated actors could condition behavior around visible entropy before the round was fully sealed.
Very quickly, the branch moved toward a dual-model design (
ttENTROPY+ttSHUFFLE) to try to close that timing gap.Experiment 2: dual-model defense (
ttENTROPY+ttSHUFFLE)The next design split responsibilities:
ttENTROPY: a UNL Validator Transaction (UVTx) — zero fee, seq=0, signed by the validator's ephemeral key, validated by UNLReport membership — used to submit blinded entropy hashes and later reveal them.ttSHUFFLE: a pseudo-transaction that derived extra entropy from proposal signatures, timed to land after the transaction set was frozen.Conceptually, this was smart defense-in-depth. Operationally, it hit three structural problems:
Experiment 3: mitigation hacks and why they still were not enough
Deterministic self-shuffle and piggyback variants improved specific failure modes. They did not remove the deeper issue: the model remained timing-sensitive and complex under real asynchronous behavior.
This was the "env var city" period (
XAHAU_SELF_SHUFFLE,XAHAU_PIGGYBACK_SHUFFLE, and brieflyXAHAU_AUTO_ACCEPT_SHUFFLES): useful for exposing failure boundaries, but also a clear signal that the architecture was being patched against the grain of consensus.Experiment 4: dedicated shuffle phase (
Open -> Establish -> Shuffle -> Validate)The branch then tried full structural separation: a top-level shuffle phase, custom RNG message flow (
TMRNGProposal), anRNGServicemanaging commits/reveals in simplestd::maps, and aforceRevealPhase()sync point to keep nodes aligned.This delivered one lasting insight: contributors should be tied to actual recent consensus participants (the seed of later expected-proposer logic).
But the phase itself was abandoned:
The conclusion from Part I was precise:
commit/reveal was the right cryptographic primitive, but the transport/convergence model was wrong.
The final commit on the initial branch landed one more practical insight: entropy participation should track actual establish-round participation (
establishProposers), not just static UNL membership. That expected-participant logic survived into the final architecture even as the dedicated shuffle phase did not.Part II: The Trap We Nearly Chose (Scalar Opinion Convergence)
The seductive simplification was to treat entropy like any disputed scalar:
"let nodes publish their computed entropy value and avalanche-converge on the majority."
A lightweight discrete-event simulator (
sim/rng_sim.cpp) was built to pressure-test this assumption under realistic latency and packet asymmetry. (This was a quick prototype model, not a faithful rippled consensus simulator — but it was sufficient to expose the core pathology.)This fails for a reason that became impossible to ignore:
When node A computes from set
S_Aand node B fromS_B, andS_A != S_B, their scalars are unrelated.At that point, you face a bad fork in design philosophy:
The final architecture deliberately chose option 2.
Part III: The Clean-Slate Branch (
featureConsensusEntropy)The new branch started as consensus documentation. That documentation work clarified failure boundaries so sharply that it became a from-scratch implementation effort.
This was not a rename exercise. It was selective reconstruction:
In other words: the primitive survived, the convergence model changed.
Hooks-facing RNG APIs such as
dice()andrandom()were among the pieces carried forward and finalized in this architecture.Breakthrough 1: converge on inputs, not output opinions
The core shift was simple and profound:
do not vote on final entropy values; converge on signed input sets.
Breakthrough 2: proposal-carried leaves + set identities
ExtendedPositioncarries:myCommitmentmyRevealcommitSetHashentropySetHashFast path: normal proposal traffic carries most of what nodes need.
Safety net: SHAMap-backed set identity enables deterministic reconciliation when packets drop or nodes lag.
Breakthrough 3: equality firewall
ExtendedPosition::operator==comparestxSetHashonly.That keeps core Tx-set convergence from being held hostage by RNG sub-state timing differences while still allowing entropy sub-state convergence to proceed and reconcile.
Breakthrough 4: sub-states, not a top-level RNG phase
Instead of adding another global phase boundary, the design runs RNG progression inside establish sub-states:
ConvergingTxConvergingCommitConvergingRevealThis preserved the existing consensus cadence while integrating entropy convergence where it belongs.
Breakthrough 5: SHAMap union convergence
Union merge is monotonic: sets grow as verified leaves arrive.
Scalar opinions can oscillate; verified set growth does not.
And SHAMap mechanics keep reconciliation practical:
So overhead is low on the golden path, with bounded recovery cost when reconciliation is needed.
Part IV: Hardening Moves That Made It Viable
The architecture became robust only after concrete hardening steps, each forced by a specific failure mode observed during testnet runs:
sfBlobin SHAMap entries): without these, any peer could inject spoofed commit/reveal entries during a Cold Path fetch. Embedding the proposal signature makes every contribution independently verifiable.sha512Half(reveal, pubKey, seq) == commitment): without this, a validator could commit to one value and reveal another (grinding attack).Important nuance:
Concrete progression: before the reveal-convergence fixes, a 15-node testnet produced 7 distinct entropy values in the same round (nodes collected different 80% subsets of reveals). After these hardening steps, 20-node testnets reported identical commit-set hashes and ~2.2s convergence with bounded recovery under node loss. (That ~2.2s came from aggressively tuned low-ms settings, including
XAHAU_RNG_POLL_MSand tight timeout windows; broader production topologies may need larger windows.)Part V: The Masterstroke (
ttCONSENSUS_ENTROPY)Once nodes converge on the relevant verified input set, final entropy is computed deterministically (
sha512Half(sorted_reveals)) and injected as a synthetic pseudo-transaction:ttCONSENSUS_ENTROPY.This injection happens locally in
doAccept(), right before ledger construction. The pseudo-transaction is sorted to execute first inBuildLedger.cpp, so all user transactions and Hooks executing in that same ledger block can consume the entropy via thedice()andrandom()WebAssembly APIs.Why no final gossip round on the derived scalar?
Because gossip resolves disagreements.
At this point, the system has already converged on verified inputs; the output function is deterministic. Forcing an extra opinion round adds delay and bandwidth cost without cryptographic benefit.
If a node suffers a local fault and synthesizes the wrong pseudo-tx, its resulting ledger hash will mismatch the network supermajority. Its validations will fail, and it will safely fork off and fetch the correct ledger from peers. Ledger safety is preserved by the validation phase, not the deliberation phase.
Safety and Liveness Framing
A useful framing that survived all iterations:
This matches the formal XRPL LCP framing in Chase & MacBrough (2018): Example 5 captures the key intuition that deliberation outcomes can vary, while fork safety itself is anchored by validation-phase overlap conditions formalized in Theorem 8.
This safety claim is specifically about ledger agreement, not about maximum entropy strength under adversarial withholding.
That distinction prevented a lot of category errors in design discussions.
Closing
The final
featureConsensusEntropyarchitecture is "least-bad" in the engineering sense:From
ttRNGto dual-tx entropy, to dedicated shuffle phase, to scalar-convergence rejection, the trajectory kept pointing to the same destination:commit/reveal inputs, SHAMap set identity, union reconciliation, deterministic synthetic injection, and bounded fallback behavior.