Skip to content

Resource-cost benchmark harness with a CI regression gate on worst-case payouts #353

Description

@grantfox-oss

Difficulty: Hard. Scope: multi-week epic. Contract + CI.

Problem

Soroban meters CPU instructions, memory, ledger reads/writes, and rent, and a transaction that exceeds its resource budget simply fails. payout() loops over up to MAX_RECIPIENTS (32) recipients doing a token transfer each, pay_many loops over splits doing that per split, and distribute plus nested Recipient::Split routing compounds it. Nobody today knows the real resource cost of the worst cases, and there is nothing stopping a future change from silently pushing the 32-recipient payout, or a deep nested tree, past the limit and bricking it in production. #53 asks to benchmark one worst case once; this asks for a standing, enforced cost budget.

Proposal

  1. A benchmark harness that measures CPU instructions, memory, and read/write/rent footprint for the expensive paths: 32-recipient pay, pay_many across many splits, distribute of a large balance, and nested trees at increasing depth.
  2. A CI regression gate that records baseline costs and fails the build when a change increases any metric beyond a set tolerance, so cost regressions are caught in review rather than in production.
  3. An optimization pass using the harness: reduce storage reads/writes and per-recipient overhead in the hot paths where the data shows it matters (for example, storage key construction, Vec access patterns, event emission cost).

Why this is hard

  • Getting reliable, low-noise cost measurements out of the Soroban test/budget tooling and making them deterministic enough to gate CI without flaky failures.
  • Establishing meaningful baselines and tolerances, and a workflow to update them intentionally when a change legitimately costs more.
  • The optimization work requires reading the generated cost profile and restructuring storage/iteration without changing behavior (must keep the full test suite and, ideally, the differential fuzzer green).
  • Modeling how close the worst realistic case (32 recipients, deep nesting, pay_many) sits to the actual ledger limits, and documenting the safe operating envelope.

Deliverables

  • Benchmark harness producing per-path cost metrics.
  • CI job that gates on cost regression against committed baselines, with a documented update process.
  • An optimization pass with before/after numbers.
  • A doc: measured costs, the safe envelope (max recipients/depth per call), and guidance for callers.

Acceptance criteria

  1. Maintainer-approved metric set and tolerance policy before implementation.
  2. Worst-case costs (32-recipient payout, deep nested tree, large pay_many) are measured and published.
  3. CI fails on a cost regression beyond tolerance and passes on a legitimate baseline update; demonstrated both ways.
  4. The optimization pass shows a measured reduction on at least one hot path with no behavior change (tests still green).
  5. The safe operating envelope is documented.

Related

Extends #53 (single benchmark) into a standing gate. Provides the cost numbers the merkle, streaming, and mandate epics all need to bound their per-call work.

Metadata

Metadata

Assignees

Labels

GrantFox OSSIssue tracked in GrantFox OSSMaybe RewardedIssue may be eligible for a GrantFox rewardOfficial Campaign | FWC26Campaign: Official Campaign | FWC26enhancementNew feature or requesthelp wantedExtra attention is neededrustPull requests that update rust code

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions