Measure every candidate scoring metric on stored runs, and record what they disagree about (#56) - #62
Open
adachi-440 wants to merge 32 commits into
Open
Measure every candidate scoring metric on stored runs, and record what they disagree about (#56)#62adachi-440 wants to merge 32 commits into
adachi-440 wants to merge 32 commits into
Conversation
Proposes ADR 0019: score the live competition as a risk-adjusted log-growth series over 4h epochs on a single continuous economy (`mean_e(x_e) - lambda*std_e(x_e)`), retiring the field-relative z-score of ADR 0017 §4. Records the benchmark choice (noop), the bankruptcy gates (G1/G2), the gates deliberately not taken (G3/G4/G5/G6) and the funding fix that "USDC-only" needs to actually be USDC-only. Two points depart from what issue #56 recorded, both from review: - G7 (TWAP of the manipulable marks) applies to *every* epoch boundary, not only the last one. Inflating an intermediate mark leaves the mean untouched but moves the variance by (2d/E)*[(x_k - x_k+1) + d], which is *negative* whenever d < x_k+1 - x_k: smoothing a bad epoch into the next one lowers std and raises the score. The manipulation also unwinds on the next block, so it is cheaper than the final-boundary one and repeatable 42 times. - The funding block needs a code change, not just YAML. `fundWallet` adds `GAS_BUFFER_WEI = 5 ETH` on top of `funding.ethWei` (sdk/src/chain.ts:528,543), so the proposed `ethWei: 1 ETH` actually lands 6 ETH in the wallet: 15.2% of the portfolio in ETH beta, not the 2.9% the section is arguing for. Folding the buffer into `funding.ethWei` is now part of implementation step 1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ADR 0019 §8 defined epoch boundaries in real time (4h), which is right for the live chain but leaves the poc with nothing to run: an anvil run has no simulated clock at all. The price path, the stress windows and the run length are all counted in blocks, and `lst.simulatedSecondsPerBlock` is a venue-local knob for the vault's yield accrual, not a run-wide clock. So the harness gets a block count instead, and no global clock is introduced. 12 blocks/epoch (E=42 -> 504 blocks/week, ~17 min at blockTimeSec=2): - leaves 3-5 blocks for G7's per-boundary median window. At 4 blocks/epoch (the 1 block = 1h reading) the window would equal the epoch, so its length could not be calibrated and no trading would fit inside an epoch. - stays inside anvil's ~1,050 block history retention, so the existing post-run reconstruction still works. 24 blocks/epoch (1,008 blocks) pins the oldest cross-section against that limit and would require moving to online cross-sections first -- now recorded as an open item, since the live chain (#33) needs that move regardless. Also corrects the gas risk: weekly gas consumption cannot be measured on a compressed harness, because the action count follows the block count rather than 168 hours. Budget it as per-action gas x allowed cadence x 168h. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#56) `fundWallet` set every wallet's balance to `ethWei + wethWei + GAS_BUFFER_WEI`, adding 5 ETH on top of whatever the roster asked for. The official regimes set `funding.wethWei: "0"` and comment it "USDC-only distribution: nobody starts already exposed", but leave `funding.ethWei` unset, so agents actually started on 105 ETH -- 92.6% of initial value in ETH beta (measured on depeg#701: 336,867 = 105 ETH + 25,000 USDC). `netPnlUsdc` hid this by valuing the initial and final cross-sections at the same final price, so the beta cancelled. ADR 0019's epoch series is live-marked and does not cancel: every agent's std term would be dominated by ETH volatility, and lambda would measure the market instead of the agent. The endowment the ADR proposes (1 ETH + 100k USDC) would also have landed as 6 ETH = 15.2% of the portfolio rather than the 2.9% it argues for. The buffer becomes a parameter. Agent wallets pass 0, which makes `funding.ethWei` the balance the agent actually holds and makes the ERIS_ECONOMIC_GAS lower-bound check (0.5 ETH, coordinator.ts) validate that same number. Environment wallets keep the default: flow bots and the whale run on `flowEthWei` (1000 ETH) where it is immaterial, and the admin USDC top-up in protocols/aave.ts passes `ethWei=0` and pays for the grant out of the buffer, so folding it there would have zeroed the admin balance (anvil_setBalance overwrites rather than adds). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ADR 0019 step 3. The scorer reconstructed a per-block value curve but only
ever used its endpoints (alphaLast - alphaFirst), so nothing produced the
input the metric actually needs: the value at each epoch boundary, from
which x_e = ln(W_e / W_{e-1}) is formed.
`reconstructValueSeries` now samples those boundaries and returns them as
`epochSeries` (raw values per agent, aligned with the boundary blocks),
which rides into summary.json and events.jsonl with the rest of the
reconstruction meta. Raw values rather than returns or a score: lambda and
the epoch length are provisional, so the series has to stay recomputable
from stored data the way standings.json is derived from matrix.json.
- `run.epochBlocks` (ERIS_EPOCH_BLOCKS), default 12 = ADR 0019 §8's
calibration epoch. 0 disables the series.
- Boundaries are read even when `scoreEvery` thins the curve: the thinned
cross-sections are equity-curve resolution, the boundaries are the score.
The reconstruction reads the union of the two.
- A trailing partial epoch is dropped. A short window yields a smaller log
return by construction, which the metric would read as the agent slowing
down.
- A missing boundary value stays `null` and warns, rather than defaulting to
0 -- a gap in the series is not a bankrupt agent (G1/G2 land in step 4).
- The series is built from the ordinary live mark, not `alphaValueUsdc`
(ADR 0019 §3: its beta removal is partial, so it would price the same bet
differently depending on the instrument).
Unit tests cover the boundary math (43 marks for 42 epochs, equal widths,
partial-epoch drop, disabled/degenerate lengths). Not yet exercised on a
live run: that needs the deployer anvil.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
) ADR 0019 step 2 (G7). A scoring cross-section reads its stable prices out of a pool, so the holder can move them: buy into a thin pool in the boundary block, be marked at the pushed price, unwind on the next one. Epoch boundaries are now valued at the median of the blocks leading up to them (`run.markMedianBlocks`, default 5 of a 12-block epoch), which forces a push to be held for most of the window to count -- a spread-cost round trip becomes a position. Every boundary, not just the last one. Moving an intermediate mark leaves the mean untouched (the next epoch cancels it) and changes only the dispersion, and the change is *negative* whenever the push is in the smoothing direction: ln W_k += d moves the variance by (2d/E)*[(x_k - x_k+1) + d]. An agent whose epoch went badly can raise its boundary mark, borrow the return from the next epoch, and lower its own std. Scope: the market-priced stable seam. That is one continuity, not one venue -- spot registry stables (#27) and the Liquity venue's Trove debt and Stability Pool deposit both price off `ctx.stablePrices()`. The LST market price and LP share reserves are named by the ADR too, but each builds its price inside its own adapter's staged reads, so they stay live-marked for now. `summary.json` reports `markMedian.surfaces` and the largest gap seen between each stable's live probe and the median it was scored at, so a partial G7 never reads as a complete one. Only the boundaries are medianed: the cross-sections in between are the equity curve, and smoothing those would hide real intra-epoch moves without protecting any score. Blocks where the pool refused to quote are dropped from the median rather than counted as par, which would drag the mark toward $1 and erase the dislocation being measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 4: G1 (bankruptcy floor) -> G2 (scoring-side freeze) -> the score itself, as a pure function over the stored series (core/src/scoring/epochScore.ts). The coordinator applies it to the reconstruction's epoch series and writes `epochScores` into summary.json next to the series it was derived from, so lambda or the epoch length can be changed and the run rescored without re-running it. - G1 floors every mark at 1% of the starting value. Not merely ln(0) avoidance: Aave's valueUsdc is collateral - debt with no clamp, so a liquidated agent's total can be negative, where ln is NaN rather than -Infinity. Flooring also bounds the worst single-epoch return, which keeps one blowup from dominating the field's dispersion. - G2 freezes the series at 0 from the epoch the floor was touched, including when the position recovers afterwards. It is a scoring rule and never blocks a tx: on a live chain a participant reaches the sequencer directly, so a chain-side stop would need sequencer-level power and would not transfer off anvil. - A missing boundary carries the previous value forward at a return of 0 and is reported. Dropping it would shorten the series for exactly the agents the environment failed to read, and a shorter series has a smaller std. - The denominator is uniform across the field rather than per-agent, so dying early cannot pay. standings.ts (the practice harness's z-score) is untouched, per ADR 0019 §7: how to align the practice harness is explicitly not decided yet. Verified against a live depeg#701 backtest (the first run of the epoch series and G7 on a chain): 9 epochs of 12 blocks, no gaps, and the median moved the DAI mark by up to 54.9bps versus the boundary's live probe. Scored offline from that summary.json, the metric ranks peg-arb-eager > peg-arb > venue-arb > noop -- venue-arb outearns peg-arb on netPnlUsdc but pays for it in dispersion, which is the metric doing what it was chosen to do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two runs of depeg#701 (same seed, 120 blocks = 9 epochs), one on the current distribution and one on the funding ADR 0019 §6 proposes. Under 100 ETH + 25k USDC the four agents' std_e agree to within 1.1% -- they are all just ETH's volatility -- so lambda does not separate anyone, and noop scores -1.963e-3 rather than the 0 the metric is read against. Under 1 ETH + 100k USDC the spread is 199.9% and noop comes back to -6.18e-5, the residual beta of the gas reserve that §4 predicted. The same run also produced the first live example of lambda as a hurdle: peg-arb earned +104 USDC and still scored -6.7e-6, because its mean/std of 0.234 sits just under lambda. And the 5%-of-endowment concern about limits.agentUsdcUnits did not bite here -- neither run had a single rejected submission, so the cap was not binding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Implementing the remaining G7 surfaces turned out to be implementing nothing. Both fall out, for different reasons, and the ADR now records why rather than leaving a "partially applied" note that invites the work again. LP shares are valued by composition -- reserves x the environment's fair price, never the pool's own price -- and the agent's spot side is marked at those same external prices. Pushing the pool moves value between the agent's two buckets instead of creating any: what the trader loses the pool gains, and the agent recovers only its own share of it. A wash when it owns the whole pool, a loss otherwise. The LST venue's scored mark is face value: `valueUsdc` is the vault's redemption rate times the WETH fair and reads no pool at all. The pool quote feeds `liquidatableValueUsdc`, which is reported next to the mark but never summed into the value series (reconstruct.ts adds `value.valueUsdc`). What makes the stables different is that there a pool quote *is* the mark of a holding whose cost basis sits elsewhere, so moving the pool moves the score. That seam is already covered. The ValuationContext.medianReads plumbing written for this is reverted unused; it is in the history if a venue ever marks a holding off a pool quote again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CLAUDE.md described scoring as taking `realizableWethWei` and dropping a queue that cannot finalize inside the run. That was an earlier revision. The value series sums `valueUsdc`, which the adapter sets to face value (shareAssets + claimable + reachable + unreachable, times the WETH fair); the realizable number goes to `liquidatableValueUsdc`, reported only for the agents whose two marks disagree. So a pending withdrawal that finalizes after the run is reported as unrealizable and still counted at par in the score. That is the behaviour issue #38 set out to avoid, and it is now also what ADR 0019 §3 asks for by choosing the ordinary live mark. Recorded as an open item on the ADR rather than resolved here: `lst` is outside the competition set, so nothing is mis-scored today, and picking one of the two marks is a decision about what the venue is supposed to measure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
matrix.json recorded netPnlUsdc and alphaUsdc, both of which are differences, and pointed at a run directory for anything else. Of the 30 runs in the 2026-08-09 sweep, 5 had already lost their directory, so those scenarios could not be rescored under a changed rule -- which is exactly what standings.json being a derivative of matrix.json is supposed to allow (ADR 0017 §4). summary.json already carried initialValueUsdc and finalValueUsdc per agent; they now ride into the matrix. Copied for disqualified agents too: the disqualification is a ranking rule, and dropping what the agent actually did would make that rule unauditable. Note this does not retrofit the runs whose directories are gone. It stops the next sweep from having the same hole. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#56) config/regimes/b-harness.yaml runs a week at the shape ADR 0019 scores: 42 epochs of 12 blocks, the §6 funding (1 ETH + 100k USDC, economic gas), and three depeg windows instead of one, because a metric with a std term has to be measured against a series that varies. The funding numbers live here and nowhere else -- the seven official regimes keep 100 ETH + 25k, since they are the practice harness and changing their funding would break comparison with every stored matrix. First run (b-harness#801, 504 blocks): - The §4 KPI holds: noop comes last, and the ranking matches netPnlUsdc. - lambda = 0.25 is a hard hurdle at this shape. One of four agents clears it (0.273 against 0.25), and two agents with positive netPnlUsdc (+63.57, +9.59) score negative. - The noise floor is still the gas reserve's beta. noop's std_e is 2.214e-4 against venue-arb's 2.213e-4 and peg-arb's 2.125e-4 -- effectively the same number. The 199.9% spread measured on the event-dense 120-block run falls to 38.6% over a week, because an agent moved in only 2-11 of the 41 epochs while ETH moves in all of them. So the metric works and the calibration does not yet: lambda, the event density, the epoch length and the gas reserve all trade against each other, and picking one needs several seeds at this shape rather than one. Two incidental findings, both recorded on the ADR: `run.blocks: 504` yields 41 epochs rather than 42 (the scoring window is fromBlock..toBlock = 503 blocks), and weekly gas cannot be measured from the artifacts at all -- blocks.csv carries priorityFeeWei only, and no gasUsed or effective price is recorded anywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#56) The single-seed reading ("lambda = 0.25 is a hard hurdle") does not survive four more draws, and the conclusion moves. Across seeds 801-805, mean/std per agent is peg-arb-eager 0.281 (sd 0.034), peg-arb 0.111 (0.039), venue-arb 0.051 (0.039), noop 0.022 (0.020). The ordering is stable -- the gap between the two peg-arb variants is about 4.4 seed-sds -- and lambda falls between them, so it is separating strategies rather than noise. noop comes last in all five, which is §4's KPI. What is miscalibrated is not lambda but what it is measuring. noop's std_e (2.22e-4) is 93% of peg-arb's total (2.38e-4): the gas reserve's beta, which shows up in all 41 epochs, swamps the agent's own risk, which shows up in the 3-16 epochs where it traded. Taking the own-risk component as sqrt(total^2 - noop^2) puts peg-arb at a Sharpe of 0.31 and eager at 0.375 -- peg-arb flips from never clearing lambda to clearing it. So the reserve's beta has to come out before lambda is tuned against it. Also recorded: venue-arb sent no transactions at all in three of the five seeds (its scores are identical to noop's there), so the evidence rests on two active strategies, not four. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
) The metric catalogue defines M9's x_e as the excess log return over the benchmark. ADR 0019 §1 wrote the formula without the benchmark term while §2 discussed its effect, and the implementation followed §1. That gap is what the five-seed calibration had been measuring. Scored raw, noop came out at -5.07e-5 with a std_e of 2.22e-4 -- 93% of an active agent's dispersion. Every agent holds the same 1 ETH gas reserve, so that number was ETH's volatility, not anyone's risk, and lambda was being calibrated against it. Subtracting the benchmark epoch by epoch cancels it: noop becomes exactly 0 (mean and std both), peg-arb's std_e falls from 2.38e-4 to 9.08e-5 and its Sharpe rises from 0.111 to 0.241, and the ordering is unchanged. §4's claim that holding cash scores exactly zero is only true in this form. The benchmark is the roster's `baseline: true` entry rather than a synthetic flat series: it has to carry the same gas reserve as everyone else for the reserve to cancel. Without one the returns stay raw, `benchmarkApplied` says so, and the run warns. This also retires the funding argument in §6. Rescoring the two 120-block controls as excess returns, lambda separates the field under the old distribution too (100 ETH + 25k: 0.725 / 0.523 / 0.399 / 0.000), so "the current distribution stops lambda discriminating" was a property of the missing subtraction, not of the funding. The §6 distribution is kept for a different reason, now stated: a benchmark that is 92% ETH makes "risk against the benchmark" and "risk" different things, and shorting ETH would be scored as a deviation rather than a position. All five B harness seeds were rescored from their stored epoch series without re-running anything, which is the first payoff of storing raw boundary values rather than derived scores. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ADR 0017 §4 and ADR 0019 both promise that the scoring rule can be changed and past runs rescored -- that is why matrix.json keeps raw scores and summary.json keeps raw boundary values. Nothing had ever exercised it, so the metric comparison the decision needs was being done in throwaway one-liners. `npm run metrics -- <runDir...>` now reads the stored epoch series and reports M1, M4, M7, M9, M13 and M27 side by side, with the rankings each produces and where they disagree. On the five B harness weeks it says something the single-metric view could not: M9 at lambda = 0.25 is the only candidate that ranks doing nothing above an active, profitable agent. M1, M4, M7 and M13 all put peg-arb (+92 to +165 USDC/week) above noop in every seed; M9 drops it below in three of five, because its per-epoch Sharpe (0.174-0.300) straddles the hurdle. Borda over the set: eager=5, peg-arb=11, venue-arb=17, noop=17 under every other metric, against eager=5, noop=12, peg-arb=16 under M9. Lowering lambda closes the gap exactly where the arithmetic says it should -- at 0.20 noop falls to third, at 0.15 M9's Borda ordering matches M1's -- so this is a calibration of the hurdle, not evidence against the metric family. Which of the two to move (lambda, or the environment's event density, which ADR 0019 §4 says to reach for first) is still open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The numbers behind the metric decision were spread across ad-hoc scripts,
commit messages and a Notes section in ADR 0019 that had started to swell.
docs/scoring-metric-measurements.md now holds them: which runs were used,
the benchmark-subtraction finding, every candidate metric's ranking over the
five B harness weeks, the lambda sensitivity, the funding controls, and --
deliberately -- the things that cannot be measured yet.
The ADR keeps the conclusions that bear on the decision and points at the
record for the numbers. Two documents rather than one because they change on
different clocks: measurements grow with every run, the decision only when
it is actually revised, and duplicating the tables would leave one of them
quietly wrong.
Correcting one such staleness in the process: the ADR still said the §4 KPI
held in all five seeds ("noop comes last"), which was true of the raw-return
scoring it was written against. Under the excess returns the metric is
actually defined on, noop ranks above a profitable agent in three of five.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…56) The first version assumed the reader already knew what an epoch was, which agents `peg-arb` and `venue-arb` are, what the catalogue's M-numbers mean, and which ADR section each conclusion hangs off. Rewritten to stand on its own: how the competition and the score work, what each candidate metric is, what the four agents do, what the events in the test week are, and only then the numbers -- which are unchanged. Also promotes one line in the open questions from a footnote to its own row: the test week contains depeg windows and nothing else, which is why `venue-arb` never traded in three of the five seeds. Measuring lambda against a single flavour of opportunity is a property of the harness, not of the metric. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…56) Two of the seven competition regimes were run-wide settings: a run either had a drifting reference price, or uninformed flow that leaned together, or it did not. That is expressible when a run is one scenario and meaningless when a run is a week that has to contain several episodes on an undisclosed schedule, so neither could enter the continuous economy at all. They are now event types with the same trapezoid as everything else: - `cexDrift` adds to the OU's per-block drift and weakens mean reversion by `kappaMultRange` while its window is open. A new event kind, `process`: an overlay multiplies the price the walk produced and leaves the walk itself untouched, which is right for a gap that heals, but it would let mean reversion erase a drift episode the moment the window closed. The seed picks the direction unless `side` fixes it. - `flowTrend` multiplies the uninformed flow's size and holds a `trendCorrelation` / `persistBlocks` lean. The size fades with the trapezoid; the shape knobs do not, because half a correlation during the ramp is a different regime rather than a weaker one. Applied by the coordinator when it fills the flow bot's per-block wire, so the bot needs no knowledge of events and there is only one copy of the schedule. Calibrated from the regimes they replace: drift 0.0015 with kappa 0.004 against the 0.02 default (`cex-drift.yaml`), and 3x size with persistence 12 and correlation 1.0 (`informed-flow.yaml`). The b-harness week now carries both alongside its depeg windows. That is the point: measured over five seeds, a week of nothing but depeg gave the cross-venue arbitrageur no work at all -- zero transactions in three of them -- so the scoring metric's lambda was being calibrated against a single kind of skill. Verified on b-harness#801: two cexDrift episodes (the seed drew one up and one down), one flowTrend, three depegs, all in their windows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reran the five B harness seeds with the two newly windowed regimes (a drifting reference price, and order flow leaning across every venue) added to the depeg windows. The ranking picture is unchanged: peg-arb still straddles lambda = 0.25 and doing nothing still places second overall. The cross-venue arbitrageur still sent zero transactions in three of the five seeds, and its own log says why -- 503 rounds of "the widest gaps need inventory this agent does not hold". Taking a cross-venue gap means selling on the expensive side, which needs ETH. The roster is funded in USDC only (1 ETH for gas), so the strategy can only take gaps that happen to point the other way, and is silent whenever they do not. Not a shortage of opportunity, and not a flaw in the agent: a constraint of the distribution. That constraint is now obsolete. USDC-only funding existed because the old metric charged everyone for the drift of whatever they were handed; the benchmark subtraction cancels anything every entry holds equally, the benchmark included. Funding the roster with ETH is therefore safe again, and is the next thing to measure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The competition's regime table calls regime 5 a *non-mean-reverting* deviation. The implementation was the opposite: every depeg window ended with the environment buying the stable back, and the OU always pulls the price to the anchor it started from, so nothing in the system could express "the market repriced and stayed there". That is not a cosmetic gap. With a guaranteed restore, "will par come back?" has a known answer, so buying the discount is a free option bounded only by size and the more patient of two otherwise identical strategies wins by construction. It is a plausible explanation for peg-arb-eager -- which buys a shallower discount and holds for par -- winning all five seeds by a factor of three to five. Two flags, both off by default: - `persist: true` on depeg/eusdDepeg holds the level to the end of the run instead of decaying. Requires `decayBlocks: 0` rather than ignoring it, so a config cannot say "closes over 15 blocks" and mean the opposite. The teardown still buys back after the last competition block -- the startup check refuses a depegged pool, so leaving it would stop the next run -- and that is outside every scored cross-section. - `repriceAnchor: true` on cexDrift moves what the walk mean-reverts to, by exactly the drift the episode applied, compounded the same way the OU compounds its own step. The B harness week now carries one of each, alongside the ones that close. Its funding also gains 20 WETH: USDC-only was a rule the old metric needed (it charged everyone for the drift of what they were handed) and the benchmark subtraction has retired it, while the absence of inventory was keeping the cross-venue arbitrageur out of three seeds in five. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…th it (#56) Third calibration pass, stopped after two of five seeds. Recorded because both halves of what it found matter. Funding the roster with 20 WETH took the cross-venue arbitrageur from 0-16 transactions to 274-296, and from 0-250 USDC to +3,405 and +4,146. It was never short of opportunity; it was short of the inventory needed to sell on the expensive side. It is also, once it can trade, the most profitable strategy in this environment by a factor of ten to forty. And that broke the harness's timing. Per-round work went from 60/90ms (p50/p95) to 331/2128ms as anvil's execution queue backed up behind the extra transactions, so 32 rounds overran the 2-second block interval and 43 blocks passed with no price update and no injected flow. The week stretched from 504 chain blocks to 547-561, and the stalls land inside the event windows -- the exact places the measurement is about. These numbers are not comparable to the two earlier passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hain (#56) `buildFlowContext` awaited each wallet in turn: 8 flow wallets, then 4 aave actors at two reads each, ~16 sequential round trips per block. The client batches contract reads into Multicall3, but only across calls issued in the same await -- so the batching never applied to the loop that needed it most. On an idle chain that cost 60ms per block and nobody noticed. Once the roster was funded with WETH and the cross-venue arbitrageur started sending ~290 transactions per run, anvil's execution queue made each round trip ~5x slower and the same 16 hops became 331ms (p95 2128ms) against a 2-second block budget. 32 rounds overran it and 43 blocks passed with no price update and no injected flow -- concentrated, of course, inside the event windows. Issuing them together (b-harness#801, same seed and roster): stateFlow p50/p95 331 / 2128 ms -> 17 / 35 ms round p50/p95 331 / 2128 ms -> 23 / 44 ms blocks skipped 43 -> 0 scored window 547 blocks -> 503 (41 epochs, comparable again) Faster than the pre-WETH baseline (60/90ms) as well, because the batching now actually happens. The clock stays at 2 seconds; the earlier plan to slow it to 4s was treating the symptom. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cleanest pass yet -- five seeds, no skipped blocks, 41 epochs each, three strategies active throughout -- and it settles the lambda question. venue-arb earns +2,951 to +5,245 USDC a week, ten to forty times what the peg strategies make, on roughly 163k of capital. Its dispersion is twenty times theirs, so its per-epoch mean/std is 0.199 and it clears lambda = 0.25 in one seed out of five. M9 therefore ranks it last, below doing nothing, while M1, M4 and M7 all rank it first. Its weekly Sharpe is about 1.28 at a 2.5% weekly return. Calling that "below the minimum needed to beat sitting in cash" is not defensible, so lambda = 0.25 is too high. At 0.15 the ordering keeps the earner on top and still demotes the strategy that earns little for its dispersion, which is what the risk term was chosen to do. Also records the falsified hypothesis from the previous pass: making a depeg permanent did not punish the patient buyer. A buyer purchases at the discounted price and is marked at the discounted price, so a peg that never returns is a missed profit rather than a loss. Punishing it needs the dislocation to deepen after entry -- a long ramp -- not merely to persist. (Restores the "not yet measurable" heading, which the previous doc commit dropped while inserting a section above it.) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… to nine (#56) Four agents were measuring lambda against two kinds of skill, and two of the four were the same code at different thresholds. A survey of all 21 agents in example/agents found nothing that borrows: `lst-carry` is the only one, and it needs a venue outside the competition set. So three of the metric's open questions had no instrument at all. `levered-long` is that instrument. It supplies its base as collateral, borrows stables, buys more base, and lands on a target health factor. It is deliberately a dumb long -- a probe rather than a contender -- because what is being measured is the metric, not the strategy: - ADR 0019 §2 records "a lucky leveraged week is rewarded" as the known limitation of leaving beta in the mean. Nothing has ever taken leverage, so nobody has seen how large that is. With cexDrift now able to run in either direction, both the lucky and the unlucky week are reachable. - G1 (bankruptcy floor) and G2 (scoring freeze) have never fired -- every run reports bankruptAtEpoch: null. `levered-long-max` sits at a health factor of 1.15, where a downward drift can liquidate it. - G6 declines a leverage cap because "the protocols' collateral limits are enough". That has not been tried. It sizes borrows to *land* on the target rather than to consume the headroom Aave reports: the headroom is an LTV limit and the health factor is a liquidation-threshold one, so spending the former overshoots the latter and the position oscillates (the mistake the LST carry agent made). The threshold is not in the observation, so it is inferred from the position: measured 0.825 on the first run that opened one. Also rostered three existing agents that were never used: `multi-arb` (an independent cross-venue implementation -- it earns too, so venue-arb's dominance is the opportunity rather than one author's code), `lp-provider` (the only strategy that holds inventory and waits) and `random` (the lower control: the field had shown lambda can demote a profitable strategy, never that it demotes an unprofitable one). Smoke-tested at 72 blocks: all nine act, both levered entries land on their targets, and the coordinator stays at 18/44ms per round with no skipped blocks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Widened the calibration roster from four to nine and reran three seeds. Two of the metric's open questions now have answers, and one has a sharper statement. The catalogue rejects ratio metrics (M13) on the grounds that scale invariance makes "stay small and safe" the optimum. That is now measured rather than argued: `lp-provider` earns +26 USDC a week at a dispersion of 7e-6 and has the highest Sharpe in the field, so ranking by M13 puts the agent that made $26 above the one that made $4,890. M9 keeps its absolute scale and does not invert. The lower control also holds: `random` loses 7,000-8,000 a week and comes last under every metric, so lambda demotes an unprofitable strategy as well as a profitable one -- the direction the field had never tested. And the lambda problem is sharper than before. `multi-arb` (0.236) and `venue-arb` (0.186) are independent implementations of the same strategy and land on opposite sides of the 0.25 hurdle. The threshold is cutting through a strategy class rather than separating classes. The leveraged probes lost in all three seeds and were never liquidated, so neither the "lucky leveraged week" that ADR 0019 §2 records as its known limitation nor the bankruptcy gates G1/G2 have been exercised. Both need a run with the drift direction and magnitude fixed rather than more random seeds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Built a run whose purpose was to bankrupt somebody -- five times leverage against a 22-28% crash -- because G1 (the bankruptcy floor) and G2 (the scoring freeze) have never executed. Nobody went bankrupt, and the two reasons are both worth having. The agent could not reach five times leverage. It got to a health factor of 1.10, about 1.45x, because the per-round caps (5,000 USDC borrowed, 5 WETH supplied, 5,000 USDC swapped) make the loop that would take it further longer than the week itself. So ADR 0019's G6 has its reasoning backwards: it says the per-round limits cap size rather than leverage and the protocols' collateral ratios do the real work, when in fact `limits.*` is what binds. Under the current calibration a participant cannot take dangerous leverage at all, which makes those caps a decision about how much risk management the competition measures -- not, as the ADR files it, a question of how many rounds it takes to deploy capital. And an underwater position is not liquidated unless somebody calls liquidationCall. The health factor reached 0.975 and nothing happened: no liquidator in the roster, and the environment only liquidates the victims it plants itself. The crash decayed, the position recovered to 1.523, and the agent finished +768 ahead. G1/G2 verification is also less urgent than it looked. With the field-relative z-score retired, each participant's score comes from its own series, so one blowup cannot poison anyone else's. The arithmetic is pinned by unit tests; what remains untested on-chain is only whether the scorer really reports a negative value for a wiped-out Aave position. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit carried the ADR correction but not this section: the insert targeted the '10. not yet measurable' heading and replaced it, which dropped the heading instead of writing above it -- the same slip as two commits ago. Restored, with the run's findings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Yesterday I recorded a correction to ADR 0019's G6: that the per-round `limits.*` caps, not the protocols' collateral ratios, are what limits leverage, on the evidence that an agent targeting five times leverage stopped at 1.45x. That attribution was wrong and is retracted in both the ADR and the measurement log. The agent stopped because its borrows were reverting. Its own log says so -- 19 submitted against 483 submit_failed, 480 of them `panic 0x11` -- and I did not read it before concluding. The cause is Aave's USDC liquidity: under local deploy the pool holds the 100,000 USDC that `LOCAL_FLASH_LIQUIDITY_USDC_UNITS` supplies at startup, and two leveraged agents plus four Aave flow actors exhaust it, after which every borrow fails with an arithmetic panic that names nothing. So G6 stands as originally written, and where the leverage ceiling actually sits is simply unmeasured: nothing in this environment can borrow enough to find out. That also voids the leveraged agents' results as evidence about leverage -- their losses are mostly failed transactions -- and leaves ADR 0019 §2's "lucky leveraged week" and the G1/G2 bankruptcy gates still untested. Fixing it is one constant, applied by cheatcode at startup, so no state dump rebake. The b-harness keeps its raised caps, but for the other reason they were useful: the three peg-arb entries differ only in the fraction of balance they trade, and at the practice regimes' 5,000 USDC cap the 5% and 10% twins both clamp to the same number -- which is how the first attempt at that comparison measured nothing. ruin-test goes back to the standard caps. Unaffected by any of this: the size comparison itself (Sharpe 0.311 -> 0.188 -> 0.113 as one strategy's trade size goes 5% -> 10% -> capped), the Sharpe metric's inversion, `random` as a lower bound, and lambda's behaviour. None of them touch Aave. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rage (#56) The hypothesis from the retraction was right: the borrows were failing because Aave's USDC was gone. Raising the pool's startup liquidity from 100,000 to 2,000,000 -- one constant, supplied by cheatcode at run start, no state dump rebake -- takes the leveraged agent from 19 submitted / 483 failed to 94 / 0, and it reaches its 1.05 health factor target (about 4.5x) in 96 blocks where 504 were not enough before. So this environment has been unable to support a borrowing strategy at all, and said so only through an arithmetic panic that names nothing. Any participant trying to lever would have seen their transactions fail for no stated reason. With that fixed, a 25% crash against 4.5x leverage finally produces a real drawdown: value 163k -> 43k -> 121k, a net loss of 43,750 and the worst score in the field at -6.02e-2, thirteen times worse than `random`. The metric punishes a leveraged blowup exactly as designed -- not only for the loss but for the dispersion, since a 74% fall followed by a 2.7x recovery is two enormous epoch returns. G1 and G2 still did not fire, and now we know why rather than guessing: the bankruptcy floor sits at 1% of the starting value (1.6k of 163k) and the worst trough the environment produced was 43k, or 26%. Reaching the floor needs serial liquidation or a crash an order of magnitude larger. That settles what G1 is for -- a guard against `ln` of a negative Aave position, not a ranking device -- and it is the last of ADR 0019's rules to have gone untested for a reason we can now state. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ADR 0019 carries both lambda = 0.25 and a 4-hour epoch as provisional numbers. Rescoring the stored per-block values at 6, 12, 24 and 48 blocks per epoch -- no rerun needed -- shows they are not independent. Sharpe scales with the square root of the epoch length (the mean grows linearly with the period, the dispersion only as its root), so a fixed lambda means something different at every epoch length. At 6 blocks the agent earning +26 a week ranks first under lambda = 0.25; at 24 blocks the one earning +4,890 does. Lengthening the epoch from 12 to 24 blocks achieves what dropping lambda from 0.25 to 0.15 achieves. Approximately, the hurdle's bite goes as lambda / sqrt(epoch length). So the choice is one of the pair, not two numbers, and which to move is a separate question: a longer epoch coarsens the measurement (10-20 observations in a week) but leaves fewer epochs in which nothing happened, while a lower lambda keeps the resolution and weakens the penalty on dispersion. The 0.15 recommendation is therefore a recommendation about a pair, at this harness's 12-block epoch. A live 4-hour epoch needs reading across by how much trading a real epoch contains, not by the number itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two additions close the metric evaluation. The last untested claim was ADR 0019 §2's own: that leaving drift in the mean rewards a lucky leveraged week. Pinning both drift episodes upward produced one. `levered-long` earned +2,122 and ranks 2nd of 11 under M1, M4 and M7 -- and 9th under M9, because its dispersion is 99 times that of the small peg arbitrageur. So the "partial answer" the ADR credits lambda with is, in this environment, close to a complete one. The same run also shows leverage is not a direction bet you win by sizing up: at 4.5x the agent lost 13,117 in a week the drift favoured, because defending its health factor means selling into every dip. And §9.8 now states a verdict per metric with the measurement behind it, rather than leaving eleven sections of evidence for a reader to assemble: M9 conditional on the lambda/epoch pair, M4 and M1 defensible but blind to the lucky gambler, M7 worthless here (identical to M1 at every rho tried), M13 unusable (it crowns the agent that earned $26), M27 fine as an aggregator. What remains is not a measurement. §9.6 showed lambda and the epoch length are one knob, so the decision is where to put the pair -- and that is a judgement about how much dispersion the competition should tolerate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The evidence sections were current; the pages a reader meets first were not. §3 still described a four-agent roster funded with 1 ETH and 100k USDC, five agents and 20 WETH ago, and described the test week as three depeg windows when it now carries six windows of three kinds, two of which do not close. It now lists all eleven agents with the section each was added for. §4 listed four runs out of nine, so the numbers in §9.3 onwards could not be traced to what produced them. It now lists every run with its roster size, what changed, and which sections rest on it -- plus the warning that follows from that: conditions differ between runs, so figures are comparable within a run and not across them. §10 still claimed the epoch length had never been varied, which §9.6 did. §11 had grown to nine rows of which five were already resolved, including one asking for agents that had since been added; the resolved ones move to a single line so the open questions are legible again. The measurement sections themselves are untouched. They are a record of what was observed under the conditions of the time, including the readings later retracted, and editing them to match today would destroy exactly what makes the retractions useful. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The measurement log had grown into an experiment notebook and a report at once, and read as neither. The verdict on the six candidate metrics sat at line 517 of 572, so a first-time reader had to walk the whole time-series -- including the interpretations later retracted -- before learning which metric was chosen. Restructure around the reader: the verdict moves to §1, §2-§4 are the premises, and §5 is the measurement log, explicitly marked as not needing a read-through. Sections whose numbers are no longer usable now say so in the heading (§5.5's leverage figures were invalidated by the Aave liquidity defect). The old §9.1-§9.8 were `##` headings -- siblings of §9 -- while numbered as if they were its children. They are now §5.1-§5.7 at `###`, so the numbering matches the hierarchy, and the parallel "結果 N" series is gone. Merges, not deletions: §6+§7+§9 covered one question with one roster and became §5.2; §9.4+§9.5 are one story (a broken measurement and its fix) and became §5.5; the funding control experiment became a note in §4. Every figure survives -- the only table dropped is a per-seed breakdown whose content is stated in prose two lines away. Fixes along the way: - §11 credited the persistent-depeg finding to §9.4; it is from §9.2 - §9.8 cited itself - the "その他" paragraph in §10 had no blank line before it, so it rendered as part of the G7 bullet - the run table was missing the five-seed run behind §9.2 (it lives across matrix-2026-08-14T06-08 through 10-18) - exponent notation had no key; §2 now gives one 571 -> 420 lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
ADR 0019 picked
mean(x_e) − λ·std(x_e)as the competition score with λ = 0.25, both provisional. This branch builds the machinery to check that choice against data, runs the measurements, and writes down what they say.It does not close #56. The metric family is settled by the evidence here; the λ / epoch-length pair is not, and the provisional z-score is still live in
standings.ts. Refs #56.What it delivers
Scoring machinery —
core/src/scoring/epochScore.tscomputes the ADR 0019 score from an epoch value series,metrics.tscomputes every candidate (M1 / M4 / M7 / M9 / M13, plus M27 Borda across runs) from that same series, andnpm run metrics -- <runDir...>rescores stored runs with no chain and no re-run.reconstruct.tsproduces the series and medians the manipulable marks at each epoch boundary (G7).matrix.jsonnow carries the two cross-sections the metrics are differences of, so a stored matrix stays rescorable after its run directories are gone.Environment work the measurements needed —
cexDriftandflowTrendbecome injectable windows instead of whole-run config;persistlets a dislocation stay open; three regimes (b-harness,ruin-test,lucky-drift); alevered-longagent, because no existing strategy usedaaveBorrow; and a fix for the Aave pool having only 100k USDC to lend, which was silently failing 99% of borrows as an arithmetic overflow.The record —
docs/adr/0019-*.mdanddocs/scoring-metric-measurements.md.What the measurements say
λ / √(epoch length); λ = 0.25 at a 24-block epoch ranks the same as λ = 0.15 at 12. So λ cannot be chosen alone — this is the open decision.random.Testing
test/epochScore.test.ts,metrics.test.ts,markMedian.test.ts,epochBoundaries.test.ts,events.test.ts,fundingGasBuffer.test.ts. The measurement runs themselves are inruns/and reproducible withnpm run metrics.🤖 Generated with Claude Code