Scenario-based evaluation (ADR 0017) and LLM-rewritten strategies (ADR 0018) - #57
Merged
Merged
Conversation
…egimes
Makes a scenario -- (regime, seed) -- the unit of evaluation, and adds the
runner and aggregation that turn a matrix of them into standings. ADR 0017
records the design and the decisions behind it.
Scenario runner
- Regimes no longer carry `run.seed`. It is the second axis of a scenario, so
it is supplied per run and `--regime` without `--seed` now fails instead of
silently scoring seed 1. Regime files lose their `-01` suffix accordingly.
- `--scenarios <set>` replays a whole {regimes} x {seeds} product on one anvil,
snapshot/revert between scenarios, and writes runs/matrix-<id>/:
matrix.json raw per-scenario, per-agent scores (both netPnlUsdc and
alphaUsdc, plus disqualifications and run dirs)
standings.json the ranking derived from them
The ranking is a derived view on purpose -- the scoring rule is expected to
change, and the raw matrix is what lets it be recomputed without re-running.
- Aggregation lives in core/src/backtest/standings.ts as pure functions:
z-score across the agents within a scenario (they shared a world, so that is
the one comparison the design guarantees is fair), then equal weight per
regime so a big-opportunity regime cannot decide the ranking on its own.
A rule-breaking, crashed or absent agent is placed below every finisher
rather than scored zero, which would make crashing a viable tactic. A
scenario that produced no result is excluded, not charged to the field.
- `--score-every N` thins the value cross-sections. Score-neutral: only the
first and last reach summary.json, so it only coarsens the equity curve.
New regimes
- cex-drift: the OU volatility/kappa/drift were read straight from process.env,
which a regime YAML cannot set. Added a market.* config section (global and
per-base) and routed the coordinator through SimConfig instead of globals.
- informed-flow: the persisted uninformed trend draws its direction per venue,
which manufactures a cross-venue spread -- the opposite shape from correlated
directional flow. Added flow.uninformedTrendCorrelation; at 0 (the default)
the RNG consumption sequence is unchanged, so existing calibration holds.
- whale: a point stress event that prints one large market order and moves the
mid while fair stays put. Trades from its own wallet, endowed at setup from
the resolved schedule, relayed through the ordinary flow path so it is
indistinguishable from background flow except by size.
Fix: netPnlUsdc valued non-WETH bases at zero
The end-of-run summary passed valueUsdc the scalar WETH price, which marks any
other base at `p[sym] ?? 0` -- so an agent ending the run holding WBTC had that
inventory deleted from its PnL. Same class of hole as issue #41, on the scoring
path #41 did not touch, and it matters now that the competition ranks on
netPnlUsdc. Found in the pilot: four WBTC-trading agents reported an identical
-6,686 USDC loss (to the cent, across two repeats) while the reconstruction put
them at +13 alpha. Now passes the per-base price map. See issue #53.
Pilot findings (recorded in ADR 0017 section 5)
- Co-location is limited by opportunity dilution, not by load. anvil survived
24 agents with mining keeping pace, but arbitrage profit fell to 1/30 at 16
agents and went negative at 24. The participant cap should be set at the
density where the field still separates, not at the breaking point.
- A 30-scenario public sweep reproduced the known ordering (multi-arb >
venue-arb > noop) in all five regimes, and exercised the exclusion path when
lending-incident hit its aave precondition.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two that mattered most made the informed-flow regime measure the wrong thing entirely. Trend direction was memorizable and partly pinned (core/src/flow/logic.ts) - The direction was a function of the block window alone -- no seed input at all -- so it was identical on every published seed and on every unpublished one. Under `uninformedTrendCorrelation: 1` the whole market flipped on an exact 12-block clock, and an agent could hard-code `floor(round/12) % 2` and front-run every reversal. The private seed set would have protected nothing. - The FNV step used float `*`, and 2^32 * 2^24 exceeds 2^53, so the product's low bits were rounded away before `% 2`. Measured: "uniswap" and "balancer" returned an even hash in *every* window -- a permanent one-way bias, and no divergence between the two deepest venues, which is the opposite of what the default (correlation 0) calibration is supposed to produce. - Both fixed by seeding the trend from flowSeed and reading the bit off an Rng (an LCG: bool() takes the high bit, the well-distributed end) instead of an FNV parity. Measured over 200 seeds x 50 windows: p(1)=0.504, flip rate 0.481. flowSeed now travels on the flow wire. The trend still consumes no draws from the shared RNG, so enabling it does not shift downstream orders. - The old tests passed because they only asserted `new Set(dirs).size === 2`, which a perfect alternation satisfies. Rewritten to check seed-dependence, absence of pinning and of fixed alternation, and pairwise venue independence. Whale endowment (core/src/realtime/whale.ts) - Sized on the largest single order, which only looks sufficient because buys and sells replenish each other; a seed drawing all four whales the same way (p ~ 1/8) outspends it and the last order reverts. Now sums per side. - Hard-coded to WETH while the parser accepts `base` on a whale event, so a WBTC whale was funded in the wrong token at the wrong price and could never execute. Now per-base, and a base with no fair price fails fast. - A whale pointed at a venue the run does not enable is now rejected at setup rather than swallowed by the relay's catch. Other review findings - market.base* overrides for WETH parsed, typechecked and did nothing: the WETH path read ou.global while only extra bases read ou.perBase. - informed-flow raised balancerMax/curveMax, which are a single cap for BOTH the uninformed and the informed leg -- tripling the gap-closing flow it claims to leave alone. Reverted to calm's value, with the coupling documented. - matrix.json/standings.json are written after every scenario instead of only at the end, so a crash 29 scenarios into a ~6 h matrix no longer discards it all. - foldRepeats took independent medians of netPnlUsdc and alphaUsdc, reporting a pair no single run produced next to a runDir explaining neither. Now picks the repeat at the median of the ranking metric and reports that run's record. - valueSeries reported granularityBlocks: 1 and a full block count even under --score-every, mis-scaling any curve derived from it. Now reports the stride and the cross-section count, with windowBlocks alongside. - A heterogeneous default roster across regimes (lending-incident carries a liquidator) makes totals incomparable, since `total` averages only the regimes an agent appeared in. Warns and points at --agents. - ERIS_PRICE_* joins RETIRED_CONFIG_ENV: the coordinator reads config.ou now, so a stale env var is a silent no-op. rng.ts's env accessors are marked legacy. - Restored the per-repeat spread output that --repeat exists for, and corrected two guides that still described the removed mean-alphaUsdc line. - Redundant per-scenario effective-YAML writes and per-seed venue revalidation hoisted to once per regime. A whale that reverts is no longer silent Fixing the whale's redundant funding by skipping it in the setup loop also skipped that loop's setupWallet approvals: all four whale orders then reverted on-chain while the schedule, the event log and the submission path all reported success, and the regime quietly became calm (venue-arb 1604 -> 0 txs, multi-arb 2772 -> 204). The loop pass is restored, and countRunRevertedTxs now checks the whale's txs after the run and both logs and prints a warning when any reverted. Verified: "[stress] 4 whale orders executed", venue-arb 1593 / multi-arb 2893. 343 tests (339 pass, 4 skip), typecheck, boundaries and strategy gate all green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rules, record B1 Step A of the follow-up plan (documentation consistency), plus the first production-scale pilot measurement. ADR 0016 §3 no longer contradicts ADR 0017 It still said every override including --seed was smoke-test-only and that scores may be read from regime-default runs alone. Since a scenario is now (regime, seed) and regimes carry no seed, --seed is a required input rather than an override, and a run without one cannot happen (the CLI fails fast). Noted inline; the rest of the section -- blockTimeSec, blocks, protocols -- stands unchanged. Co-location is now stated as a competition rule, not left in the ADR docs/guide/backtest.md gains "What the competition actually scores": shared finite opportunities (arb profit per agent fell ~30x from 4 to 16 co-located agents), in-block ordering bought with gas, and the speed difference between rule and prompt agents. Also documents how operators build the private seed set, and why nothing but the seeds is secret. ADR 0017 -> Accepted, with what is not done stated up front crash's liquidity withdrawal (#52), depeg, and the §5 calibration values are listed in the status block rather than left for a reader to discover. B1: prompt-mode agents are far more disadvantaged than estimated Ran calm#101 at production settings (R=360, 2 s blocks, 718 s) with the same strategy in both runtimes, co-located: multi-arb rule every block 192 tx +254.4 multi-arb prompt 13 decisions, 56.1 s 3 tx +0.7 venue-arb prompt 44 decisions, 15.9 s 0 tx 0.0 That is one decision every 8-28 blocks and 1/64 the actions, against an estimate of one per 5 blocks. The ADR's R=360 rationale claimed ~72 LLM decisions per scenario; the measurement is 13-44, so that claim is corrected rather than left standing. The number is claude-cli specific -- it spawns a process per call and the two prompt agents contend for it -- so an API backend would be faster. But closing an 8-28x gap by raising blockTimeSec would break the one-week budget in §3, which pushes toward implementing the third agent mode (a fast rule loop the LLM periodically rewrites) instead. §6 now spells out why that mode does not exist today: prompt mode routes *every trading decision* through the LLM, and ERIS_PROMPT_REVISE_EVERY (default off) only rewrites the prompt body. Also found: the bundled venue-arb self-rejected 359 of 359 decisions in calm ("amountIn exceeds balance") -- it only ever proposes selling WETH while funded USDC-only, so it never bootstraps inventory. It is one of the three known-strength agents the pilot used, which means that check confirmed less than it looked like. Filed as issue #54. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…the LLM mode change
B2: the co-location limit is still unknown
Ran calm#101 at production R=360 with rosters of 8/16/24/32, cycling the five
bundled arb strategies. The measurement cannot answer the question it was asked:
roster multi-arb mean cross-venue mean agents that never traded
8 -79.7 +11.1 2 / 7
16 -79.4 -13.4 6 / 15
24 -74.8 -14.7 9 / 23
32 -65.0 -12.2 18 / 31
- More than half the bundled agents never trade in calm. clean-arb and stat-arb
report exactly 0.00, and at 32 so does adaptive-arb -- the same symptom
isolated for venue-arb in #54 (only ever proposes selling WETH while funded
USDC-only, so it self-rejects forever). At a nominal 32 agents only 13 were
touching the market, so neither the load nor the competition was what the
roster size suggested.
- Duplicate instances of one strategy cannibalise. multi-arb made +254 alone in
B1 and sits at -65 to -80 whenever two or more are present, almost
independently of roster size -- they chase the same gap and pay the fees.
That also reinterprets the first pilot: "opportunity dries up at 16 agents"
(R=40) was most likely the same cannibalisation, not dilution across a field.
Real participants submit different strategies, so neither number transfers.
anvil itself stayed healthy at 32 (360 blocks in 720 s, no early exits), so the
load ceiling has not been found either. Re-measuring needs distinct, actually
trading strategies, which depends on #54.
ADR 0018: LLM rewrites the strategy instead of making every trade
Records the decision to drop prompt mode and move the LLM outside the trading
loop, with B1 as the evidence and the deleted src/llm two-layer machinery as
prior art -- including that it lost to frozen strategies on multi-seed
validation and its rollback never fired in 18 runs. The ADR is explicit that
the design has to answer why this time differs (alpha-dominant env, in-run
rather than across-run feedback, self-improvement skill as the competitive
axis).
Decided: the LLM emits executor code run in a node:vm sandbox; participants
submit an initial strategy plus an improvement prompt; bad revisions roll back;
and the improvement prompt decides when revision fires -- bounded by an
operator cap, since a participant declaring "revise every block" would consume
the shared LLM budget in a co-located run. Generated code must pass
check:strategy before it is installed, and a frozen control stays in the roster
so "is self-improvement actually winning" is visible every run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…018 Phase 1) Prompt mode put the LLM in the trade path: one decision cost a round trip, which measured at 8-28 blocks per decision and 1/64 the actions of the same strategy in rule mode (ADR 0017 §5 B1). It is removed. The LLM now sits outside the loop and rewrites the strategy the loop runs. A self-improving agent is agent.ts + improve.md. decide() runs every block at rule-agent speed; periodically the model is handed the current executor source and how it has been doing, and may return a replacement. improve.md is not a renamed prompt.md. prompt.md said "given this observation, what do you do"; improve.md says "when, on what evidence, and how should the strategy change". Its frontmatter declares the cadence, which is the participant's lever and costs no LLM call to evaluate -- clamped so that one agent declaring "revise every block" cannot drain the LLM budget a co-located run shares, with the clamp recorded rather than applied silently. Three guards, each because the deleted src/llm machinery lacked or under-used it (it lost to frozen strategies on multi-seed validation, and its rollback never fired in 18 runs without anyone noticing): - Generated code passes the cheatcode static check before installation. An LLM-authored strategy is not trusted code, and the submission gate cannot see code that does not exist yet. findCheatcodeUsage moved core -> sdk so the runtime can call it (example cannot import core). - Code that fails to compile, or is not a function, is never installed; the previous strategy keeps running. - A revision whose value went backwards is rolled back, and every accept, decline, rejection and rollback lands in the agent log via `state`, so "did self-improvement do anything" is answerable from one run. Returning null from the model means "keep the current strategy", and the system prompt says so explicitly -- declining to touch a winner is the direct fix for the diagnosed failure of the previous attempt (shaving upside in the regimes that were going well). ERIS_AGENT_FROZEN=1 runs an improve.md agent without the loop. That is the frozen control ADR 0018 §5 wants in every roster, without a duplicate directory. Found while testing: actions built inside the vm carry the sandbox's Object.prototype, so they are not `instanceof Object` in the host and deep equality against host objects fails. Left alone that surfaces far from its cause -- in validation or logging. The boundary now structurally clones the action back into this realm. ERIS_AGENT_MODE and ERIS_PROMPT_* now fail fast. A roster still asking for prompt mode would otherwise run as a plain rule agent and look healthy, which is worse than an error: the participant believes an LLM is involved and nothing says otherwise. Phase 2 (still to do): the 19 bundled prompt.md files, config/example.yaml's Quick Start roster, the guides, and the bundled arb agents that never trade under USDC-only funding (#54). 349 tests (345 pass, 4 skip), typecheck, boundaries and strategy gate green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reviewing Phase 1 against its actual behaviour rather than its intent. Two of
these meant the feature did not work as designed.
The first revision fired before the strategy had traded
obs.round is the absolute chain block (read.ts passes `round: bn`), and the
trigger's baseline started at 0, so the very first observation satisfied
`block - lastBlock >= reviseEvery`. The model was asked to rewrite a strategy
with zero blocks of history and no performance to reason about, spending one of
the participant's ~12 revisions on nothing. The baseline is now seeded from the
first block seen.
The evidence handed to the model was empty
`recent` only ever recorded `{ round }`, so every entry in the context rendered
as "block N: no action" -- the one channel showing what the strategy had been
doing carried nothing. Actions are produced in invokeDecide, not in the
observation stream, so the buffer is now filled there and holds the action or
the decide error.
The rollback baseline went stale
After a rollback `previous` was cleared but `valueAtRevision` was not, so the
next revision was judged against a measurement of the revision that had just
been undone.
Generated code could wedge the agent permanently
The Script timeout covers evaluating the function expression, not calling it. A
generated body that loops or awaits forever held the caller's `deciding` guard,
so the agent stopped deciding for the rest of the run while every log still
looked healthy, and the process never exited. The call is now raced against a
2 s bound (one production block) with the timer cleared on the fast path. It
cannot kill the runaway work -- vm cannot interrupt an async body -- but it frees
the loop and the throw is recorded as a decide error.
The raw revision exchange was unrecoverable
Removing prompt mode took ERIS_PROMPT_LOG_CALLS with it and nothing replaced it,
leaving a bad revision visible only through its outcome. ERIS_IMPROVE_LOG_CALLS=1
writes the system prompt, the context and the response to
runs/<id>/agents/<id>.llm.jsonl. Off by default: it holds every generated
strategy in full.
Also corrected the comment on compileExecutor, which claimed more containment
than exists. The vm removes ambient capability (no require, process, fs, fetch)
but `ctx` is passed in and carries publicClient / walletClient, so generated code
can trade exactly as freely as the strategy it replaces. That is the same
capability, not an escalation, but the comment read as though the sandbox
isolated the chain. The cheatcode check is the part that addresses intent.
Not fixed, and worth saying plainly: rollback still triggers on any loss at all
(`delta < 0`). In regimes where everyone loses that fires on every revision
regardless of quality, and run noise alone flips the sign often. ADR 0018 lists
the criterion as undecided, but shipping `delta < 0` is the opposite extreme
from the previous implementation's never-fires. It needs the frozen-control
comparison before it means anything.
351 tests (347 pass, 4 skip), typecheck, boundaries and strategy gate green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…8 Phase 2, closes #54) Fixes the bundled agents that never traded (#54) venue-arb, stat-arb and adaptive-arb picked their direction from the price gap alone and sized from obs.limits alone, never consulting the wallet. Under the competition's USDC-only funding they therefore proposed selling WETH they did not hold, the runtime rejected it, and they proposed it again the next block: venue-arb self-rejected 359 of 359 decisions in calm, adaptive-arb 353, stat-arb 21 of 21. A rejected action scores exactly like choosing not to trade, so the failure was silent, and it made those agents useless as the "known strength" controls the ADR 0017 pilot leaned on. example/agents/lib/affordable.ts holds the rule now: cap by wallet as well as by limit, and where there is a choice of venue, choose among the ones you can fund. An agent holding only USDC can still arbitrage -- it buys the cheap venue rather than selling the rich one, and sells once it has inventory. Measured on calm#101 over 60 blocks: 0 rejections across all four arb agents; venue-arb 359-rejections-and-nothing -> 12 fills and +10.5, adaptive-arb -> 2 fills. clean-arb was deliberately left alone: it already caps by balance and bundles both legs, and its zero is a disciplined strategy correctly finding no spread that beats fees -- not the same bug. Migrates the agent contract The 19 prompt.md decision prompts are removed. Three reference agents (venue-arb, multi-arb, lst-carry) gain an improve.md, which is a different kind of document: it says when, on what evidence, and how the strategy should change, and its most important instruction is when *not* to change it. config/example.yaml's Quick Start now ships a self-improving agent next to a frozen copy of the same strategy (ERIS_AGENT_FROZEN), because without the control there is no way to read whether revising helped. config/lst.yaml gets the same treatment. An LLM backend is now optional rather than required: with no key the revisions are recorded as failed and the strategies trade unchanged. Docs: llm-agents.md rewritten around the new model, and architecture, writing-agents, repository-layout, run-output, configuration, README, CLAUDE.md and ADR 0015's contract table updated. First end-to-end run of the loop (claude-cli, calm#101, 150 blocks) The model read the strategy's own noop reason -- "the widest gaps need inventory this agent does not hold", repeated for a dozen blocks -- diagnosed it as a deadlock, and rewrote the strategy. At the next opportunity it declined to touch it again because it had started working, which is exactly the behaviour ADR 0018 wants and the direct counter to the previous implementation's habit of shaving upside. No generated code was rejected. Final: +57.6 against the frozen control's +10.0. That is one run on one seed. It shows the mechanism works, not that self-improvement wins -- the previous attempt lost to frozen strategies on multi-seed paired comparison, and only the same scale of validation can settle this. ADR 0018 is marked Accepted with that caveat stated in the status block. 351 tests (347 pass, 4 skip), typecheck, boundaries and strategy gate green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…R 0018 §5)
Automatic rollback is gone. The model can now return
`{ "notes": "...", "revertTo": <version> }` and is given the version history --
each version's stated intent, the block it went in, and what the agent was worth
at the time -- so undoing a change is an informed choice rather than a guess.
The automatic rule had no defensible setting, which is why it was still listed as
undecided in the ADR while the code shipped one anyway. The previous
implementation's criterion was loose enough that it never fired in 18 runs. The
version this branch shipped was the opposite extreme -- revert on any loss at all
-- which in `crash` or `lending-incident`, where the whole field loses, reverts
every revision regardless of its quality, and which run noise alone flips
constantly. Whether a dip is the strategy or the market is a judgment, and a
judgment belongs in improve.md next to the cadence, for the same reason.
Nothing is lost on timing: the automatic check also ran at a revision
opportunity, so the model has exactly the same latency to act. What is genuinely
given up is the safety net -- a participant whose improve.md never reverts lets a
bad rewrite ride to the end of the run. That is a legitimate outcome in a
competition that measures how well you direct an LLM, and the frozen control
makes it visible. The ADR says so rather than leaving it implied.
The guards that are not judgment calls stay in the harness: cheatcode check,
compile failure, and the execution timeout still refuse a revision outright.
Reverting re-installs the old version as a *new* version rather than rewinding
the list, so the history stays a record of what actually ran and when. Asking for
both executorTs and revertTo is rejected instead of guessed at.
Verified on calm#101 over 150 blocks with claude-cli: the history reaches the
model, it installed a rewrite at block 774 after reading its own repeated
"cannot fund this side of the gap" noop reason, and declined at block 834 with
"v1 is doing exactly what it was written to do and it is winning". +57.7 against
the frozen control's +10.1.
352 tests, typecheck, boundaries and strategy gate green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ranking With #54 fixed there are finally enough agents that actually trade to ask the question. Eight strategies of clearly different quality (venue-arb, multi-arb, clean-arb, cross-venue-arb, adaptive-arb, arb-bot, max-profit-arb, and `random` as a known-bad) were held fixed as the measured field while the roster was padded to 9 / 18 / 27 / 36 at production R=360. The ranking holds. Spearman against the smallest roster: 1.000 / 0.929 / 0.905 / 1.000. What moves is the middle, where clean-arb, venue-arb and arb-bot sit within ±2 USDC of each other; the top, the bottom, and `random`'s last place never move. Load is fine too: 360 blocks in 718 s against a 720 s theoretical floor, no agent exited early. No breaking point was found at 36. So the earlier framing was wrong twice over. The first pilot's "opportunity dries up at 16 agents" was measuring cannibalisation between duplicate strategies, and the follow-up's "measurement did not hold" was measuring agents that could not trade at all. **The real constraint is neither load nor rank stability — it is that the spread between winners and losers shrinks as density rises.** clean-arb stops trading entirely from 18 agents on, because no spread clears the round trip any more. That changes what the cap means: it should be set where the gap between winners and losers still exceeds the run-to-run noise, not where the ranking breaks. That needs the noise measurement from §5, which is still open. Worth recording separately: `random` sits at roughly -1,100 at every density. Opportunity drying up does not stop a bad strategy from losing. What gets hard to separate as the field grows is adequate from good, not good from bad. Caveats stated in the ADR: the padding is duplicates of the core strategies, so cannibalisation is probably stronger here than in a field of genuinely distinct participants, and this is one regime on one seed. Also closes a Phase 2 loose end. my-arb was a prompt-only agent, so deleting the prompt.md left an empty directory that git dropped -- the participant-facing starter sample vanished, and the docs still used its name. It is back as agent.ts + improve.md: deliberately naive (fixed threshold, flat size, no fee awareness) so there is room to improve, with the funding check kept because omitting that does not make an agent naive, it makes it silently broken. And a diagnosability fix found while verifying it: a strategy that returns noop left nothing in its log at all, because send.ts drops noops before they are recorded. An empty agent log could not distinguish "never started" from "looked and declined" -- which cost time twice on this branch, once for clean-arb and once for the new sample. Declined decisions are now logged with their reason. 352 tests, typecheck, boundaries and strategy gate green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ADR treated splitting the field into heats as a cost multiplier -- run count times heat count -- and left it there. It is worse than that: §4 computes the z-score across the agents *within a scenario*, so co-location is what makes "who did better" mean anything. Split the field and each participant is normalized against a different set of opponents, so someone in a strong heat scores below an equally good player in a weak one and there is no basis for combining the two rankings. Handling heats therefore needs a mechanism the ADR does not have (a common reference agent in every heat to align the scales), not just more machine time. "Everyone in the same run" is a precondition of the scoring method, not an operational preference. Said plainly in §3, and in the participant-facing guide. B2 found no breaking point at 36 co-located agents, so this is likely moot at a realistic field size -- but it changes what happens if it is not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… is outlier-fragile Three corrections to §5, all of them things the earlier write-up got wrong by reasoning from the summary line instead of from the run. **"Opportunity dried up" was wrong.** At 36 co-located agents the fill counts are cross-venue-arb 240, multi-arb 185, max-profit-arb 62 — and clean-arb 0. The opportunities existed; the agents that took them lost money (-61, -25, -28). clean-arb's zero was it correctly judging that nothing cleared the round trip, and the others' losses are the evidence that it was right. What density raises is the share of opportunities that are not worth taking, not the absence of them. **The noise measurement did not need its own sweep.** Identical strategies in the same run differ only by execution-order luck, so the spread between duplicates *is* the noise. It was already in the B2 data: 7.44 USDC at 36 agents against a 65 USDC signal range, a ratio of 7-9x. The two-hour repeat sweep I was about to run would have measured something already measured. **What actually breaks at density is the top of the field, and one cause is the metric itself.** In z units the top four at 36 agents are all copies of the same strategy, separated by 0.0002-0.0029 against a noise floor of 0.041 — the ordering is luck. Being copies, that is expected and says nothing about whether distinct good strategies separate; the field has no two near-equal distinct strategies to test with. But the reason the gaps are that small is worth its own issue (#55): the sd used for normalization is 181.5 across the field and 20.91 with `random` excluded, so **one blown-up participant compresses everyone else 8.7x**. At hundreds of participants there will be several. Filed separately because it is a defect in how measurements are combined, fixable now and cheaply, as opposed to the open question of which metric to measure — and median/MAD would fix it while keeping the "won by how much" property that z was chosen for. Ordered by cost, the levers for top-of-field resolution are: robust normalization (#55), averaging over ~140 scenarios (already in the design), then raising flow intensity (needs environment recalibration). Flow was my first suggestion; it should be the last resort. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…atrix path The job still invoked `--regime calm-01 --blocks 12`. Renaming the regimes and making `--seed` required (ADR 0017 §1) broke it, and I missed it because the sweep I ran for stale references excluded `.github/`. The first re-run of the job failed earlier, in the deployer's Aave seeding, which this branch does not touch; that step passed on the retry, so it was a flake and the real failure was hiding behind it. While fixing the command, added coverage for the path the competition actually uses. `--scenarios` is a different code path from a single `--regime` run — it writes matrix.json / standings.json and has to keep going when one scenario fails — and nothing exercised it. Two scenarios is enough to run the loop, and the assertion checks that both scenarios produced results, that the standings ranked somebody, and that **both** metrics survive into matrix.json, since the scoring rule is expected to change and matrix.json is what makes a finished run re-scorable (#56). Verified locally against the state dump: both commands run and the assertion script passes on the real artifacts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns the simulator into something that can rank a field of participants, and changes how agents use an LLM. Two ADRs, both driven by measurements taken along the way — several of which contradicted the design they were meant to confirm.
ADR 0017 — scenario-based evaluation
The unit of evaluation is now a scenario = (regime, seed), written
<regime>#<seed>. Regimes carry no seed;--regimewithout--seedfails rather than silently scoring seed 1.npm run backtest -- --regime calm --seed 101 npm run backtest -- --scenarios config/scenarios/public.yaml # whole matrix + standings--scenariosreplays a regimes × seeds product on one anvil, snapshot/revert between scenarios, and writesmatrix.json(raw per-scenario per-agent scores, bothnetPnlUsdcandalphaUsdc) andstandings.json. The ranking is a derived view on purpose: the scoring rule is expected to change (see #56), and keeping the raw matrix means a new rule can be applied to a finished competition without re-running anything.Six regimes:
calm,cex-drift,informed-flow,whale,lending-incident,crash. Three needed new environment machinery:process.env, which a regime YAML cannot set. Added amarket.*config section and routed the coordinator throughSimConfig.flow.uninformedTrendCorrelation.--score-every Nthins the scoring cross-sections. Score-neutral (only the first and last reachsummary.json); it only coarsens the equity curve.ADR 0018 — the LLM rewrites the strategy instead of driving it
Prompt mode put the LLM in the trade path. Measured at production settings, the same strategy managed one decision every 8–28 blocks and 1/64 the actions of its rule-mode twin. It is removed.
A self-improving agent is
agent.ts+improve.md.decide()runs every block at rule speed; periodically the model is handed the current source plus how it has been doing and may return a replacement.improve.mdis not a renamedprompt.md— the old file answered "given this observation, what do you do", the new one answers "when, on what evidence, and how should the strategy change".Guards, each one because the deleted
src/llmtwo-layer machinery lacked it (it lost to frozen strategies on multi-seed validation, and its rollback never fired in 18 runs):revertTo.ERIS_AGENT_FROZEN=1gives every roster the frozen control the comparison needsVerified end to end: over 150 blocks the model read the strategy's own repeated "cannot fund this side of the gap" noop reason, rewrote it, then declined to touch it again because it had started working. +57.7 against the frozen control's +10.0 — one run on one seed, an existence proof that the loop works, not evidence that self-improvement wins.
Bugs found and fixed
netPnlUsdcvalued non-WETH bases at zero. The end-of-run summary passed the scalar WETH price tovalueUsdc, which marks any other base atp[sym] ?? 0, so an agent ending in WBTC had that inventory deleted from its PnL. Found because four agents reported an identical -6,686 to the cent while the reconstruction put them at +13 alpha. Same class as [sim] Scoring black hole: positions outside the registered market set are valued at exactly zero #41, on the scoring path [sim] Scoring black hole: positions outside the registered market set are valued at exactly zero #41 did not touch — and it mattered because the ranking usesnetPnlUsdc.obs.limits, never consulting the wallet, so under USDC-only funding they proposed selling WETH they did not hold and self-rejected forever: venue-arb 359/359 in calm. A rejected action scores exactly like choosing not to trade, so it was silent.example/agents/lib/affordable.tsnow caps by wallet and picks among fundable venues."uniswap"and"balancer"were pinned to one direction in every window. The old test passed because it only asserted two distinct values existed, which a perfect alternation satisfies.Measurements, including the ones that contradicted the plan
Co-location holds to 36 agents. Eight strategies of known, different quality held fixed while the roster grew to 9/18/27/36 at R=360. Spearman against the smallest roster: 1.000 / 0.929 / 0.905 / 1.000. 360 blocks in 718 s against a 720 s floor, no agent exited early. No breaking point found.
Three earlier readings were wrong and are corrected in the ADR:
The noise measurement did not need its own sweep. Identical strategies in one run differ only by execution-order luck, so the spread between duplicates is the noise: 7.44 USDC at 36 agents against a 65 USDC signal range. The two-hour repeat sweep I was about to run would have measured something already measured.
What does break at density is the top of the field, and one cause is the metric: the sd used for normalization is 181.5 across the field and 20.9 excluding the one deliberately-bad agent, so a single blown-up participant compresses everyone else 8.7×. Filed as #55.
Known gaps
crashis the price gap only; the liquidity-withdrawal half is [sim] Liquidity-pull stress event: withdraw pool depth during crash windows #52.depegis unimplemented. Its real blocker turned out not to be the numeraire debate but thatvalueUsdcaddsusdcUnitsat face value — a stable is worth exactly 1.0 no matter what.evm_snapshot/evm_revert, which a real chain does not have ([sim] Run the environment on a real L2 (cheatcode-free mode) #33). The ADR now says so.Verification
352 tests (348 pass, 4 skip),
typecheck,check:boundaries,check:strategygreen. Every regime and both agent modes exercised on-chain against the distributed state dump.Closes #53, closes #54.
🤖 Generated with Claude Code