Benchmarks delegation

Benchmarks: delegation and expertise

The delegation half of the deliberation benchmark: what happens when a room's members are not interchangeable — one of them holds the fact that decides the question, and the others do not.

The short version, simulated and live: the fact reaches the floor when members deposit before they argue, and reaching it is not what decides the answer. Across twenty-seven live rounds the fact-holder spoke before the commit in every room that had one and fourteen of those twenty-three rooms were still wrong (subject to a caveat on that metric — see the delegation page below), no turn was ever awarded on BidReason::Knows, and !defer was used on none of 266 turns.

It answers three questions, written down before any of these numbers existed.

• Q1 — does the floor reach the member who holds the fact? fact % is the share of episodes in which the decisive member deposited a topiced !evidence line before the commit boundary, and to-fact is the mean turn index at which it did. A deposit landing at or after the boundary is compute the room paid for and could not use, and is scored as a miss.
• Q2 — does an informative router beat a uniform ladder? route %, over the two ladder arms.
• Q3 — what does accuracy cost, once seats have prices? cost/ep and correct/kU, right answers per thousand units, under --cost-tiers.

A fourth number, rho, is an obligation rather than a question: the spec requires this benchmark to publish the rank correlation between a member's directory weight and its share of the episode's turns, and to report the mechanism as having failed if it merely tracks who talked.

The flags

flag	what it does
--specialists N	N members read one topic each far more tightly than everybody else, and everybody else's read of that topic widens to match. Information is redistributed, not created.
--hidden-profile	One decoy is planted above every member's own argmax except one member's, and that member alone holds the fact that rules it out.
--blind-evidence	A member's first turn, while the room is blind, is a deposit rather than a position. Off by default.
--defer-cap N	Turns a member may spend saying a topic is not theirs.
--cost-tiers	A specialist's turn costs ten units against a lay member's one.
--history N	Prior episodes of hive+ from which ladder+dir earns its directory.

The arms

hive+dir is the tuned policy with the directory on, so BidReason::Knows is reachable. hive+defer adds defer_cap with nothing routing the vacated turn, and hive+dir+defer is both. ladder+dir is the responder ladder with each candidate's earned directory lines rendered into its description and the contested topic named in the request — the routing rule a subagent description and a role string implement — validated through the real accept_selection. hive+cost and all-reasoning appear under --cost-tiers only.

The uniform room, which predicted zero

5000 rooms, ±90 noise. With homogeneous expertise there is nothing to route on, and the spec predicted exactly that.

                      ordinary opening        --blind-evidence
5000 rooms          correct %      95% CI   correct %      95% CI  knows %
ladder                   57.6   56.2–59.0        57.6   56.2–59.0        —
vote                     78.5   77.4–79.6        78.5   77.4–79.6        —
hive+                    82.1   81.0–83.1        75.3   74.1–76.5      0.0
hive+dir                 82.1   81.0–83.1        75.7   74.5–76.8     75.1
hive+defer               82.1   81.0–83.1        75.3   74.1–76.5      0.0
hive+dir+defer           82.1   81.0–83.1        75.7   74.5–76.8     75.1
ladder+dir               49.5   48.1–50.9        98.8   98.5–99.1      0.0

Under the ordinary opening the delegation arms are hive+ to the digit and knows % is 0.0: BidReason::Knows never fires. It needs a member that is the directory's top holder of the contested topic and has taken no position on it, and where every member opens with a !propose those never hold together. ladder+dir is eight points worse than the uninformed draw over the same five candidates.

Two specialists

                      ordinary opening        --blind-evidence
5000 rooms          correct %      95% CI   correct %      95% CI  knows %  route %
ladder                   52.6   51.2–54.0        52.6   51.2–54.0        —     18.9
vote                     71.1   69.9–72.4        71.1   69.9–72.4        —        —
hive+                    74.2   72.9–75.4        67.6   66.3–68.9      0.0        —
hive+dir                 74.2   72.9–75.4        68.0   66.7–69.2     78.3        —
hive+defer               74.3   73.1–75.5        67.2   65.9–68.5      0.0        —
hive+dir+defer           74.3   73.1–75.5        68.1   66.8–69.4     79.8        —
ladder+dir               45.1   43.8–46.5        84.4   83.4–85.4      0.0     22.3

ladder+dir routes to the decisive member more often than ladder does — 22.3% against 18.9% — and is seven and a half points less accurate. Directory weight on a topic is earned by grounding it, so the heaviest holder is the member who argued it hardest rather than the one who reads it best. Routing precisely to the wrong criterion is worse than not routing.

Under --blind-evidence the specialist deposits before the commit boundary in 93.4% of episodes, at a mean turn index of 2.0.

The price of a seat

The same rooms under --cost-tiers, with the evidence-first opening.

arm              correct %   cost/ep    correct/kU
vote                  71.1     69.00         10.31
ladder                52.6      4.56        115.39
ladder+dir            84.4      3.24        260.33
hive+                 67.6     52.40         12.91
hive+cost             68.1     52.62         12.95
all-reasoning         67.6    115.27          5.87

hive+cost buys 12.95 right answers per thousand units against all-reasoning's 5.87, and the whole of that gap is that all-reasoning puts every seat on the ten-unit tier for the same 67.6%. Nothing about the delegation mechanism produced it; not spending ten units on seats that do not need them did.

The hidden profile

5000 rooms, ±50 noise. This is the shape the mechanism was built for.

                      ordinary opening                --blind-evidence
5000 rooms          correct %   fact %    rho   correct %      95% CI   fact %  knows %    rho
ladder                   35.1        —      —        35.1   33.7–36.4        —        —      —
vote                     15.0        —      —        15.0   14.1–16.1        —        —      —
hive+                    15.3      1.5   0.17        66.3   65.0–67.6     96.8      0.0   0.32
hive+dir                 15.3      1.5   0.17        65.8   64.5–67.1     95.7     77.5   0.37
hive+defer               15.4      1.4   0.17        66.8   65.4–68.1     96.8      0.0  -0.02
hive+dir+defer           15.4      1.4   0.17        66.6   65.3–67.9     95.7     77.3  -0.00
hive+ref                 15.3      1.8   0.32        53.3   51.9–54.7     96.4      0.0   0.31
hive+ev                  12.7      7.1   0.33        26.0   24.8–27.2     96.5      0.0   0.48
ladder+dir               34.6        —      —        64.1   62.8–65.4        —      0.0      —

Under the ordinary opening no deliberating arm solves it, and fact % says why: the deciding fact reaches the floor in time in 1.5% of episodes. A !propose counts as a supporter, so four lay members who privately favour the planted decoy carry it inside the blind round; the room is in Phase::Commit before the fact-holder ever sees a floor to deposit against.

With the evidence-first opening fact % goes to 96.8% and the answer goes from 15% to 66%. hive+ − vote is +51.3 [+49.8, +52.8], and it reproduces at three seeds. That is a finding about when a member speaks, not about any mechanism in the library: same rooms, same evaluations, same policy, same fold, and a participant policy that states what it knows before what it wants.

On this shape hive+dir loses — 65.8% against hive+'s 66.3% — with Knows winning the floor in 77.5% of episodes. The directory does not beat the policy without it anywhere: +0.0 on the uniform room, +0.4 under the evidence-first opening, +0.0 and +0.4 with two specialists, −0.5 here. !defer is neutral by the same measure, moving ±0.5 and never leaving the interval.

Two circularity notes

ladder+dir under --blind-evidence is an artifact. The arm tells its router which topic the call turns on, and in this benchmark that topic is the correct option; with an evidence-first opening the directory records who deposited a reading of it, and a member who deposited on the truth usually favours the truth. The 92-point swing from the same arm's 49.5% under the ordinary opening is the size of the leak, not of any mechanism. The arm is kept, unchanged and labelled, because deleting an arm that started scoring well would be worse than explaining why its score is not evidence.

rho is the number the mechanism is judged by, and it separates the arms. It is Pearson's correlation on tie-averaged ranks, computed in exact integer arithmetic. Every deliberating arm of the uniform bench under --blind-evidence sits at 0.61–0.72, and under the ordinary opening the same bench, which has nothing to route on, sits much lower — hive+ at 0.19. On the hidden profile under the ordinary opening hive reads 0.08, and under the evidence-first opening it rises to 0.32 before !defer pulls it to -0.02, -0.08 at --defer-cap 2. Depositing before arguing earns specialisation without earning a position; deferring zeroes a member's own weight on a topic. Neither of those is the directory, and even at the uniform bench's 0.61–0.72 the directory's weight tracks speech share only weakly, so very little of any result on the default bench could be credited to the estimate having found something.

And history makes it worse. Over 2000 rooms with two specialists, ladder+dir scores 52.1% at --history 0 — the null control, where every description is None — then 44.8, 44.0, 43.9 and 43.6 at one, three, five and ten prior episodes. It is not undertrained; it converges, slowly, on the wrong member.

Live rooms

scenarios/index-lock-expert.txt is the hidden profile written for a room with a named specialist, and scenarios/index-lock-tiers.txt is the same scenario with a tier: on every seat so the routing can be priced. The live matrix ran ten rows, twenty-seven rounds and two hundred and sixty-six agent turns against a single model family.

row	backend	rounds	hive ✓/decided	poll ✓	turns/ep	s/ep	cost/ep	expert
checkout-503	HTTP flash	3	3 / 3	0 / 3	7.3	451	24.7	—
index-lock-expert	HTTP flash	3	1 / 2	0 / 3	11.7	842	46.0	3 / 2
index-lock-expert	HTTP, --specialist-model reasoning	3	0 / 3	0 / 3	7.0	322	197.7	3 / 1
index-lock-expert	claude -p --model flash	3	3 / 3	0 / 3	13.3	697	—	3 / 2
index-lock-expert	opencode run -m ladder/flash	3	1 / 2	0 / 3	12.3	374	—	3 / 1
index-lock-expert	codex exec → deepseek/deepseek-v4-flash	3	0 / 2	0 / 3	12.3	305	—	3 / 0
index-lock-tiers	HTTP, --specialist-model reasoning	3	0 / 3	0 / 3	6.0	318	166.0	3 / 1
index-lock-tiers	HTTP, every seat reasoning	2	0 / 2	0 / 2	7.5	489	207.0	2 / 1
checkout-503-federated	HTTP flash, --swarm	1	0 / 1	0 / 1	15	764	28.0	—
index-lock-tiers	HTTP, four flash seats + dba on reasoning	3	1 / 3	0 / 3	8.7	363	108.7	3 / 3

expert reports two independent counts, not a ratio: the rounds in which the scenario's truth_expert spoke before the commit, and (separately) the rounds whose winning commit chain reaches something it said. codex exec is a CLI row by necessity: the router cannot relay a streaming Responses request, so it cannot go through the harness's own HTTP backend.

A caveat on expert's first number: this follow-up's print_expert fix changed what "before" means — it now compares a turn against the round's actual first commit-phase turn, gated on the round having decided, rather than approximating the boundary as the episode's last turn regardless of outcome. Every row above except the freshly-measured mixed-tier row was recorded under the old approximation and has not been reprocessed against the corrected definition; the two can disagree for any row, not only one with an undecided round, and the raw per-round turn indices needed to settle it were not retained.

The poll never found the answer in any row — 0 of 27 rounds, wrong or tied every time.

The fact-holder spoke before the commit in all 23 rounds that had one, at turn 4 in eighteen of the first twenty (subject to the caveat above), and fourteen of those twenty-three rooms still got it wrong. Reaching the holder is not the bottleneck; weighing what it says is.

!defer was used on none of the 266 turns, in any harness, although it sits in the move list every participant reads. The federated runs showed a move outside the list is never used; this is narrower and less comfortable — a move inside it is not necessarily used either.

The directory rendered into every deliberation prompt and never won a turn. No turn in any round was awarded on BidReason::Knows; the recorded reasons are salience, dissent, addressed and quiet, for the structural reason the simulation gives — by the time a topic is contested, its top holder has taken a position on it.

A reasoning model on the expert seat did not help and the rooms lost faster, converging on the decoy in six turns because three of the five blind opening turns were the same !propose, which is already quorum. dba never stated its numbers in those rooms: it spent its blind turn arguing that the two batch sizes have not moved in a year and are therefore not the cause.

The cost columns compared flash-only rooms against all-reasoning rooms, not mixed tiers — a harness defect, now fixed. is_specialist was true for any seat with an expert_on line and every seat in these scenarios has one, so --specialist-model put the whole room on the expensive model. On that comparison the rows with a reasoning model cost 166–207 units an episode and scored 0 correct in 8 rounds, against 25–46 for the flash rows. Seating by tier fixed it: the corrected row above, with only dba on reasoning, scored 1 correct in 3 rounds at 108.7 units — better than either buggy all-reasoning row, still costlier than flash-only without clearly beating it on accuracy.

One harness scored 3 of 3 where the same model behind it lost elsewhere. claude -p --model flash and the direct HTTP rows sit behind the same router and the same model; the claude rooms simply ran longer — 13.3 turns against 7.0 — and spent the extra turns killing the decoy with !object lines before settling anything, then carried the answer on a citation chain. They did not perform the scenario's intended arithmetic; the one room that visibly did was an opencode room that then exhausted without committing. With three rounds a row this is a description of transcripts, not a ranking of harnesses.

The federated round decided #retries, wrong. Two messages crossed, nothing was stranded, and what crossed was accurate: Release confirmed the retry path shipped disabled and fired zero retries. One desk read that and supported #scale, another read it and committed #retries.

One model family, three rounds a row, one scenario family, and a single federated episode. Nothing here is a rate.

Why both knobs ship off

EpisodePolicy::DEFAULT carries directory: None and defer_cap: None. The mechanism costs nothing on the uniform bench and buys nothing on the hidden profile, which is a weaker case for defaulting it on rather than a stronger one. --blind-evidence is off too, and for a sharper reason: it costs about seven points on an ordinary room. It is a participant policy, not a library setting.

Reproducing

cargo run --release -p tinyhivemind-hive --example bench -- --episodes 5000
cargo run --release -p tinyhivemind-hive --example bench -- --episodes 5000 --blind-evidence
cargo run --release -p tinyhivemind-hive --example bench -- --episodes 5000 --specialists 2
cargo run --release -p tinyhivemind-hive --example bench -- --episodes 5000 \
  --specialists 2 --blind-evidence --cost-tiers
cargo run --release -p tinyhivemind-hive --example bench -- --episodes 5000 --hidden-profile
cargo run --release -p tinyhivemind-hive --example bench -- --episodes 5000 \
  --hidden-profile --blind-evidence

The full write-up, every table and what the simulation does not show are in docs/experiments/2026-09-05-expert-delegation.md, and the harness's own notes are in DELEGATION.md.