diff --git a/AGENTS.md b/AGENTS.md index 988925e..e14b726 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -114,7 +114,7 @@ tests/compat/run-matrix.sh --tier blocking # whole tier ### srt-sink extended metrics -`tests/compat/srt-sink/` is the mock SRT receiver used by the compat harness. Beyond the original 4 frozen keys (`bytes_received`, `disconnects`, `handshake_ms`, `error`), `srt-sink` now emits 6 additional keys in `result.json`: +`tests/compat/srt-sink/` is the mock SRT receiver used by the compat harness. Beyond the original 4 frozen keys (`bytes_received`, `disconnects`, `handshake_ms`, `error`), `srt-sink` now emits 9 additional keys in `result.json`: | Key | Source | Description | |-----|--------|-------------| @@ -124,6 +124,9 @@ tests/compat/run-matrix.sh --tier blocking # whole tier | `pkt_rcv_loss` | `srt_bstats` `pktRcvLossTotal` | SRT-level receive loss (cumulative, summed across reconnects) | | `pkt_rcv_drop` | `srt_bstats` `pktRcvDropTotal` | SRT-level receive drop (too-late packets) | | `pkt_retrans` | `srt_bstats` `pktRetransTotal` | SRT-level retransmissions | +| `nakreport_readback` | `srt_getsockflag` `SRTO_NAKREPORT` | NAK-report policy **negotiated** on the accepted socket (`-1` = unreadable) | +| `lossmaxttl_readback` | `srt_getsockflag` `SRTO_LOSSMAXTTL` | reorder-tolerance ceiling negotiated on the accepted socket (`-1` = unreadable) | +| `reorderfreeze_readback` | `srt_getsockflag` opt id `120` | decay-freeze flag negotiated on the accepted socket (`-1` = unreadable / stock libsrt) | The TS parser lives in `srt-sink/ts_continuity.h` (header-only, dependency-free). It reassembles 188-byte packets across `srt_recv` boundaries, tracks per-PID continuity counters, and excludes null PID `0x1FFF` and adaptation-only packets from CC checks. Unit tests: `srt-sink/ts_continuity_test.cpp` (registered as ctest `ts-continuity` in `srt-sink/CMakeLists.txt`). @@ -133,6 +136,8 @@ The TS parser lives in `srt-sink/ts_continuity.h` (header-only, dependency-free) **`--reorderfreeze 0|1`** — new `srt-sink` flag. Sets `SRTO_REORDERFREEZE` via the raw numeric opt id `(SRT_SOCKOPT)120` so it compiles against any libsrt version. Reports `reorderfreeze=on|off|unsupported` in the startup banner. +**Sockopt read-back (`nakreport_readback` / `lossmaxttl_readback` / `reorderfreeze_readback`).** After `srt_accept`, `srt-sink` reads the three policy options back off the **accepted** socket via `srt_getsockflag` (`SRTO_NAKREPORT`, `SRTO_LOSSMAXTTL`, opt id `120`) and writes them to `result.json` — the **negotiated** values, not the requested ones the banner echoes. This is what lets a campaign measured via `srt-sink` flags prove it reproduces irl-srt-server's `kSrtProfileTable` (`SLSSrt.cpp`) L1/L2 tuples. The fidelity assertion `tests/compat/lib/srt-sink-proxy-fidelity.sh` runs the L1 `{freeze=1,nak=1,ttl=40}` and L2 `{freeze=1,nak=0,ttl=40}` flag sets, asserts the read-backs equal those tuples, and runs a falsifier (`--lossmaxttl 30` must FAIL the L2 ttl=40 check, proving the value is read back not echoed). Evidence: `test-results/srt-sink-proxy-fidelity.json` (gitignored). + ### Profile validation A/B matrix `tests/compat/scenarios/profile-validation-matrix.sh` is the A/B orchestrator for the four non-FEC receive profiles. It runs paired alternating reps (baseline patched libsrt vs freeze profile) under netem reorder stress and gates on six quality clauses: @@ -148,9 +153,31 @@ Registered in `matrix.yaml` as scenario `profile-validation-matrix` (tier: block The `reorder-stress.sh` scenario is parameterized (BITRATE_KBPS, RX_LATENCY_MS, NAKREPORT, LOSSMAXTTL, REORDERFREEZE, PROFILE_LABEL, NETEM_SEED) plus the gain-hunt adverse-config axes (STEADY_LOSS_PCT, BURST_LOSS_PCT, RTT_SPREAD_MS — see Gain-hunt scaffold below) and emits TS-continuity + SRT counters + `goodput_bps` + `wire_amp` into `result.json`. Default run (all axes unset) is byte-identical to the pre-matrix behavior (Rule E). -### Gain-hunt scaffold (FEC-mixture R&D) +**Reverse-channel metric (`reverse_wire_bytes` / `reverse_wire_amp`).** `setup_topology` now installs a countable `prio` root qdisc on `$PEERIF` **inside** the receiver netns (the veth peer defaults to `noqueue`, so the receiver→sender egress had no Sent counter). After phase ii the scenario reads that qdisc's Sent bytes and emits `metrics.reverse_wire_bytes` plus `metrics.reverse_wire_amp` (`reverse_wire_bytes / bytes_received`). This makes the periodic-NAK reverse cost visible so a recipe cannot false-promote on forward `wire_amp` alone (B3/O1). Additive keys; existing callers unaffected (Rule E). **Empirical finding:** in this SRTLA topology the reverse channel is dominated by per-packet broadcast ACKs, so NAK-**off** (which retransmits more without precise NAKs) typically costs **more** reverse than NAK-**on** — the metric's value is visibility, not a fixed direction. + +**Sender seam (`SRTLA_SEND_RS_BIN` / `REQUIRE_RS_SENDER`).** The campaign's PRIMARY sender is the Rust fork `srtla-send-rs` (ADR-003), CLI-identical to the C `srtla_send`. When `SRTLA_SEND_RS_BIN` resolves (env, or a `srtla_send_rs` on PATH) it replaces the C sender; `result.json` records `config.sender_kind` (`c`|`rust`) and `config.sender_bin`. `REQUIRE_RS_SENDER=1` makes a missing fork a clean SKIP (exit 77) rather than silently measuring the deprecated C sender as production. Default (both unset) keeps the C sender, so existing callers (`profile-validation-matrix.sh`) are unaffected (Rule E). + +**FEC caller arm — `CALLER_PACKETFILTER`.** For the FEC arms of the gain hunt, `reorder-stress.sh` accepts `CALLER_PACKETFILTER` (an SRT FEC packet-filter config, must match `^fec,`). When set, the SRT caller switches from ffmpeg-direct to ffmpeg (MPEG-TS generator) piped into `srt-live-transmit` carrying `&packetfilter=` — ffmpeg's libsrt wrapper has a fixed option allow-list with NO `packetfilter` (appending it hard-fails `Option not found`), while `srt-live-transmit` (libsrt 1.5.5) accepts it (same caller as `fec-connect-matrix.sh`). Pair with a FEC-accepting sink via `SINK_EXTRA_ARGS="--packetfilter fec"`; the negotiated filter lands in `result.json` as `sink.negotiated_packetfilter` and the requested filter as `config.caller_packetfilter`. Requires `srt-live-transmit` on PATH — absent ⇒ SKIP (exit 77). Pure FEC (`arq:never`) is refused; FEC is always an `arq:onreq` hybrid. Empty default = today's ffmpeg-direct caller, byte-identical (Rule E). + +### Gain-hunt orchestrator (FEC×NAK×FREEZE) + +`tests/compat/scenarios/gain-hunt-matrix.sh` is the ORCHESTRATOR for the receiver gain hunt — the campaign that decides whether any receiver recipe earns a place in the operator-facing receiver-capability catalog (which ships EMPTY). It fixes the pre-registered "real gain + no regression" decision rule in code (now including a `reverse_wire_amp ≤ 1.10×` guardrail), and enumerates a **3-axis candidate matrix: REORDERFREEZE × NAKREPORT × FEC** (2×2×2 = 8 tuples; the baseline tuple `freeze=1,nak=0,fec=off` = Classic L2 + latency is excluded → **7 candidates**). `LOSSMAXTTL=40` is held constant. FEC is **always** the `arq:onreq` hybrid (`fec,cols:16,rows:1,layout:even,arq:onreq`); pure FEC `arq:never` is BANNED and **REFUSED (exit 2)** in every mode (overridable spec via `GAIN_FEC_FILTER`, validated by `assert_no_arq_never`). + +Modes: +- bare → notice + matrix summary (exit 0) +- `--dry-run` → prints the full 3-axis matrix with each cell's SRTO tuple (exit 0) +- `--smoke` → runs the falsifiability control FIRST, then ONE paired cell (candidate `f1-n1-plain` NAK-on vs baseline `f1-n0-plain`) bounded, writing per-rep `result.json`; a control that PASSES SKIPs (exit 77, instrument not falsifiable); needs CAP_NET_ADMIN **and** a resolvable `srtla-send-rs` (else SKIP exit 77) +- `--stage screen|deep` → the **two-stage** campaign (T-A6). `screen` sweeps all 7 candidates × a reduced adverse grid (`STEADY_LOSS_PCT ∈ {3,7}`, `BURST_LOSS_PCT ∈ {0,20}`) at `SCREEN_REPS=4` and emits `survivors.json` (a combo survives on a directional gain — `goodput ≥ 1.03×` OR `late-drop ≤ 0.80×` — in ≥1 cell with no hard-gate failure: `disconnects==0`, `ts_sync==0`). `deep` runs the **deep set = survivors ∪ top-K(2)/family ∪ the high-loss SENTINEL cells (`STEADY=7,BURST=20` per candidate, ALWAYS)** at `DEEP_REPS=10`, then runs `--analyze` across **every** deep cell and writes `verdict.json` (`promoted:[…]` or `NULL`). The sentinels are deep-tested even when the screen rejected them — the **anti-false-NULL rescue** (Oracle O4): a NULL is recorded only when the FULL deep set, sentinels included, shows no promotable candidate. Each stage runs a `PORT_MISMATCH=1` falsifiability control first and ABORTS (exit 2) if it passes. `--stage --plan` prints the cell set with no privilege (the deep plan lists the sentinel cells for every family, incl. screen-rejected ones). Evidence lands under `test-results/gain-hunt/` (Rule D — gitignored, inside the repo) +- `--analyze

` → applies the pre-registered §2 decision-rule statistics to ALREADY-MEASURED paired evidence `

` and emits a verdict JSON to stdout. `

` is either a self-contained fixture JSON (`{candidate_id, cells:{:{candidate:[reps],baseline:[reps]}}}`) or a directory of `/{candidate,baseline}/rep-*.json` (the `run_cell` layout). It computes the **exact Mann-Whitney U** (pure-stdlib subset-sum permutation DP, two-sided `p = 2·P(U ≤ min(U, mn−U))`, tie-aware midranks; **no scipy** — it is absent on the box, only numpy + stdlib) for n ≤ 20, falling back to the tie-corrected normal approximation for larger n. It then applies **Holm-Bonferroni across every cell** in the supplied set (not just survivors), checks all seven no-regression guardrails per cell (`disconnects==0`, `ts_sync==0`, `ts_cc ≤ B`, `goodput ≥ 0.99×B`, `wire_amp ≤ 1.10×B`, `reverse_wire_amp ≤ 1.10×B`, `p95 pkt_rcv_drop ≤ B`), and promotes **only** with a Holm-significant real gain in ≥1 cell AND no regression in EVERY cell. Exit 0 = promoted, 1 = not promoted, 2 = no usable evidence. This COMPUTES a verdict over supplied evidence — it does not RUN the campaign (no CAP_NET_ADMIN, no sender needed). +- `--help` → header (exit 0); `--claim-gain` → REFUSED (exit 3), the falsifiability anchor — a gain cannot be claimed by running this script. + +The PRIMARY sender is `srtla-send-rs` (resolved via `SRTLA_SEND_RS_BIN` or a `srtla_send_rs` on PATH); run modes pass `REQUIRE_RS_SENDER=1` to `reorder-stress.sh` so a missing fork SKIPs (exit 77) rather than measuring the C sender as production. Registered in `matrix.yaml` as `tier: informational` (NON-blocking; not run by `run-matrix.sh --tier`, which iterates pairs, not scenarios). Full protocol: [`docs/GAIN-HUNT-PROTOCOL.md`](docs/GAIN-HUNT-PROTOCOL.md). The adverse axes (STEADY_LOSS_PCT / BURST_LOSS_PCT / RTT_SPREAD_MS) remain additive on `reorder-stress.sh`; T-A6 wired the two-stage **structure** (screen→deep + sentinel rescue + falsifiability + verdict), and Wave B **already ran** it under CAP_NET_ADMIN, producing a **NULL** verdict (`test-results/gain-hunt/verdict.json`). Falsifiability self-test hooks (`GAIN_TEST_CONTROL_PASS` / alias `PORT_MISMATCH_PASS_OVERRIDE`, `GAIN_TEST_CONTROL_FAIL`) inject a synthetic control so the abort gate is verifiable without privilege. + +FEC is **always** the `arq:onreq` hybrid at one pre-registered geometry — `fec,cols:16,rows:1,layout:even,arq:onreq` (column-only parity, ~6% overhead). The `cols≥16` floor is pre-registered (Oracle O5) so the forward FEC overhead (`1/cols`) stays clear of the `wire_amp ≤ 1.10×` budget *by construction*: `cols:16` ⇒ 6.25% (clear headroom), `cols:10` ⇒ 10% (on the cliff), `cols:8` ⇒ 12.5% (over budget). The campaign's NULL verdict is **first-class**: a NULL is recorded only when the FULL deep set — sentinels included — shows no promotable candidate under the §2 rule, with the stage's `PORT_MISMATCH` falsifiability control having FAILED first. Full protocol §3/§6: [`docs/GAIN-HUNT-PROTOCOL.md`](docs/GAIN-HUNT-PROTOCOL.md). + +**Golden-fixture stats test.** `tests/compat/scenarios/gain-hunt-analyze-test.sh` pins the `--analyze` decision-rule engine against committed golden fixtures in `tests/compat/fixtures/gain-hunt-golden/` (`gain` → promoted; `regression` → rejected on the `disconnects` hard gate; `reverse-spam` → rejected on `reverse_wire_amp`; `tie` → `winner: none`). `expected.json` carries the by-hand exact statistic — when all 10 candidate goodput samples beat all 10 baseline samples, `U = m·n = 100`, `U_min = 0`, and `p = 2 / C(20,10) = 2/184756 ≈ 1.0825×10⁻⁵`; Holm over 2 such cells gives `≈ 2.165×10⁻⁵`. The test is pure-stdlib (no netem, no sender, no scipy) so it runs anywhere `python3` does. -`tests/compat/scenarios/gain-hunt-matrix.sh` is the ORCHESTRATOR STUB for the FEC-mixture gain hunt — the campaign that decides whether any FEC packet-filter mixture earns a place in the operator-facing receiver-capability catalog (which ships EMPTY). It is a scaffold: it fixes the pre-registered "real gain + no regression" decision rule in code, enumerates the candidate-mixture matrix (FEC **always** `arq:onreq` hybrid; pure FEC `arq:never` is BANNED — present only as a control row), and lists the adverse-config axes it drives `reorder-stress.sh` across (STEADY_LOSS_PCT / BURST_LOSS_PCT / RTT_SPREAD_MS). It does **not** run the campaign (R&D track). Modes: bare / `--dry-run` / `--help` exit 0; `--claim-gain` is REFUSED (exit 3) — falsifiable, a gain cannot be claimed without measured evidence. Registered in `matrix.yaml` as `tier: informational` (NON-blocking; not run by `run-matrix.sh --tier`, which iterates pairs, not scenarios). Full protocol: [`docs/GAIN-HUNT-PROTOCOL.md`](docs/GAIN-HUNT-PROTOCOL.md). The adverse axes are additive on `reorder-stress.sh`: STEADY_LOSS_PCT (`netem loss %` on both links), BURST_LOSS_PCT (loss correlation % with steady, else `loss gemodel`), RTT_SPREAD_MS (extra slow-link delay; slow = 150 + spread). +**Geometry / wire-amp lint.** `tests/compat/scenarios/gain-hunt-geometry-lint.sh` is the executable form of the §3 geometry constraint: it reads the orchestrator's active `GAIN_FEC_FILTER` default, asserts its geometry is promotable (`cols≥16` AND `1/cols < 0.10`), and runs a discrimination self-test table (`cols:16` PASS, `cols:10`/`cols:8` REJECT) proving the budget check actually discriminates. A regression that narrows the FEC geometry to `cols:8` (~12.5% > the `wire_amp ≤1.10×` budget) trips this lint in CI rather than in a privileged campaign run. Pure stdlib (awk) — no netem, no sender, no scipy. ### FEC connect-matrix @@ -292,11 +319,13 @@ pre-registered tie-break rule resolves to 40 (BELABOX parity / max compat). This is now locked into `irl-srt-server` L1 and L2 profiles (Task 4). Evidence: `test-results/srt-receive-profiles/lossmaxttl-3040.json`. -**Gain-hunt scaffold (Task 2, pending):** `tests/compat/scenarios/gain-hunt-matrix.sh` -will be an orchestrator stub documenting the pre-registered adverse-config A/B protocol -and candidate-mixture matrix. Protocol doc: `docs/GAIN-HUNT-PROTOCOL.md`. The full -campaign is an R&D track; the scaffold runs `--help`/dry-run only and does not run the -full campaign. Registered in `matrix.yaml` as informational (non-blocking). +**Gain-hunt orchestrator (Task 2, DONE):** `tests/compat/scenarios/gain-hunt-matrix.sh` +is the fully-wired two-stage screen→deep orchestrator for the pre-registered +adverse-config A/B protocol and candidate-mixture matrix. The campaign RAN under +CAP_NET_ADMIN with the `srtla-send-rs` sender and produced a **NULL** verdict (no +recipe cleared the gate; the catalog stays empty). Protocol doc: `docs/GAIN-HUNT-PROTOCOL.md`; +full mechanics and current status are in the "Gain-hunt orchestrator (FEC×NAK×FREEZE)" +section above. Registered in `matrix.yaml` as informational (non-blocking). **FEC policy:** FEC is always `arq:onreq` hybrid; pure FEC (`arq:never`) is BANNED. The mixture catalog is EMPTY until the gain-hunt evidence gate passes. diff --git a/docs/GAIN-HUNT-PROTOCOL.md b/docs/GAIN-HUNT-PROTOCOL.md index 755f4c7..74beeec 100644 --- a/docs/GAIN-HUNT-PROTOCOL.md +++ b/docs/GAIN-HUNT-PROTOCOL.md @@ -13,13 +13,39 @@ tries to pass it. - **Decision record:** [`../../docs/RECEIVER-RECONCILIATION.md`](../../docs/RECEIVER-RECONCILIATION.md) §"Gain-Hunt Protocol" and the L1–L3 locked decisions. - **ADR cross-ref:** [`adr/ADR-002-srt-patch-necessity.md`](adr/ADR-002-srt-patch-necessity.md) — "C is SAFE": stock libsrt + `nakreport=0` + `lossmaxttl` is a proven baseline equivalent. The gain hunt borrows ADR-002's pre-registration discipline. -- **Harness scaffold:** [`../tests/compat/scenarios/gain-hunt-matrix.sh`](../tests/compat/scenarios/gain-hunt-matrix.sh) — orchestrator **stub** (this effort). It documents the rule and the matrix and **does not run the campaign** (R&D track). +- **Harness orchestrator:** [`../tests/compat/scenarios/gain-hunt-matrix.sh`](../tests/compat/scenarios/gain-hunt-matrix.sh) — the fully-wired two-stage screen→deep orchestrator (T-A6). It encodes the decision rule and the candidate matrix **and has run the campaign** — `--stage deep` collected the evidence and its `--analyze` stats layer wrote the verdict (**NULL**; see Status below). - **Measurement instrument:** [`../tests/compat/scenarios/reorder-stress.sh`](../tests/compat/scenarios/reorder-stress.sh) — the same A/B instrument the profile-validation matrix uses, now extended with the adverse-config axes below. -> **Status:** SCAFFOLD. The decision rule and candidate matrix are fixed here and -> in the stub; the full campaign (build/pin each FEC geometry, run the reps, do the -> stats) is an explicit R&D track and is **not** wired. See "Running the Full -> Campaign" for what implementing it entails. +> **Status:** RAN — verdict **NULL** (catalog stays empty). The two-stage +> screen→deep campaign was executed under `CAP_NET_ADMIN` with the `srtla-send-rs` +> sender; the deep stage's built-in `--analyze` wrote the verdict. Evidence: +> `test-results/gain-hunt/verdict.json` (`{"verdict":"NULL","n_cells":20,"promoted":[],"reason":"regression_in_>=1_cell"}`; +> gitignored evidence dir, independently re-derivable via +> `tests/compat/scenarios/gain-hunt-matrix.sh --analyze test-results/gain-hunt/deep`). +> **Reason:** no candidate showed BOTH a Holm-significant real gain AND no regression +> across the full deep set (20 cells: 0 screen-survivors, so the anti-false-NULL rescue +> deep-tested top-K(2)/family + all 7 high-loss sentinels). One directional-gain cell +> (`f0-n1-plain-s7-b0`, +3.3% goodput) was **not** Holm-significant (adj p=0.380); +> 15/20 cells regressed. The FEC cells showed a caller-transport `ts_sync` artifact +> (ffmpeg→`srt-live-transmit` re-mux, not a receiver defect), but no FEC cell showed +> gain regardless. The stage's `PORT_MISMATCH=1` falsifiability control FAILED first +> (`falsifiability_control:{pass:false,verified:true}`), so the NULL is well-formed. +> The `low-latency-fec` FULL_PROFILE remains intentionally **unreachable** pending +> future evidence — this campaign found no promotable FEC/NAK/FREEZE combo. +> +> **How it ran:** `--stage screen` sweeps all 7 candidates +> (REORDERFREEZE × NAKREPORT × FEC, baseline excluded, `LOSSMAXTTL=40` held) across a +> reduced adverse grid at low reps (=4) and emits the **survivors** set; `--stage +> deep` runs the **deep set = screen-survivors ∪ top-K(2)/family ∪ the high-loss +> SENTINEL cells (`STEADY_LOSS=7,BURST=20` per candidate, ALWAYS)** at reps=10, then +> applies `--analyze` across **every** deep cell and writes `verdict.json`. The +> sentinels are deep-tested even when the screen rejected them — the **anti-false-NULL +> rescue** (Oracle O4: a low-rep screen can miss a real effect at the directional +> survivor threshold; testing only survivors would then bury it as a FALSE NULL). +> Each stage runs a `PORT_MISMATCH=1` falsifiability control FIRST and ABORTS (exit 2) +> if it passes. The **§2 statistics engine** is `--analyze`: exact Mann-Whitney U +> (pure-stdlib — scipy is absent on the box) + Holm-Bonferroni across every cell (§5). +> The PRIMARY sender is `srtla-send-rs`; a run with no fork resolvable SKIPs (exit 77). --- @@ -74,6 +100,7 @@ In **every** cell: | TS continuity | `ts_cc_errors(C) ≤ ts_cc_errors(B)` (median) | | Goodput floor | `goodput_bps(C) ≥ 0.99 × goodput_bps(B)` (no goodput sold to buy drops) | | Wire overhead | `wire_amp(C) ≤ 1.10 × wire_amp(B)` (FEC overhead bounded — the point of `arq:onreq` over pure FEC is to pay parity bytes only when they buy a gain) | +| Reverse overhead | `reverse_wire_amp(C) ≤ 1.10 × reverse_wire_amp(B)` (periodic-NAK's receiver→sender cost is bounded; the instrument meters `$PEERIF` egress inside the netns so this is no longer invisible) | | Tail latency | `p95 pkt_rcv_drop(C) ≤ p95 pkt_rcv_drop(B)` | ### Verdict @@ -88,21 +115,47 @@ cloud capability descriptor or the CeraUI catalog. ## 3. Candidate-Mixture Matrix -FEC is always `arq:onreq`. Candidates vary the SRT packet-filter geometry only: - -| id | packetfilter spec (caller side) | arq | -|----|--------------------------------|-----| -| `m-even-8x8` | `fec,cols:8,rows:8,layout:even` | `onreq` | -| `m-even-10x10` | `fec,cols:10,rows:10,layout:even` | `onreq` | -| `m-stair-8x8` | `fec,cols:8,rows:8,layout:staircase` | `onreq` | -| `m-stair-12x6` | `fec,cols:12,rows:6,layout:staircase` | `onreq` | -| `m-cols-only` | `fec,cols:10,rows:1,layout:even` (column-only parity) | `onreq` | -| (baseline) | *no packet filter — Classic L2 + latency, ARQ only* | n/a | -| (banned control) | `fec,cols:8,rows:8,layout:even,arq:never` | `never` — asserted to NEVER promote | - -The even vs staircase layout and the cols/rows ratio trade reconstruction latency -against burst-loss coverage; column-only parity is the cheapest geometry and the -natural "does any FEC pay off at all" probe. +Three binary axes define the screen matrix (2 × 2 × 2 = 8 tuples). The baseline +tuple is one of the eight; the other 7 are candidates measured against it: + +| Axis | Values | SRT option | +|------|--------|-----------| +| `REORDERFREEZE` | `{1, 0}` | `SRTO_REORDERFREEZE` (decay-freeze on/off) | +| `NAKREPORT` | `{0, 1}` | `SRTO_NAKREPORT` (periodic NAK report off/on) | +| `FEC` | `{off, on}` | caller `SRTO_PACKETFILTER` (see geometry below) | +| `LOSSMAXTTL` | `40` (held constant) | `SRTO_LOSSMAXTTL` (BELABOX-parity reorder-tolerance cap) | + +**Baseline B** = `REORDERFREEZE=1, NAKREPORT=0, FEC=off` (Classic L2 + latency, ARQ always on). + +The 7 candidates (all non-baseline tuples): + +| id | REORDERFREEZE | NAKREPORT | FEC | SRTO tuple | +|----|--------------|-----------|-----|-----------| +| `f1-n0-fec` | 1 | 0 | on | freeze=1, nak=0, packetfilter=fec | +| `f1-n1-plain` | 1 | 1 | off | freeze=1, nak=1 | +| `f1-n1-fec` | 1 | 1 | on | freeze=1, nak=1, packetfilter=fec | +| `f0-n0-plain` | 0 | 0 | off | freeze=0, nak=0 | +| `f0-n0-fec` | 0 | 0 | on | freeze=0, nak=0, packetfilter=fec | +| `f0-n1-plain` | 0 | 1 | off | freeze=0, nak=1 | +| `f0-n1-fec` | 0 | 1 | on | freeze=0, nak=1, packetfilter=fec | + +**FEC geometry constraint (pre-registered, Oracle O5):** FEC is always `arq:onreq` +hybrid. The screen matrix fixes one geometry: `fec,cols:16,rows:1,layout:even,arq:onreq` +(column-only parity, ~6% overhead). Column-only (`rows:1`) is the cheapest geometry and +the natural "does any FEC pay off at all" probe. The `cols≥16` constraint is pre-registered +to leave clear headroom under the `wire_amp ≤1.10×` budget: parity overhead is `1/cols`, so +`cols:16` ⇒ ~6.25% (clear headroom), while `cols:10` sits exactly on the 10% cliff and +`cols:8` ⇒ ~12.5% would **fail the budget by construction** (Oracle O5). Pure FEC +(`arq:never`) is BANNED across the whole stack — it has no retransmit floor and regressed +on every prior trial. The orchestrator REFUSES (exit 2) any tuple whose packetfilter +contains `arq:never`. + +The executable geometry guard is +[`../tests/compat/scenarios/gain-hunt-geometry-lint.sh`](../tests/compat/scenarios/gain-hunt-geometry-lint.sh): +it parses every `fec,…,cols:,…` spec in this doc and the orchestrator, computes the +column-parity overhead (`1/cols`), and FAILS any geometry whose overhead exceeds the +`wire_amp ≤1.10×` budget — so a regression to `cols:8` (~12.5%) or `cols:10` (exactly 10%) +trips the lint, not a privileged campaign run. --- @@ -119,37 +172,87 @@ byte-identical to its pre-axis behaviour (Rule E). | `RTT_SPREAD_MS` | `{0, 150, 400}` | Extra one-way delay added to the **slow** link only, widening cross-link skew past the built-in 50/150 ms band (slow delay = `150 + RTT_SPREAD_MS`) | A **cell** is one `(candidate, steady, burst, rtt)` point; the baseline is measured -in the **same** cell with the same seed. The default candidate-cell count is -`5 candidates × 4 × 2 × 3 = 120` candidate-cells, each paired with a baseline run. -`reorder-stress.sh` already sweeps bitrate (`BITRATE_KBPS`) and receive latency -(`RX_LATENCY_MS`); the gain hunt holds those at the production profile while -sweeping the loss/RTT axes, then spot-checks the winners across bitrates. +in the **same** cell with the same seed. The full-grid candidate-cell count is +`7 candidates × 4 × 2 × 3 = 168` candidate-cells, each paired with a baseline run +(the two-stage screen→deep sweep in §6 runs a far smaller reduced grid first, so the +full grid is the upper bound, not the screen cost). `reorder-stress.sh` already sweeps +bitrate (`BITRATE_KBPS`) and receive latency (`RX_LATENCY_MS`); the gain hunt holds +those at the production profile while sweeping the loss/RTT axes, then spot-checks the +winners across bitrates. Where the axes pay off (the search hypothesis): FEC redundancy should pay under high steady + bursty loss; a higher `lossmaxttl` should pay under extreme reorder; wide RTT spread stresses the bonded reassembly window. +**Reverse-channel guardrail (NAK axis).** The `NAKREPORT` axis turns on periodic +receiver→sender NAK reports, which cost reverse-channel bytes. That cost is bounded by +the `reverse_wire_amp ≤1.10×` guardrail in §2: a NAK-on candidate cannot buy a forward +gain by flooding the receiver→sender path. The instrument now meters `$PEERIF` egress +**inside** the receiver netns (T-A4) — `setup_topology` installs a countable `prio` root +qdisc on the veth peer (which defaults to `noqueue`, so the reverse egress had no Sent +counter), and the scenario emits `reverse_wire_bytes` / `reverse_wire_amp`. So the +periodic-NAK reverse cost is no longer invisible and cannot false-promote on forward +`wire_amp` alone. **Empirical note (T-A4):** in this SRTLA topology the reverse channel +is dominated by per-packet broadcast ACKs, so NAK-**off** (more imprecise retransmits) +typically costs *more* reverse than NAK-**on** — the guardrail's value is visibility, not +a fixed direction. + +**Geometry constraint (FEC axis).** The FEC axis fixes one pre-registered geometry +(`fec,cols:16,rows:1,layout:even,arq:onreq`, ~6% column-only parity). The `cols≥16` +floor keeps the forward FEC overhead clear of the `wire_amp ≤1.10×` budget by +construction (§3, Oracle O5); the executable lint +(`gain-hunt-geometry-lint.sh`) rejects any narrower geometry (e.g. `cols:8` ⇒ ~12.5%). + --- -## 5. Using the Scaffold +## 5. Using the Orchestrator -The stub is falsifiable: it documents the protocol and **refuses to claim a gain** -without measured evidence. +The orchestrator is falsifiable: it REFUSES `arq:never` in every mode (exit 2) and +**refuses to claim a gain** without the measured cross-cell campaign (exit 3). ```bash # Print the notice + matrix summary (exit 0): tests/compat/scenarios/gain-hunt-matrix.sh -# Print the full planned campaign (exit 0): +# Print the full 3-axis FEC×NAK×FREEZE matrix + each cell's SRTO tuple (exit 0): tests/compat/scenarios/gain-hunt-matrix.sh --dry-run # The decision rule + candidate matrix in full (exit 0): tests/compat/scenarios/gain-hunt-matrix.sh --help -# Attempt to assert a gain — REFUSED (exit 3). A gain cannot be claimed by -# running this stub; no measured candidate-vs-baseline evidence exists. +# Run ONE paired cell (NAK-on candidate vs Classic baseline). The PORT_MISMATCH +# falsifiability control runs FIRST (must fail); a control that passes SKIPs (exit 77). +# Needs CAP_NET_ADMIN and a resolvable srtla-send-rs; SKIPs (exit 77) otherwise. +SRTLA_SEND_RS_BIN=/path/to/srtla_send_rs \ + tests/compat/scenarios/gain-hunt-matrix.sh --smoke --duration 8 + +# Print the planned cell set with NO privilege (nothing executed). The deep plan +# lists the SENTINEL cells for EVERY family — including screen-rejected ones — so you +# can see the anti-false-NULL rescue before committing a privileged run: +tests/compat/scenarios/gain-hunt-matrix.sh --stage screen --plan +tests/compat/scenarios/gain-hunt-matrix.sh --stage deep --plan + +# STAGE 1 — screen: 7 candidates × a reduced adverse grid at low reps (=4); emits +# survivors.json (possibly empty). Falsifiability control first; ABORTS (exit 2) if it passes. +SRTLA_SEND_RS_BIN=/path/to/srtla_send_rs \ + tests/compat/scenarios/gain-hunt-matrix.sh --stage screen + +# STAGE 2 — deep: deep set = survivors ∪ top-K(2)/family ∪ sentinels (ALWAYS) at +# reps=10, then --analyze across every deep cell -> verdict.json (promoted[...] or NULL). +SRTLA_SEND_RS_BIN=/path/to/srtla_send_rs \ + tests/compat/scenarios/gain-hunt-matrix.sh --stage deep + +# Apply the §2 decision-rule statistics to ALREADY-MEASURED paired evidence and emit a +# verdict JSON (exact Mann-Whitney U + Holm-Bonferroni; pure stdlib, no scipy, no +# privilege).

is a fixture JSON or a dir of /{candidate,baseline}/rep-*.json. +# Exit 0 = promoted, 1 = not promoted, 2 = no usable evidence. +tests/compat/scenarios/gain-hunt-matrix.sh --analyze tests/compat/fixtures/gain-hunt-golden/gain-fixture.json + +# Attempt to assert a gain — REFUSED (exit 3). A gain cannot be claimed by running +# this script; only the cross-cell §2 stats (which --analyze implements, fed the +# T-A6 deep-stage evidence) may. tests/compat/scenarios/gain-hunt-matrix.sh --claim-gain \ - --candidate m-even-8x8 \ + --candidate f1-n1-fec \ --baseline tests/compat/results/.../result.json \ --decision-rule docs/GAIN-HUNT-PROTOCOL.md ``` @@ -174,32 +277,82 @@ STEADY_LOSS_PCT=3 BURST_LOSS_PCT=20 RTT_SPREAD_MS=150 \ --- -## 6. Running the Full Campaign (Future R&D Track) - -Implementing the campaign — explicitly **out of scope** for the scaffold effort — -means wiring `gain-hunt-matrix.sh` to: - -1. **Build and pin each candidate's FEC libsrt geometry.** The device-side FEC - packet-filter is only compiled when a FEC mixture is actively being earned - (deferred per `RECEIVER-RECONCILIATION.md` "Out of Scope"). The receiver-side - libsrt is swapped under `srt-sink` via `SINK_LD_LIBRARY_PATH`, exactly as - `profile-validation-matrix.sh` swaps its baseline/freeze artifacts. -2. **Run `reorder-stress.sh` paired/alternating** (candidate vs baseline) per cell, - with a shared per-rep `NETEM_SEED`, `N` reps each (start at `N=10`, matching the - profile matrix), passing the FEC packetfilter to `srt-sink` via `SINK_EXTRA_ARGS` - `--packetfilter`. -3. **Collect** `goodput_bps`, `pkt_rcv_drop`, `ts_sync_errors`, `ts_cc_errors`, - `wire_amp`, `disconnects` from each run's `result.json`. -4. **Apply the §2 rule** with the Holm-Bonferroni correction across cells. -5. **Emit** a results JSON + update the evidence table; on a pass, the cloud - capability descriptor and the CeraUI catalog gain the entry. On anything short - of a pass, the mixture stays out of the UI. +## 6. Running the Full Campaign (Wave B R&D Track) + +The two-stage **structure** is wired (T-A6) and Wave B **already ran** it under +`CAP_NET_ADMIN`, producing the **NULL** verdict recorded in the Status block above +(`test-results/gain-hunt/verdict.json`). The steps below document how the +orchestrator drove that run (and how to reproduce or re-run it in the future): + +1. **Stage 1 — screen (`--stage screen`).** Runs all 7 candidates × a reduced adverse + grid (`STEADY_LOSS_PCT ∈ {3,7}`, `BURST_LOSS_PCT ∈ {0,20}`) paired/alternating vs + the Classic baseline at `SCREEN_REPS=4`, shared per-rep `NETEM_SEED`. A combo + **survives** if it shows a directional gain (`goodput ≥ 1.03×` OR `late-drop ≤ + 0.80×`) in ≥1 cell with **no hard-gate failure** (`disconnects==0`, `ts_sync==0`). + Writes `screen-results.json` + `survivors.json` (survivors may be empty). +2. **Stage 2 — deep (`--stage deep`) with the anti-false-NULL rescue.** The deep set = + **screen-survivors ∪ top-K(2)/family by directional effect size ∪ the high-loss + SENTINEL cells (`STEADY_LOSS=7,BURST=20` per candidate, ALWAYS)**. An empty survivor + set STILL deep-tests top-K + sentinels — a NULL verdict is recorded **only** when + the full deep set, sentinels included, shows no promotable candidate. Runs at + `DEEP_REPS=10`. This is Oracle O4's guard: a low-rep screen can miss a real effect + at the directional survivor threshold, so testing only survivors would bury it as a + FALSE NULL; the sentinels (where FEC should pay off) are re-tested at full power + regardless of the screen outcome. `--stage deep --plan` prints the set without + privilege — the sentinels appear for every family, including screen-rejected ones. +3. **FEC geometry / libsrt swap.** The device-side FEC packet-filter is only compiled + when a mixture is actively being earned (deferred per `RECEIVER-RECONCILIATION.md` + "Out of Scope"); the receiver-side libsrt is swapped under `srt-sink` via + `SINK_LD_LIBRARY_PATH`. The screen/deep cells already pass the FEC packetfilter to + `srt-sink` (via `SINK_EXTRA_ARGS --packetfilter`) on every FEC arm. +4. **Verdict — §2 rule, Holm-Bonferroni across the FULL deep set.** `--stage deep` + runs `--analyze` over the whole deep tree (`/{candidate,baseline}/rep-*.json`) + — exact Mann-Whitney U + Holm across **every** deep cell, not just survivors — and + writes `verdict.json` (`promoted: [...]` or `verdict: NULL`). Each promoted combo + carries `{combo, srt_flags, caller_packetfilter, nak, freeze, evidence_cells}`. The + `--analyze` engine is pinned by the golden fixtures in + `tests/compat/fixtures/gain-hunt-golden/` (validated by + `tests/compat/scenarios/gain-hunt-analyze-test.sh`): a clean-separation gain (exact + `U=100`, `p=2/C(20,10)≈1.0825×10⁻⁵`) promotes; a goodput win bought by a disconnect + or by `reverse_wire_amp > 1.10×` is rejected naming the tripped guardrail; a tie + yields `winner: none`. +5. **Promote.** On a pass the cloud capability descriptor and the CeraUI catalog gain + the entry; on anything short of a pass the mixture stays out of the UI. + +### NULL-verdict policy + +NULL is a **first-class, valid result** — the catalog shipping empty *forever* is an +acceptable, evidence-backed outcome, not a failure of the campaign. The decision rule +puts the burden of proof on the candidate, so "no recipe earns a button" is exactly +what an honest gain hunt should report when no recipe actually pays off. + +NULL is recorded **only** when the **full deep set — sentinels included — shows no +promotable candidate** under the §2 rule (Holm-Bonferroni across every deep cell). The +two guards that make a NULL trustworthy rather than a measurement artifact: + +- **An empty screen-survivor set STILL deep-tests top-K + sentinels.** A NULL is never + recorded just because the screen rejected everything — the high-loss SENTINEL cells + (`STEADY_LOSS=7, BURST=20` per candidate) are deep-tested at full power regardless of + the screen outcome (the anti-false-NULL rescue, Oracle O4). Only if those too fail to + promote is the NULL honest. +- **The stage's falsifiability control must have FAILED** (`PORT_MISMATCH` ⇒ zero bytes + ⇒ `pass:false`) before any arm ran. A control that *passes* ABORTS the stage (exit 2); + a verdict — NULL or promotion — from an instrument that cannot see a broken stream is + not trusted. + +A NULL verdict is written to `verdict.json` as `verdict: NULL` with the deep-set +evidence retained, so the empty catalog is auditable against the cells that produced it. + +**Falsifiability:** each stage runs a `PORT_MISMATCH=1` control **before** the real +arms; it must FAIL (wrong receiver port ⇒ zero bytes ⇒ `pass:false`). A control that +PASSES proves the instrument cannot see a broken stream — the stage ABORTS (exit 2, +"instrument not falsifiable") and no verdict is trusted. **Privilege:** the campaign needs `CAP_NET_ADMIN` (real root, passwordless sudo, or mapped-root in a user+net namespace), gated via `tests/compat/lib/netem.sh` `require`; without it the harness must SKIP (exit 77), never fabricate a -measurement. A `PORT_MISMATCH=1` falsifiability control run must precede the real -arms and must FAIL, proving the instrument can see a broken stream. +measurement. Evidence lands under `test-results/gain-hunt/` (screen/deep rep trees + +`screen-results.json`, `survivors.json`, `deep-results.json`, `verdict.json`). **Rule D:** all artifacts stay within the `srtla` repo (`tests/compat/results/`, `test-results/`); nothing is written above the repo root. diff --git a/tests/compat/fixtures/gain-hunt-golden/expected.json b/tests/compat/fixtures/gain-hunt-golden/expected.json new file mode 100644 index 0000000..73c203d --- /dev/null +++ b/tests/compat/fixtures/gain-hunt-golden/expected.json @@ -0,0 +1,101 @@ +{ + "_README": "Pre-computed expected verdicts for the gain-hunt golden fixtures. p_clean is the EXACT two-sided Mann-Whitney U p-value when all 10 candidate goodput samples strictly exceed all 10 baseline samples: U=100 (=m*n), U_min=0, p = 2 * P(U<=0) = 2 / C(20,10) = 2/184756. No scipy: the exact permutation rank-sum distribution is enumerated by a stdlib subset-sum DP in gain-hunt-matrix.sh --analyze. Verified against the textbook 3v3 case (p=0.1) and 4v4 separated case (p=2/70).", + "constants": { + "p_clean_exact": 1.082508822446903e-05, + "p_clean_fraction": "2/184756", + "holm_2cell_adjusted": 2.165017644893806e-05, + "alpha": 0.05 + }, + "gain-fixture.json": { + "verdict": "promoted", + "promoted": true, + "winner": "f1-n1-fec", + "reason": "real_gain_in_2_cell(s)_no_regression", + "real_gain_cells": [ + "steady3-burst20", + "steady7-burst20" + ], + "regression_cells": {}, + "cells": { + "steady3-burst20": { + "U": 100.0, + "p": 1.082508822446903e-05, + "holm_adjusted_p": 2.165017644893806e-05, + "no_regression": true, + "gain_win": true + }, + "steady7-burst20": { + "U": 100.0, + "p": 1.082508822446903e-05, + "holm_adjusted_p": 2.165017644893806e-05, + "no_regression": true, + "gain_win": true + } + } + }, + "regression-fixture.json": { + "verdict": "not_promoted", + "promoted": false, + "winner": "none", + "reason": "regression_in_>=1_cell", + "real_gain_cells": [], + "regression_cells": { + "steady3-burst20": [ + "disconnects_zero" + ] + }, + "cells": { + "steady3-burst20": { + "U": 100.0, + "p": 1.082508822446903e-05, + "holm_adjusted_p": 1.082508822446903e-05, + "no_regression": false, + "gain_win": true, + "tripped_guardrails": [ + "disconnects_zero" + ] + } + } + }, + "reverse-spam-fixture.json": { + "verdict": "not_promoted", + "promoted": false, + "winner": "none", + "reason": "regression_in_>=1_cell", + "real_gain_cells": [], + "regression_cells": { + "steady3-burst20": [ + "reverse_wire_amp_le_110pct" + ] + }, + "cells": { + "steady3-burst20": { + "U": 100.0, + "p": 1.082508822446903e-05, + "holm_adjusted_p": 1.082508822446903e-05, + "no_regression": false, + "gain_win": true, + "tripped_guardrails": [ + "reverse_wire_amp_le_110pct" + ] + } + } + }, + "tie-fixture.json": { + "verdict": "not_promoted", + "promoted": false, + "winner": "none", + "reason": "no_real_gain", + "real_gain_cells": [], + "regression_cells": {}, + "cells": { + "steady3-burst20": { + "U": 50.0, + "p": 1.0, + "holm_adjusted_p": 1.0, + "no_regression": true, + "gain_win": false + } + } + } +} diff --git a/tests/compat/fixtures/gain-hunt-golden/gain-fixture.json b/tests/compat/fixtures/gain-hunt-golden/gain-fixture.json new file mode 100644 index 0000000..e02fc14 --- /dev/null +++ b/tests/compat/fixtures/gain-hunt-golden/gain-fixture.json @@ -0,0 +1,379 @@ +{ + "fixture": "gain", + "description": "Candidate wins goodput (>=1.03x, clean separation -> exact U=100, p=2/C(20,10)) in BOTH cells; every no-regression guardrail holds. Holm over 2 cells keeps it significant. Verdict: promoted.", + "candidate_id": "f1-n1-fec", + "cells": { + "steady3-burst20": { + "baseline": [ + { + "goodput_bps": 7000000, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000100, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000200, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000300, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000400, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000500, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000600, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000700, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000800, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000900, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + } + ], + "candidate": [ + { + "goodput_bps": 7300000, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300100, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300200, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300300, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300400, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300500, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300600, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300700, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300800, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300900, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + } + ] + }, + "steady7-burst20": { + "baseline": [ + { + "goodput_bps": 6400000, + "pkt_rcv_drop": 30, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.052, + "reverse_wire_amp": 0.024, + "disconnects": 0 + }, + { + "goodput_bps": 6400100, + "pkt_rcv_drop": 30, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.052, + "reverse_wire_amp": 0.024, + "disconnects": 0 + }, + { + "goodput_bps": 6400200, + "pkt_rcv_drop": 30, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.052, + "reverse_wire_amp": 0.024, + "disconnects": 0 + }, + { + "goodput_bps": 6400300, + "pkt_rcv_drop": 30, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.052, + "reverse_wire_amp": 0.024, + "disconnects": 0 + }, + { + "goodput_bps": 6400400, + "pkt_rcv_drop": 30, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.052, + "reverse_wire_amp": 0.024, + "disconnects": 0 + }, + { + "goodput_bps": 6400500, + "pkt_rcv_drop": 30, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.052, + "reverse_wire_amp": 0.024, + "disconnects": 0 + }, + { + "goodput_bps": 6400600, + "pkt_rcv_drop": 30, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.052, + "reverse_wire_amp": 0.024, + "disconnects": 0 + }, + { + "goodput_bps": 6400700, + "pkt_rcv_drop": 30, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.052, + "reverse_wire_amp": 0.024, + "disconnects": 0 + }, + { + "goodput_bps": 6400800, + "pkt_rcv_drop": 30, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.052, + "reverse_wire_amp": 0.024, + "disconnects": 0 + }, + { + "goodput_bps": 6400900, + "pkt_rcv_drop": 30, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.052, + "reverse_wire_amp": 0.024, + "disconnects": 0 + } + ], + "candidate": [ + { + "goodput_bps": 6800000, + "pkt_rcv_drop": 4, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.09, + "reverse_wire_amp": 0.021, + "disconnects": 0 + }, + { + "goodput_bps": 6800100, + "pkt_rcv_drop": 4, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.09, + "reverse_wire_amp": 0.021, + "disconnects": 0 + }, + { + "goodput_bps": 6800200, + "pkt_rcv_drop": 4, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.09, + "reverse_wire_amp": 0.021, + "disconnects": 0 + }, + { + "goodput_bps": 6800300, + "pkt_rcv_drop": 4, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.09, + "reverse_wire_amp": 0.021, + "disconnects": 0 + }, + { + "goodput_bps": 6800400, + "pkt_rcv_drop": 4, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.09, + "reverse_wire_amp": 0.021, + "disconnects": 0 + }, + { + "goodput_bps": 6800500, + "pkt_rcv_drop": 4, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.09, + "reverse_wire_amp": 0.021, + "disconnects": 0 + }, + { + "goodput_bps": 6800600, + "pkt_rcv_drop": 4, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.09, + "reverse_wire_amp": 0.021, + "disconnects": 0 + }, + { + "goodput_bps": 6800700, + "pkt_rcv_drop": 4, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.09, + "reverse_wire_amp": 0.021, + "disconnects": 0 + }, + { + "goodput_bps": 6800800, + "pkt_rcv_drop": 4, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.09, + "reverse_wire_amp": 0.021, + "disconnects": 0 + }, + { + "goodput_bps": 6800900, + "pkt_rcv_drop": 4, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.09, + "reverse_wire_amp": 0.021, + "disconnects": 0 + } + ] + } + } +} diff --git a/tests/compat/fixtures/gain-hunt-golden/regression-fixture.json b/tests/compat/fixtures/gain-hunt-golden/regression-fixture.json new file mode 100644 index 0000000..1562faf --- /dev/null +++ b/tests/compat/fixtures/gain-hunt-golden/regression-fixture.json @@ -0,0 +1,193 @@ +{ + "fixture": "regression", + "description": "Candidate wins goodput (clean separation) but ONE candidate rep disconnects (disc=1) -> disconnects_zero hard gate fails. The gain is real but bought by a stream break. Verdict: not_promoted (tripped: disconnects_zero).", + "candidate_id": "f0-n1-fec", + "cells": { + "steady3-burst20": { + "baseline": [ + { + "goodput_bps": 7000000, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000100, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000200, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000300, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000400, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000500, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000600, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000700, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000800, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000900, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + } + ], + "candidate": [ + { + "goodput_bps": 7300000, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300100, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300200, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300300, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300400, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300500, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 1 + }, + { + "goodput_bps": 7300600, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300700, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300800, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + }, + { + "goodput_bps": 7300900, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.08, + "reverse_wire_amp": 0.018, + "disconnects": 0 + } + ] + } + } +} diff --git a/tests/compat/fixtures/gain-hunt-golden/reverse-spam-fixture.json b/tests/compat/fixtures/gain-hunt-golden/reverse-spam-fixture.json new file mode 100644 index 0000000..da8b7d0 --- /dev/null +++ b/tests/compat/fixtures/gain-hunt-golden/reverse-spam-fixture.json @@ -0,0 +1,193 @@ +{ + "fixture": "reverse-spam", + "description": "NAK-on candidate wins goodput (clean separation) but its reverse_wire_amp median (0.030) exceeds 1.10x baseline (0.015 -> ceil 0.0165) -> reverse_wire_amp_le_110pct trips. A forward gain bought by reverse-channel spam. Verdict: not_promoted (tripped: reverse_wire_amp_le_110pct).", + "candidate_id": "f1-n1-plain", + "cells": { + "steady3-burst20": { + "baseline": [ + { + "goodput_bps": 7000000, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.015, + "disconnects": 0 + }, + { + "goodput_bps": 7000100, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.015, + "disconnects": 0 + }, + { + "goodput_bps": 7000200, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.015, + "disconnects": 0 + }, + { + "goodput_bps": 7000300, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.015, + "disconnects": 0 + }, + { + "goodput_bps": 7000400, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.015, + "disconnects": 0 + }, + { + "goodput_bps": 7000500, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.015, + "disconnects": 0 + }, + { + "goodput_bps": 7000600, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.015, + "disconnects": 0 + }, + { + "goodput_bps": 7000700, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.015, + "disconnects": 0 + }, + { + "goodput_bps": 7000800, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.015, + "disconnects": 0 + }, + { + "goodput_bps": 7000900, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.015, + "disconnects": 0 + } + ], + "candidate": [ + { + "goodput_bps": 7300000, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.06, + "reverse_wire_amp": 0.03, + "disconnects": 0 + }, + { + "goodput_bps": 7300100, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.06, + "reverse_wire_amp": 0.03, + "disconnects": 0 + }, + { + "goodput_bps": 7300200, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.06, + "reverse_wire_amp": 0.03, + "disconnects": 0 + }, + { + "goodput_bps": 7300300, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.06, + "reverse_wire_amp": 0.03, + "disconnects": 0 + }, + { + "goodput_bps": 7300400, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.06, + "reverse_wire_amp": 0.03, + "disconnects": 0 + }, + { + "goodput_bps": 7300500, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.06, + "reverse_wire_amp": 0.03, + "disconnects": 0 + }, + { + "goodput_bps": 7300600, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.06, + "reverse_wire_amp": 0.03, + "disconnects": 0 + }, + { + "goodput_bps": 7300700, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.06, + "reverse_wire_amp": 0.03, + "disconnects": 0 + }, + { + "goodput_bps": 7300800, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.06, + "reverse_wire_amp": 0.03, + "disconnects": 0 + }, + { + "goodput_bps": 7300900, + "pkt_rcv_drop": 0, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.06, + "reverse_wire_amp": 0.03, + "disconnects": 0 + } + ] + } + } +} diff --git a/tests/compat/fixtures/gain-hunt-golden/tie-fixture.json b/tests/compat/fixtures/gain-hunt-golden/tie-fixture.json new file mode 100644 index 0000000..7c54eb2 --- /dev/null +++ b/tests/compat/fixtures/gain-hunt-golden/tie-fixture.json @@ -0,0 +1,193 @@ +{ + "fixture": "tie", + "description": "Candidate arm is byte-identical to the baseline arm: medians equal, no >=1.03x goodput gain and no >=20% drop reduction -> no real gain. Verdict: not_promoted, winner: none (burden of proof unmet).", + "candidate_id": "f0-n0-fec", + "cells": { + "steady3-burst20": { + "baseline": [ + { + "goodput_bps": 7000000, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000100, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000200, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000300, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000400, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000500, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000600, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000700, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000800, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000900, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + } + ], + "candidate": [ + { + "goodput_bps": 7000000, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000100, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000200, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000300, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000400, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000500, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000600, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000700, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000800, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + }, + { + "goodput_bps": 7000900, + "pkt_rcv_drop": 12, + "ts_sync_errors": 0, + "ts_cc_errors": 0, + "wire_amp": 1.05, + "reverse_wire_amp": 0.02, + "disconnects": 0 + } + ] + } + } +} diff --git a/tests/compat/lib/probe-fec-capability.sh b/tests/compat/lib/probe-fec-capability.sh new file mode 100755 index 0000000..bf16f79 --- /dev/null +++ b/tests/compat/lib/probe-fec-capability.sh @@ -0,0 +1,282 @@ +#!/usr/bin/env bash +# +# probe-fec-capability.sh — empirically settle whether the SRT packet-filter +# API (the prerequisite for Forward Error Correction, SRTO_PACKETFILTER) is +# COMPILED into the libsrt builds the cloud receiver path consumes. +# +# This exists to falsify a documentation claim. srt/AGENTS.md states FEC is +# "not compiled" without -DENABLE_PACKET_FILTER=ON — yet srtcore/filelist.maf +# lists fec.cpp + packetfilter.cpp UNCONDITIONALLY (no build flag gates them), +# and upstream libsrt has no ENABLE_PACKET_FILTER option at all. Rather than +# trust either side, this probe MEASURES each library two independent ways: +# +# 1. Symbol probe — `nm -D ` for the FEC/packet-filter symbols +# (FECFilterBuiltin, PacketFilter::ParseConfig, ...). +# Present => the code was compiled into the .so. +# 2. Runtime probe — a tiny C program creates an SRT socket and calls +# srt_setsockopt(SRTO_PACKETFILTER,"fec"). A return of 0 +# means libsrt parsed the FEC filter config and the FEC +# builtin is registered => FEC is usable at runtime. +# +# A library is reported FEC-capable iff EITHER method confirms (they agree in +# practice; both are recorded so neither can be a hard-coded guess). +# +# Three libraries are probed, matching the build-libsrt-matrix.sh slot names +# plus the system loader's libsrt: +# +# system_libsrt the libsrt the runtime loader resolves (the lib +# srt-live-transmit / srt-sink actually link) +# vanilla build-libsrt-matrix.sh --vanilla slot (stock +# Haivision v1.5.5) +# srt_patched_reorderfreeze build-libsrt-matrix.sh --patched slot (CERALIVE +# reorderfreeze-1.5.5) +# +# Output (repo-local, gitignored — Rule D, never escapes the srtla checkout): +# test-results/fec-capability-probe.json +# { system_libsrt: bool, srt_patched_reorderfreeze: bool, vanilla: bool, +# method: "...", evidence: "..." } +# +# By default the two matrix slots are built on demand via build-libsrt-matrix.sh +# pinned to v1.5.5 / reorderfreeze-1.5.5 if they are not already present. Pass +# --no-build to probe only what already exists (missing slots record null, not +# a false "FEC absent" claim). +# +# Usage: +# probe-fec-capability.sh [--no-build] [--matrix-dir DIR] [--system-lib PATH] +# [--jobs N] [-h|--help] +# +# Exit status: 0 on a completed probe (regardless of the booleans); non-zero +# only on a harness error (missing nm/jq, unwritable output). +# +set -uo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd -P)" +REPO_ROOT="$(cd -- "${SCRIPT_DIR}/../../.." >/dev/null 2>&1 && pwd -P)" + +OUT_DIR="${REPO_ROOT}/test-results" +RESULT_JSON="${OUT_DIR}/fec-capability-probe.json" +MATRIX_DIR="${OUT_DIR}/libsrt-matrix/install" +BUILDER="${SCRIPT_DIR}/build-libsrt-matrix.sh" + +VANILLA_REF="v1.5.5" +# build-libsrt-matrix.sh's clone_at checks out --detach ; a bare BRANCH +# name trips git's "--detach cannot be used with -b/-B" DWIM/tracking conflict, +# so the patched slot is pinned to the reorderfreeze-1.5.5 TIP SHA (the builder's +# own documented --patched-ref form). PATCHED_LABEL is the human-readable branch. +PATCHED_REF="66b3609cc004e6a4c485e0adc11149025e782083" +PATCHED_LABEL="reorderfreeze-1.5.5" +VANILLA_URL="https://github.com/Haivision/srt" +PATCHED_URL="https://github.com/CERALIVE/srt" + +DO_BUILD=1 +SYSTEM_LIB="" +JOBS="$(nproc 2>/dev/null || echo 4)" + +# nm -D pattern from the task spec; the decisive signal is a non-zero count of +# FEC/packet-filter symbols (FECFilterBuiltin is unambiguous). +SYM_PATTERN='fec|FECFilterBuiltin|PacketFilter' + +log() { printf '%s\n' "$*" >&2; } +die() { printf 'probe-fec-capability: %s\n' "$*" >&2; exit 2; } + +while [[ $# -gt 0 ]]; do + case "$1" in + --no-build) DO_BUILD=0; shift ;; + --matrix-dir) MATRIX_DIR="${2:?--matrix-dir needs a value}"; shift 2 ;; + --system-lib) SYSTEM_LIB="${2:?--system-lib needs a value}"; shift 2 ;; + --jobs) JOBS="${2:?--jobs needs a value}"; shift 2 ;; + -h|--help) sed -n '2,/^set -uo/p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//; /^set -uo/d'; exit 0 ;; + *) die "unknown argument: $1 (try --help)" ;; + esac +done + +for tool in nm jq; do + command -v "$tool" >/dev/null 2>&1 || die "required tool not found: $tool" +done + +mkdir -p "${OUT_DIR}" + +# --------------------------------------------------------------------------- # +# Resolve the system libsrt the runtime loader actually uses. # +# --------------------------------------------------------------------------- # +resolve_system_lib() { + [[ -n "$SYSTEM_LIB" ]] && { printf '%s' "$SYSTEM_LIB"; return 0; } + # Prefer what an SRT tool links; fall back to ldconfig, then well-known paths. + local p + if command -v srt-live-transmit >/dev/null 2>&1; then + p="$(ldd "$(command -v srt-live-transmit)" 2>/dev/null \ + | awk '/libsrt\.so/ {print $3; exit}')" + [[ -n "$p" && -e "$p" ]] && { printf '%s' "$p"; return 0; } + fi + p="$(ldconfig -p 2>/dev/null | awk '/libsrt\.so\.[0-9]/ {print $NF; exit}')" + [[ -n "$p" && -e "$p" ]] && { printf '%s' "$p"; return 0; } + for p in /usr/local/lib/libsrt.so.1.5 /usr/lib/libsrt.so.1.5 \ + /usr/lib/x86_64-linux-gnu/libsrt.so.1.5; do + [[ -e "$p" ]] && { printf '%s' "$p"; return 0; } + done + return 1 +} + +# Real (non-symlink) shared object inside an install prefix. +find_so() { + local prefix="$1" + find "$prefix" -name 'libsrt.so.*' -type f 2>/dev/null | sort | head -1 +} + +# --------------------------------------------------------------------------- # +# Compile the runtime probe once. Empty RUNTIME_BIN => runtime method skipped. # +# --------------------------------------------------------------------------- # +RUNTIME_BIN="" +TMP_PROBE="$(mktemp -d "${TMPDIR:-/tmp}/fecprobe.XXXXXX")" +cleanup() { rm -rf "${TMP_PROBE}"; } +trap cleanup EXIT INT TERM + +compile_runtime_probe() { + command -v cc >/dev/null 2>&1 || command -v gcc >/dev/null 2>&1 || return 1 + local cc; cc="$(command -v cc || command -v gcc)" + [[ -e /usr/include/srt/srt.h ]] || return 1 + cat > "${TMP_PROBE}/fecprobe.c" <<'PROBE_C' +/* Returns 0 iff srt_setsockopt(SRTO_PACKETFILTER,"fec") is accepted by the + * linked libsrt — i.e. the packet-filter API + FEC builtin are compiled in. */ +#include +#include +int main(void) { + srt_startup(); + SRTSOCKET s = srt_create_socket(); + if (s == SRT_INVALID_SOCK) { srt_cleanup(); return 2; } + int rc = srt_setsockopt(s, 0, SRTO_PACKETFILTER, "fec", 3); + if (rc != 0) fprintf(stderr, "%s\n", srt_getlasterror_str()); + srt_close(s); + srt_cleanup(); + return rc == 0 ? 0 : 1; +} +PROBE_C + "$cc" "${TMP_PROBE}/fecprobe.c" -o "${TMP_PROBE}/fecprobe_bin" -lsrt 2>/dev/null \ + || return 1 + RUNTIME_BIN="${TMP_PROBE}/fecprobe_bin" + return 0 +} + +# symbol_count -> echoes count of matching FEC/packet-filter symbols +symbol_count() { + local lib="$1" + [[ -e "$lib" ]] || { echo 0; return; } + nm -D "$lib" 2>/dev/null | grep -icE "$SYM_PATTERN" || true +} + +# runtime_ok -> 0 if the probe links the lib in and FEC sets. +# Empty uses the default loader (system lib). +runtime_ok() { + local ld_dir="$1" + [[ -n "$RUNTIME_BIN" ]] || return 2 # 2 = method unavailable + if [[ -n "$ld_dir" ]]; then + LD_LIBRARY_PATH="$ld_dir" "$RUNTIME_BIN" >/dev/null 2>&1 + else + "$RUNTIME_BIN" >/dev/null 2>&1 + fi +} + +# --------------------------------------------------------------------------- # +# Optionally build the two matrix slots pinned to the evaluated refs. # +# --------------------------------------------------------------------------- # +VANILLA_SO="$(find_so "${MATRIX_DIR}/vanilla")" +PATCHED_SO="$(find_so "${MATRIX_DIR}/patched")" + +if [[ "$DO_BUILD" -eq 1 && ( -z "$VANILLA_SO" || -z "$PATCHED_SO" ) ]]; then + if [[ -x "$BUILDER" ]]; then + log "==> matrix libs absent; building vanilla=${VANILLA_REF} patched=${PATCHED_LABEL}(${PATCHED_REF})" + if bash "$BUILDER" \ + --vanilla-url "$VANILLA_URL" --vanilla-ref "$VANILLA_REF" \ + --patched-url "$PATCHED_URL" --patched-ref "$PATCHED_REF" \ + --jobs "$JOBS" >&2; then + VANILLA_SO="$(find_so "${MATRIX_DIR}/vanilla")" + PATCHED_SO="$(find_so "${MATRIX_DIR}/patched")" + else + log "==> WARN: build-libsrt-matrix.sh failed; probing what exists only" + fi + else + log "==> WARN: builder not executable at ${BUILDER}; skipping build" + fi +fi + +SYSTEM_LIB="$(resolve_system_lib || true)" + +compile_runtime_probe && RUNTIME_AVAIL=1 || RUNTIME_AVAIL=0 + +# --------------------------------------------------------------------------- # +# Probe each library. Sets _BOOL (true|false|null) + appends evidence. # +# --------------------------------------------------------------------------- # +EVIDENCE="" +add_ev() { EVIDENCE="${EVIDENCE}${EVIDENCE:+ | }$*"; } + +# probe_one

apply the pre-registered §2 decision-rule +# statistics to ALREADY-MEASURED paired evidence +#

(a self-contained fixture JSON, or a dir of +# /{candidate,baseline}/rep-*.json). Exact +# Mann-Whitney U (stdlib-only, no scipy) + Holm- +# Bonferroni across every cell; emits a verdict +# JSON to stdout. This is the §2 stats layer the +# deep stage calls — it COMPUTES a verdict over +# supplied evidence, it does not RUN the campaign. +# gain-hunt-matrix.sh --claim-gain … attempt to assert a gain. REFUSED (exit 3) — +# a gain cannot be claimed by running this +# script; only the measured campaign + the §2 +# rule may. This is the falsifiability anchor. # -# Rule D: writes nothing above the srtla repo root (the stub writes nothing at all). +# Sender: SRTLA_SEND_RS_BIN (or a srtla_send_rs on PATH) selects the Rust fork as +# the production sender; run modes SKIP (exit 77) when it is absent. +# Rule D: writes nothing above the srtla repo root (results stay in tests/compat/). # set -uo pipefail SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)" +REPO_ROOT="$(cd -- "${SCRIPT_DIR}/../../.." >/dev/null 2>&1 && pwd)" PROTOCOL_DOC="${SCRIPT_DIR}/../../../docs/GAIN-HUNT-PROTOCOL.md" +REORDER="${SCRIPT_DIR}/reorder-stress.sh" +REORDER_RESULT="${SCRIPT_DIR}/../results/reorder-stress/result.json" +NETEM_LIB="${SCRIPT_DIR}/../lib/netem.sh" +RESULTS_DIR="${SCRIPT_DIR}/../results/gain-hunt-matrix" log() { printf '%s\n' "$*" >&2; } die() { printf 'gain-hunt-matrix: %s\n' "$*" >&2; exit 2; } -# Candidate ids and adverse axes (mirrors the header matrix + the protocol doc). -CANDIDATES=(m-even-8x8 m-even-10x10 m-stair-8x8 m-stair-12x6 m-cols-only) -STEADY_LOSS_GRID=(0 1 3 7) -BURST_LOSS_GRID=(0 20) -RTT_SPREAD_GRID=(0 150 400) +# --------------------------------------------------------------------------- # +# Matrix constants (fixed before any measurement; mirror the header + protocol). # +# --------------------------------------------------------------------------- # +GAIN_FEC_FILTER="${GAIN_FEC_FILTER:-fec,cols:16,rows:1,layout:even,arq:onreq}" +LOSSMAXTTL_FIXED=40 +BASE_FREEZE=1; BASE_NAK=0; BASE_FEC_ON=0 # baseline B = Classic +RX_LATENCY_MS="${RX_LATENCY_MS:-1200}" +BITRATE_KBPS="${BITRATE_KBPS:-8000}" +NETEM_SEED_BASE="${NETEM_SEED_BASE:-1000}" +GAIN_STEADY_LOSS_PCT="${GAIN_STEADY_LOSS_PCT:-}" # adverse axis (deep stage / smoke) + +# --------------------------------------------------------------------------- # +# Two-stage campaign (screen -> deep) + anti-false-NULL rescue (T-A6). # +# --------------------------------------------------------------------------- # +# Evidence tree lives under the repo's gitignored test-results/ (Rule D: still +# inside the srtla repo, never above it). Stage summaries land at GAIN_OUT root; +# per-cell paired reps land under GAIN_OUT/{screen,deep}//{candidate,baseline}. +GAIN_OUT="${GAIN_OUT:-${REPO_ROOT}/test-results/gain-hunt}" +SCREEN_REPS="${SCREEN_REPS:-4}" # low reps for the wide, cheap screen sweep +DEEP_REPS="${DEEP_REPS:-10}" # full-power reps for the selected deep set +DEEP_TOPK="${DEEP_TOPK:-2}" # top-K cells per candidate family rescued into deep +# Reduced adverse grid for the screen sweep. It MUST include the sentinel point so +# the rescue can re-promote a sentinel cell the low-rep screen marked non-surviving. +SCREEN_STEADY=(3 7) # STEADY_LOSS_PCT grid (screen) +SCREEN_BURST=(0 20) # BURST_LOSS_PCT grid (screen) +SENTINEL_STEADY="${SENTINEL_STEADY:-7}" # high-loss SENTINEL cell — ALWAYS in deep, +SENTINEL_BURST="${SENTINEL_BURST:-20}" # one per candidate (STEADY=7,BURST=20) +CONTROL_PASS="" # set by run_falsifiability_control + +# Resolve the Rust fork sender exactly as reorder-stress.sh does (env or PATH). +SRTLA_SEND_RS_BIN="${SRTLA_SEND_RS_BIN:-}" +if [[ -z "$SRTLA_SEND_RS_BIN" ]] && command -v srtla_send_rs >/dev/null 2>&1; then + SRTLA_SEND_RS_BIN="$(command -v srtla_send_rs)" +fi + +# Build the FEC×NAK×FREEZE tuple list. Each entry is "freeze:nak:fecon"; the +# baseline tuple is skipped so CANDIDATES holds the 7 challengers. +CANDIDATES=() +build_candidates() { + CANDIDATES=() + local frz nak fec + for frz in 1 0; do + for nak in 0 1; do + for fec in 0 1; do + [[ "$frz" == "$BASE_FREEZE" && "$nak" == "$BASE_NAK" && "$fec" == "$BASE_FEC_ON" ]] && continue + CANDIDATES+=("${frz}:${nak}:${fec}") + done + done + done +} + +# Falsifiability: REFUSE any tuple that would use arq:never, and require every FEC +# tuple to carry arq:onreq. Runs in EVERY mode (even --dry-run) so a banned recipe +# cannot be enumerated. GAIN_FEC_FILTER is the single FEC spec all FEC cells share. +assert_no_arq_never() { + [[ "$GAIN_FEC_FILTER" == *arq:never* ]] && \ + die "REFUSED: GAIN_FEC_FILTER carries arq:never (pure FEC is BANNED; FEC must be arq:onreq)" + [[ "$GAIN_FEC_FILTER" =~ ^fec, ]] || \ + die "REFUSED: GAIN_FEC_FILTER must start with 'fec,' (got '$GAIN_FEC_FILTER')" + [[ "$GAIN_FEC_FILTER" == *arq:onreq* ]] || \ + die "REFUSED: FEC arms must be arq:onreq (GAIN_FEC_FILTER='$GAIN_FEC_FILTER' lacks it)" +} + +# Human label + the SRTO tuple a cell drives reorder-stress.sh with. +cell_label() { printf 'f%s-n%s-%s' "$1" "$2" "$([[ "$3" == 1 ]] && echo fec || echo plain)"; } +cell_filter() { [[ "$1" == 1 ]] && printf '%s' "$GAIN_FEC_FILTER" || printf ''; } +cell_srto() { # freeze nak fecon + local pf; pf="$([[ "$3" == 1 ]] && printf '%s' "$GAIN_FEC_FILTER" || printf '')" + printf 'REORDERFREEZE=%s NAKREPORT=%s LOSSMAXTTL=%s packetfilter=%s' \ + "$1" "$2" "$LOSSMAXTTL_FIXED" "$pf" +} + +print_matrix() { + build_candidates + log " axes: REORDERFREEZE in {1,0} NAKREPORT in {0,1} FEC in {off,on} (LOSSMAXTTL=${LOSSMAXTTL_FIXED} held)" + log " FEC spec (arq:onreq hybrid; arq:never BANNED): ${GAIN_FEC_FILTER}" + log " baseline B: $(cell_srto "$BASE_FREEZE" "$BASE_NAK" "$BASE_FEC_ON") [$(cell_label "$BASE_FREEZE" "$BASE_NAK" "$BASE_FEC_ON")]" + log " candidates (${#CANDIDATES[@]} = 2x2x2 - baseline):" + local t frz nak fec + for t in "${CANDIDATES[@]}"; do + IFS=: read -r frz nak fec <<<"$t" + log " - $(cell_label "$frz" "$nak" "$fec") :: $(cell_srto "$frz" "$nak" "$fec")" + done +} # --------------------------------------------------------------------------- # # CLI # # --------------------------------------------------------------------------- # MODE="notice" +STAGE="" +PLAN=0 +ANALYZE_PATH="" +AGGREGATE_DEEP_DIR="" CANDIDATE="" BASELINE="" DECISION_RULE="" +REPS="${REPS:-1}" +PHASE_SEC="${PHASE_SEC:-12}" +BUILD_DIR="${SRTLA_BUILD_DIR:-}" while [[ $# -gt 0 ]]; do case "$1" in --dry-run) MODE="dry-run"; shift ;; + --plan) PLAN=1; shift ;; + --smoke) MODE="smoke"; shift ;; + --stage) MODE="stage"; STAGE="${2:?--stage needs screen|deep}"; shift 2 ;; + --analyze) MODE="analyze"; ANALYZE_PATH="${2:?--analyze needs a path}"; shift 2 ;; + --aggregate-deep) MODE="aggregate-deep"; AGGREGATE_DEEP_DIR="${2:?--aggregate-deep needs a deep dir}"; shift 2 ;; --claim-gain) MODE="claim-gain"; shift ;; --candidate) CANDIDATE="${2:?--candidate needs a value}"; shift 2 ;; --baseline) BASELINE="${2:?--baseline needs a value}"; shift 2 ;; --decision-rule) DECISION_RULE="${2:?--decision-rule needs a value}"; shift 2 ;; - -h|--help) sed -n '2,95p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; + --reps) REPS="${2:?--reps needs a value}"; shift 2 ;; + --duration) PHASE_SEC="${2:?--duration needs a value}"; shift 2 ;; + --build-dir) BUILD_DIR="${2:?--build-dir needs a value}"; shift 2 ;; + -h|--help) sed -n '2,110p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; *) die "unknown argument '$1' (try --help)" ;; esac done +[[ "$REPS" =~ ^[0-9]+$ && "$REPS" -ge 1 ]] || die "--reps must be a positive integer" +[[ "$PHASE_SEC" =~ ^[0-9]+$ && "$PHASE_SEC" -ge 1 ]] || die "--duration must be a positive integer" -print_matrix() { - log " candidate mixtures (FEC always arq:onreq; arq:never is BANNED):" - local c - for c in "${CANDIDATES[@]}"; do log " - ${c}"; done - log " adverse-config axes (drive scenarios/reorder-stress.sh):" - log " STEADY_LOSS_PCT in {${STEADY_LOSS_GRID[*]}}" - log " BURST_LOSS_PCT in {${BURST_LOSS_GRID[*]}}" - log " RTT_SPREAD_MS in {${RTT_SPREAD_GRID[*]}}" - local cells=$(( ${#CANDIDATES[@]} * ${#STEADY_LOSS_GRID[@]} * ${#BURST_LOSS_GRID[@]} * ${#RTT_SPREAD_GRID[@]} )) - log " matrix size: ${#CANDIDATES[@]} candidates x ${#STEADY_LOSS_GRID[@]} x ${#BURST_LOSS_GRID[@]} x ${#RTT_SPREAD_GRID[@]} = ${cells} candidate-cells (+ paired baseline per cell)" +# Every mode self-checks the arq:never ban first — a banned recipe is never enumerated. +assert_no_arq_never + +# --------------------------------------------------------------------------- # +# Run helpers (cloned from profile-validation-matrix.sh run_rep/run_cell): one # +# reorder-stress rep per arm, alternating arm order per rep, shared per-rep seed # +# so both arms meet the same netem reorder draw (paired comparison). # +# --------------------------------------------------------------------------- # +RUN_LOG="" + +run_rep() { # out freeze nak fecon label seed [steady] [burst] [port_mismatch] + local out="$1" frz="$2" nak="$3" fec="$4" label="$5" seed="$6" + local steady="${7-$GAIN_STEADY_LOSS_PCT}" burst="${8-}" pm="${9:-0}" + local -a env_kv=( + "SRTLA_BUILD_DIR=${BUILD_DIR}" + "SRTLA_SEND_RS_BIN=${SRTLA_SEND_RS_BIN}" + "REQUIRE_RS_SENDER=1" + "RX_LATENCY_MS=${RX_LATENCY_MS}" + "BITRATE_KBPS=${BITRATE_KBPS}" + "LOSSMAXTTL=${LOSSMAXTTL_FIXED}" + "REORDERFREEZE=${frz}" + "NAKREPORT=${nak}" + "PROFILE_LABEL=${label}" + "NETEM_SEED=${seed}" + ) + [[ -n "$steady" ]] && env_kv+=("STEADY_LOSS_PCT=${steady}") + [[ -n "$burst" ]] && env_kv+=("BURST_LOSS_PCT=${burst}") + [[ "$pm" == "1" ]] && env_kv+=("PORT_MISMATCH=1") + if [[ "$fec" == 1 ]]; then + env_kv+=("CALLER_PACKETFILTER=${GAIN_FEC_FILTER}" "SINK_EXTRA_ARGS=--packetfilter fec") + fi + rm -f "$REORDER_RESULT" + env "${env_kv[@]}" bash "$REORDER" --duration "$PHASE_SEC" >>"$RUN_LOG" 2>&1 + local rc=$? + if [[ -f "$REORDER_RESULT" ]]; then cp "$REORDER_RESULT" "$out"; else printf '{}\n' > "$out"; fi + return "$rc" +} + +run_cell() { # dir cand_freeze cand_nak cand_fecon cand_label [steady] [burst] [reps] + local dir="$1" cf="$2" cn="$3" cfec="$4" clabel="$5" + local steady="${6-$GAIN_STEADY_LOSS_PCT}" burst="${7-}" reps="${8:-$REPS}" + mkdir -p "${dir}/baseline" "${dir}/candidate" + local r seed + for (( r=1; r<=reps; r++ )); do + seed=$(( NETEM_SEED_BASE + r )) + log " [${clabel}] rep ${r}/${reps} (seed ${seed}, steady=${steady:-0} burst=${burst:-0})" + if (( r % 2 == 1 )); then + run_rep "${dir}/baseline/rep-${r}.json" "$BASE_FREEZE" "$BASE_NAK" "$BASE_FEC_ON" "baseline-classic" "$seed" "$steady" "$burst" 0 || true + run_rep "${dir}/candidate/rep-${r}.json" "$cf" "$cn" "$cfec" "$clabel" "$seed" "$steady" "$burst" 0 || true + else + run_rep "${dir}/candidate/rep-${r}.json" "$cf" "$cn" "$cfec" "$clabel" "$seed" "$steady" "$burst" 0 || true + run_rep "${dir}/baseline/rep-${r}.json" "$BASE_FREEZE" "$BASE_NAK" "$BASE_FEC_ON" "baseline-classic" "$seed" "$steady" "$burst" 0 || true + fi + done +} + +# Adverse-grid cell label: recipe tuple + steady/burst suffix (deep set keys on this). +adv_cell_label() { printf '%s-s%s-b%s' "$(cell_label "$1" "$2" "$3")" "$4" "$5"; } + +# Falsifiability control: a PORT_MISMATCH=1 candidate rep MUST fail (wrong receiver +# port yields zero bytes yields pass:false). A control that PASSES proves the +# instrument cannot see a broken stream, so any "no gain" verdict is untrustworthy +# and the caller ABORTS. Mirrors profile-validation-matrix.sh:204-217. The +# GAIN_TEST_CONTROL_* hooks inject a synthetic control so the gate is verifiable +# without CAP_NET_ADMIN. Sets CONTROL_PASS; returns 0 when correctly failed, 1 else. +# True (0) when any falsifiability self-test hook is set, so the control gate can be +# exercised without CAP_NET_ADMIN. PORT_MISMATCH_PASS_OVERRIDE is an alias for +# GAIN_TEST_CONTROL_PASS (force the control to wrongly pass -> abort). +control_selftest_hook() { + [[ "${GAIN_TEST_CONTROL_PASS:-0}" == "1" || "${PORT_MISMATCH_PASS_OVERRIDE:-0}" == "1" \ + || "${GAIN_TEST_CONTROL_FAIL:-0}" == "1" ]] +} + +run_falsifiability_control() { # out_dir label + local cdir="$1/control" label="${2:-stage}" cj + mkdir -p "$cdir"; cj="${cdir}/result.json" + if [[ "${GAIN_TEST_CONTROL_PASS:-0}" == "1" || "${PORT_MISMATCH_PASS_OVERRIDE:-0}" == "1" ]]; then + printf '{"scenario":"reorder-stress","pass":true,"_forced":"GAIN_TEST_CONTROL_PASS"}\n' > "$cj" + elif [[ "${GAIN_TEST_CONTROL_FAIL:-0}" == "1" ]]; then + printf '{"scenario":"reorder-stress","pass":false,"_forced":"GAIN_TEST_CONTROL_FAIL"}\n' > "$cj" + else + log " ==> falsifiability control (${label}): PORT_MISMATCH=1 run (must NOT pass)" + run_rep "$cj" 1 1 0 "falsifiability-control-${label}" "$(( NETEM_SEED_BASE + 1 ))" \ + "$SENTINEL_STEADY" "$SENTINEL_BURST" 1 || true + fi + CONTROL_PASS="$(jq -r '.pass // false' "$cj" 2>/dev/null || echo false)" + local cbytes; cbytes="$(jq -r '.sink.bytes_received // 0' "$cj" 2>/dev/null || echo 0)" + log " control: pass=${CONTROL_PASS} bytes_received=${cbytes} (expected pass=false)" + [[ "$CONTROL_PASS" == "true" ]] && return 1 || return 0 +} + +# Run the control first and ABORT (exit 2) the stage if the instrument is not +# falsifiable — a passing PORT_MISMATCH control means the campaign cannot be trusted. +gate_falsifiability() { # out_dir label + run_falsifiability_control "$1" "$2" && return 0 + log "FATAL: falsifiability control PASSED — instrument not falsifiable; ABORTING ${2}." + jq -n --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg stage "$2" \ + '{stage:$stage, error:"instrument not falsifiable", + falsifiability_control:{pass:true}, aborted:true, timestamp:$ts}' \ + > "$1/falsifiability-abort.json" + exit 2 +} + +# Capability + sender gate shared by smoke/stage. SKIP (exit 77) — never fabricate. +require_run_env() { + if [[ -z "$SRTLA_SEND_RS_BIN" ]]; then + log "SKIP gain-hunt-matrix: no srtla-send-rs resolvable (set SRTLA_SEND_RS_BIN or put" + log " srtla_send_rs on PATH). The campaign's PRIMARY sender is the Rust fork; refusing" + log " to measure the deprecated C srtla_send as production." + exit 77 + fi + if ! bash "$NETEM_LIB" require >/dev/null 2>&1; then + log "SKIP gain-hunt-matrix: netem unavailable (need CAP_NET_ADMIN: root, sudo, or mapped-root userns)" + exit 77 + fi + resolve_build_dir +} + +resolve_build_dir() { + local d + for d in "$BUILD_DIR" "${REPO_ROOT}/build" "/tmp/srtla-build"; do + [[ -n "$d" ]] || continue + if [[ -x "${d}/srtla_rec" && -x "${d}/tests/compat/srt-sink/srt-sink" ]]; then + BUILD_DIR="$d"; return 0 + fi + done + die "no usable build dir (need srtla_rec + srt-sink). Build with: + cmake -B build -DBUILD_COMPAT_TESTS=ON && cmake --build build -j" +} + +# Print candidate-vs-baseline metrics for one finished cell (jq; no python needed). +summarize_cell() { # dir + local dir="$1" + command -v jq >/dev/null 2>&1 || { log " (jq absent — skipping numeric summary)"; return 0; } + local cf bf cg bg cwa bwa crwa brwa cdisc bdisc + cf="$(ls "${dir}/candidate"/rep-*.json 2>/dev/null | head -1)" + bf="$(ls "${dir}/baseline"/rep-*.json 2>/dev/null | head -1)" + [[ -f "$cf" && -f "$bf" ]] || { log " (no rep JSON produced)"; return 0; } + cg="$(jq -r '.metrics.goodput_bps // 0' "$cf")"; bg="$(jq -r '.metrics.goodput_bps // 0' "$bf")" + cwa="$(jq -r '.metrics.wire_amp // 0' "$cf")"; bwa="$(jq -r '.metrics.wire_amp // 0' "$bf")" + crwa="$(jq -r '.metrics.reverse_wire_amp // 0' "$cf")" + brwa="$(jq -r '.metrics.reverse_wire_amp // 0' "$bf")" + cdisc="$(jq -r '.sink.disconnects // -1' "$cf")"; bdisc="$(jq -r '.sink.disconnects // -1' "$bf")" + log "" + log " ---- cell metrics (candidate / baseline) ----" + log " goodput_bps : ${cg} / ${bg}" + log " wire_amp : ${cwa} / ${bwa}" + log " reverse_wire_amp : ${crwa} / ${brwa}" + log " disconnects : ${cdisc} / ${bdisc}" + # The deliverable is VISIBILITY: the reverse channel is now metered and resolves + # the two recipes (B3/O1 — periodic-NAK's reverse cost is no longer invisible, so + # a recipe cannot false-promote on forward amplification alone). The direction is + # an empirical result, not an assertion: in this SRTLA topology the reverse + # channel is dominated by per-packet broadcast ACKs, so NAK-OFF (which retransmits + # more, lacking precise NAKs) often costs MORE reverse than NAK-ON. + if awk -v c="$crwa" -v b="$brwa" 'BEGIN{exit !(c!=b)}'; then + log " -> reverse channel VISIBLE and distinguishes the recipes (candidate ${crwa} vs baseline ${brwa})." + awk -v c="$crwa" -v b="$brwa" 'BEGIN{ printf " -> NAK-on reverse cost is %s the NAK-off baseline (empirical).\n", (c>b?"ABOVE":"AT/BELOW") }' >&2 + else + log " NOTE: candidate and baseline reverse_wire_amp coincide (${crwa}); rerun with more reps/loss." + fi +} + +# --------------------------------------------------------------------------- # +# Two-stage driver: SCREEN (cheap wide sweep) -> DEEP (full-power selected set) # +# with the anti-false-NULL rescue. See docs/GAIN-HUNT-PROTOCOL.md §3-4. # +# --------------------------------------------------------------------------- # + +print_screen_plan() { # (dry; no privilege, nothing executed) + build_candidates + log "gain-hunt screen PLAN (dry — nothing executed):" + log " reps=${SCREEN_REPS} grid steady={${SCREEN_STEADY[*]}} burst={${SCREEN_BURST[*]}} | ${#CANDIDATES[@]} candidates" + local t frz nak fec st bu lbl tag + for t in "${CANDIDATES[@]}"; do + IFS=: read -r frz nak fec <<<"$t" + for st in "${SCREEN_STEADY[@]}"; do + for bu in "${SCREEN_BURST[@]}"; do + lbl="$(adv_cell_label "$frz" "$nak" "$fec" "$st" "$bu")"; tag="" + [[ "$st" == "$SENTINEL_STEADY" && "$bu" == "$SENTINEL_BURST" ]] && tag=" [SENTINEL]" + log " ${lbl}${tag}" + done + done + done +} + +summarize_screen() { # screen_dir out_root + python3 - "$1" "$2" <<'PY' +import json, os, sys, glob +sdir, outroot = sys.argv[1], sys.argv[2] +GAIN_GOODPUT, GAIN_LATEDROP = 1.03, 0.80 + +def median(xs): + xs = sorted(xs); n = len(xs) + if n == 0: return 0.0 + m = n // 2 + return float(xs[m]) if n % 2 else (xs[m-1] + xs[m]) / 2.0 + +def field(d, k, default): + if k in d: return d[k] + for sub in ("metrics", "sink"): + s = d.get(sub) or {} + if k in s: return s[k] + return default + +def load(d): + out = [] + for f in sorted(glob.glob(os.path.join(d, "rep-*.json"))): + try: + out.append(json.load(open(f))) + except (OSError, ValueError): + out.append({}) + return out + +manifest = json.load(open(os.path.join(sdir, "manifest.json"))) +cells, survivors = {}, [] +for m in manifest: + cell = m["cell"]; cdir = os.path.join(sdir, cell) + C = load(os.path.join(cdir, "candidate")); B = load(os.path.join(cdir, "baseline")) + cg = [float(field(r, "goodput_bps", 0) or 0) for r in C] + bg = [float(field(r, "goodput_bps", 0) or 0) for r in B] + cd = [float(field(r, "pkt_rcv_drop", 0) or 0) for r in C] + bd = [float(field(r, "pkt_rcv_drop", 0) or 0) for r in B] + mcg, mbg, mcd, mbd = median(cg), median(bg), median(cd), median(bd) + goodput_ratio = (mcg / mbg) if mbg > 0 else 0.0 + latedrop_ratio = (mcd / mbd) if mbd > 0 else 1.0 + effect = max(goodput_ratio - 1.0, 1.0 - latedrop_ratio) + disc_ok = len(C) > 0 and all(int(field(r, "disconnects", 0) or 0) == 0 for r in C) + sync_ok = len(C) > 0 and all(int(field(r, "ts_sync_errors", 0) or 0) == 0 for r in C) + hard_gate_ok = disc_ok and sync_ok + directional = (mbg > 0 and goodput_ratio >= GAIN_GOODPUT) or \ + (mbd > 0 and latedrop_ratio <= GAIN_LATEDROP and (mbg <= 0 or mcg >= 0.99 * mbg)) + survivor = bool(directional and hard_gate_ok) + if survivor: survivors.append(cell) + rec = dict(m) + rec.update({"n_candidate": len(C), "n_baseline": len(B), + "goodput_ratio": goodput_ratio, "latedrop_ratio": latedrop_ratio, + "effect_size": effect, "hard_gate_ok": hard_gate_ok, + "directional": bool(directional), "survivor": survivor, + "medians": {"goodput_c": mcg, "goodput_b": mbg, + "pkt_rcv_drop_c": mcd, "pkt_rcv_drop_b": mbd}}) + cells[cell] = rec + +json.dump({"stage": "screen", "n_cells": len(cells), "survivors": survivors, "cells": cells}, + open(os.path.join(outroot, "screen-results.json"), "w"), indent=2, sort_keys=True) +json.dump({"survivors": survivors, "n": len(survivors)}, + open(os.path.join(outroot, "survivors.json"), "w"), indent=2, sort_keys=True) +sys.stderr.write("screen: %d cells, %d survivor(s): %s\n" + % (len(cells), len(survivors), ", ".join(survivors) or "(none)")) +PY +} + +do_screen() { + local sdir="${GAIN_OUT}/screen" + mkdir -p "$GAIN_OUT"; rm -rf "$sdir"; mkdir -p "$sdir" + RUN_LOG="${sdir}/run.log"; : > "$RUN_LOG" + build_candidates + log "================ gain-hunt screen stage ================" + log " out: ${sdir} | reps=${SCREEN_REPS} | grid steady={${SCREEN_STEADY[*]}} burst={${SCREEN_BURST[*]}}" + log " candidates=${#CANDIDATES[@]} | sentinel=(s${SENTINEL_STEADY},b${SENTINEL_BURST})" + log "========================================================" + gate_falsifiability "$sdir" "screen" + : > "${sdir}/manifest.jsonl" + local t frz nak fec st bu lbl fam sentinel + for t in "${CANDIDATES[@]}"; do + IFS=: read -r frz nak fec <<<"$t" + fam="$(cell_label "$frz" "$nak" "$fec")" + for st in "${SCREEN_STEADY[@]}"; do + for bu in "${SCREEN_BURST[@]}"; do + lbl="$(adv_cell_label "$frz" "$nak" "$fec" "$st" "$bu")" + sentinel=false + [[ "$st" == "$SENTINEL_STEADY" && "$bu" == "$SENTINEL_BURST" ]] && sentinel=true + jq -nc --arg cell "$lbl" --arg family "$fam" \ + --argjson freeze "$frz" --argjson nak "$nak" --argjson fec "$fec" \ + --argjson steady "$st" --argjson burst "$bu" --argjson sentinel "$sentinel" \ + '{cell:$cell,family:$family,freeze:$freeze,nak:$nak,fec:$fec,steady:$steady,burst:$burst,sentinel:$sentinel}' \ + >> "${sdir}/manifest.jsonl" + run_cell "${sdir}/${lbl}" "$frz" "$nak" "$fec" "$lbl" "$st" "$bu" "$SCREEN_REPS" + done + done + done + jq -s '.' "${sdir}/manifest.jsonl" > "${sdir}/manifest.json" + summarize_screen "$sdir" "$GAIN_OUT" + log "" + log " screen-results: ${GAIN_OUT}/screen-results.json" + log " survivors : ${GAIN_OUT}/survivors.json" +} + +# Compute the deep set: survivors ∪ top-K/family (by directional effect) ∪ sentinels +# (one per candidate family, ALWAYS). Writes /deep-manifest.json; needs no +# privilege, so it doubles as the deep --plan dry-run. Survivors/screen evidence are +# optional — with none, survivors are empty and only top-K + sentinels populate deep. +plan_deep() { # ddir + local ddir="$1" + build_candidates + mkdir -p "$ddir" + local t frz nak fec fam_file="${ddir}/families.json" + # Families list to a FILE (not a pipe): a heredoc would override piped stdin (SC2259). + for t in "${CANDIDATES[@]}"; do + IFS=: read -r frz nak fec <<<"$t" + jq -nc --arg family "$(cell_label "$frz" "$nak" "$fec")" \ + --argjson freeze "$frz" --argjson nak "$nak" --argjson fec "$fec" \ + '{family:$family,freeze:$freeze,nak:$nak,fec:$fec}' + done | jq -s '.' > "$fam_file" + python3 - "$ddir" "${GAIN_OUT}/screen-results.json" "${GAIN_OUT}/survivors.json" \ + "$SENTINEL_STEADY" "$SENTINEL_BURST" "$DEEP_TOPK" "$fam_file" <<'PY' +import json, os, sys +ddir, screen_path, surv_path, sent_st, sent_bu, topk, fam_file = sys.argv[1:8] +sent_st, sent_bu, topk = int(sent_st), int(sent_bu), int(topk) +families = json.load(open(fam_file)) +screen = json.load(open(screen_path)) if os.path.isfile(screen_path) else {"cells": {}} +cells = screen.get("cells", {}) +survivors = set() +if os.path.isfile(surv_path): + survivors = set(json.load(open(surv_path)).get("survivors", [])) +else: + survivors = set(screen.get("survivors", [])) + +selected, reasons = {}, {} +def add(env, why): + selected[env["cell"]] = env + reasons.setdefault(env["cell"], set()).add(why) + +KEYS = ("cell", "family", "freeze", "nak", "fec", "steady", "burst", "sentinel") +def from_screen(sc): + return {k: sc[k] for k in KEYS} +def sentinel_env(fam): + return {"cell": "%s-s%d-b%d" % (fam["family"], sent_st, sent_bu), + "family": fam["family"], "freeze": fam["freeze"], "nak": fam["nak"], + "fec": fam["fec"], "steady": sent_st, "burst": sent_bu, "sentinel": True} + +for c in sorted(survivors): + if c in cells: add(from_screen(cells[c]), "survivor") + +by_fam = {} +for c, sc in cells.items(): + by_fam.setdefault(sc.get("family"), []).append(sc) + +for fam in families: + ranked = sorted(by_fam.get(fam["family"], []), + key=lambda x: x.get("effect_size", 0.0), reverse=True) + for sc in ranked[:topk]: + add(from_screen(sc), "top%d" % topk) + add(sentinel_env(fam), "sentinel") # ALWAYS — the anti-false-NULL rescue + +deep = sorted(selected.values(), key=lambda e: e["cell"]) +for e in deep: + e["why"] = ",".join(sorted(reasons[e["cell"]])) +json.dump({"stage": "deep_plan", "n_cells": len(deep), "n_survivors": len(survivors), + "topk": topk, "sentinel": {"steady": sent_st, "burst": sent_bu}, "cells": deep}, + open(os.path.join(ddir, "deep-manifest.json"), "w"), indent=2, sort_keys=True) + +surv_fams = {cells.get(c, {}).get("family") for c in survivors} +sys.stderr.write("\n deep set = survivors(%d) ∪ top-%d/family ∪ sentinels -> %d cells\n" + % (len(survivors), topk, len(deep))) +for e in deep: + sys.stderr.write(" %-24s %s [%s]\n" + % (e["cell"], "SENTINEL" if e["sentinel"] else " ", e["why"])) +rejected = [f["family"] for f in families if f["family"] not in surv_fams] +sys.stderr.write(" screen-rejected families still deep-tested via sentinel/top-K: %s\n" + % (", ".join(rejected) or "(none)")) +PY +} + +emit_verdict() { # ddir + local ddir="$1" vraw="${ddir}/analyze-verdict.json" + bash "${BASH_SOURCE[0]}" --analyze "$ddir" > "$vraw" 2> "${ddir}/analyze.log" || true + python3 - "$ddir" "$vraw" "$GAIN_OUT" "$GAIN_FEC_FILTER" "$LOSSMAXTTL_FIXED" <<'PY' +import json, os, sys +ddir, vraw, gain_out, fec_filter, lossmaxttl = sys.argv[1:6] +try: + v = json.load(open(vraw)) +except (OSError, ValueError): + v = {"verdict": "error", "promoted": False, "cells": {}, "reason": "analyze_no_output"} +cells = v.get("cells", {}) +mani = json.load(open(os.path.join(ddir, "deep-manifest.json"))) +meta = {c["cell"]: c for c in mani.get("cells", [])} +gain_cells = v.get("real_gain_cells", []) +all_no_regr = bool(cells) and all(cr.get("no_regression", False) for cr in cells.values()) + +fam_ev = {} +for cell in gain_cells: + m = meta.get(cell, {}); fam = m.get("family", cell) + info = fam_ev.setdefault(fam, {"freeze": m.get("freeze"), "nak": m.get("nak"), + "fec": m.get("fec"), "evidence_cells": []}) + info["evidence_cells"].append(cell) + +promoted = [] +if gain_cells and all_no_regr: + for fam, info in sorted(fam_ev.items()): + promoted.append({ + "combo": fam, + "srt_flags": "REORDERFREEZE=%s NAKREPORT=%s LOSSMAXTTL=%s" + % (info["freeze"], info["nak"], lossmaxttl), + "caller_packetfilter": (fec_filter if info["fec"] == 1 else None), + "nak": info["nak"], "freeze": info["freeze"], + "evidence_cells": sorted(info["evidence_cells"])}) + +out = {"stage": "deep", "alpha": v.get("alpha", 0.05), "n_cells": len(cells), + "falsifiability_control": {"pass": False, "verified": True}} +if promoted: + out["verdict"], out["promoted"] = "PROMOTED", promoted +else: + out["verdict"], out["promoted"] = "NULL", [] + out["reason"] = v.get("reason", "no_promotable_candidate_in_full_deep_set_incl_sentinels") + +json.dump(out, open(os.path.join(gain_out, "verdict.json"), "w"), indent=2, sort_keys=True) +sys.stderr.write("VERDICT: %s (%d promoted)\n" % (out["verdict"], len(out["promoted"]))) +PY +} + +do_deep() { + local ddir="${GAIN_OUT}/deep" + mkdir -p "$GAIN_OUT"; rm -rf "$ddir"; mkdir -p "$ddir" + RUN_LOG="${ddir}/run.log"; : > "$RUN_LOG" + log "================ gain-hunt deep stage ================" + log " out: ${ddir} | reps=${DEEP_REPS} | top-K=${DEEP_TOPK} | sentinel=(s${SENTINEL_STEADY},b${SENTINEL_BURST})" + log " deep set = screen-survivors ∪ top-K/family ∪ sentinels (sentinels ALWAYS in)" + log "======================================================" + gate_falsifiability "$ddir" "deep" + plan_deep "$ddir" + local count i cell frz nak fec st bu + count="$(jq -r '.cells | length' "${ddir}/deep-manifest.json")" + log " deep set: ${count} cell(s) at ${DEEP_REPS} reps each." + for (( i=0; i survivors" + log " --stage deep stage 2: survivors UNION top-K UNION sentinels, reps=10 -> verdict.json" + log " --stage --plan print the planned cell set (no privilege, nothing run)" + log " --analyze

apply the §2 decision-rule stats (exact Mann-Whitney U," + log " Holm-Bonferroni) to measured paired evidence -> verdict JSON" log " --help the decision rule + candidate matrix in full" - log " --claim-gain REFUSED until the campaign is implemented (falsifiable)" + log " --claim-gain REFUSED until the campaign is run (falsifiable)" exit 0 ;; dry-run) - log "gain-hunt-matrix: DRY RUN — planned campaign (nothing is executed)." + log "gain-hunt-matrix: DRY RUN — the FEC×NAK×FREEZE matrix (nothing is executed)." log "" print_matrix log "" - log " per cell, the campaign WOULD:" - log " 1. build/pin the candidate's FEC libsrt geometry (arq:onreq)" - log " 2. run scenarios/reorder-stress.sh paired/alternating (candidate vs" - log " BellaBox-parity baseline), shared per-rep NETEM_SEED, N reps each" - log " 3. collect goodput_bps, pkt_rcv_drop, ts_*_errors, wire_amp, disconnects" - log " 4. apply the pre-registered 'real gain + no regression' rule with a" - log " Holm-Bonferroni correction across cells (see header / protocol doc)" - log "" - log " VERDICT POLICY: a mixture earns a catalog button ONLY with REAL GAIN in" - log " >=1 cell AND NO REGRESSION in EVERY cell. Ties keep the baseline." - log " arq:never (pure FEC) is a banned control row, never a catalog candidate." + log " per candidate cell, the campaign drives scenarios/reorder-stress.sh paired/" + log " alternating (candidate vs baseline B), shared per-rep NETEM_SEED, collecting" + log " goodput_bps, pkt_rcv_drop, ts_sync_errors, ts_cc_errors, wire_amp," + log " reverse_wire_amp, disconnects; then applies the §2 rule (Holm-Bonferroni)." log "" + log " sender: ${SRTLA_SEND_RS_BIN:-} (srtla-send-rs, primary)" + log " two-stage: --stage screen (reps=4, reduced grid) -> survivors; --stage deep (reps=10)" + log " deep set = survivors UNION top-K(${DEEP_TOPK})/family UNION sentinels (s${SENTINEL_STEADY},b${SENTINEL_BURST}, ALWAYS) -> verdict.json" log " protocol: ${PROTOCOL_DOC}" - log " NOTE: campaign execution is intentionally NOT implemented (R&D track)." + exit 0 + ;; + + smoke) + # Self-test hooks exercise the falsifiability gate without CAP_NET_ADMIN; the + # real path needs a sender + netem (require_run_env SKIPs exit 77 without them). + control_selftest_hook || require_run_env + rm -rf "${RESULTS_DIR:?}/smoke"; mkdir -p "${RESULTS_DIR}/smoke" + RUN_LOG="${RESULTS_DIR}/smoke/run.log"; : > "$RUN_LOG" + # One NAK-axis cell: candidate (freeze1,nak1,fec-off) vs baseline (freeze1,nak0, + # fec-off). Only NAK differs, so the reverse-channel metric isolates NAK's cost. + # A little steady loss guarantees real NAK traffic on the reverse path. + GAIN_STEADY_LOSS_PCT="${GAIN_STEADY_LOSS_PCT:-3}" + log "================ gain-hunt-matrix --smoke ================" + log " build dir: ${BUILD_DIR} | sender: ${SRTLA_SEND_RS_BIN}" + log " reps=${REPS} phase=${PHASE_SEC}s bitrate=${BITRATE_KBPS}k steady_loss=${GAIN_STEADY_LOSS_PCT}%" + log " falsifiability control runs FIRST (PORT_MISMATCH=1 must fail), then 1 cell" + log " cell: candidate f1-n1-plain (NAK-on) vs baseline f1-n0-plain (Classic)" + log "=========================================================" + # Part 6: control precedes the real arm. A control that PASSES means the + # instrument cannot see a broken stream -> SKIP (exit 77), never a false pass. + if ! run_falsifiability_control "${RESULTS_DIR}/smoke" "smoke"; then + log "SKIP gain-hunt-matrix --smoke: falsifiability control PASSED (instrument not falsifiable)" + exit 77 + fi + run_cell "${RESULTS_DIR}/smoke/nak-on" 1 1 0 "smoke-freeze+nak-on" "$GAIN_STEADY_LOSS_PCT" "" "$REPS" + summarize_cell "${RESULTS_DIR}/smoke/nak-on" + log "" + log " per-rep JSON: ${RESULTS_DIR}/smoke/nak-on/{candidate,baseline}/rep-*.json" + log " control : ${RESULTS_DIR}/smoke/control/result.json (pass=${CONTROL_PASS})" + log " run log : ${RUN_LOG}" + # Smoke passes if both arms produced a result.json with metrics (instrument ran). + cand="$(ls "${RESULTS_DIR}/smoke/nak-on/candidate"/rep-*.json 2>/dev/null | head -1)" + base="$(ls "${RESULTS_DIR}/smoke/nak-on/baseline"/rep-*.json 2>/dev/null | head -1)" + # Pass = control correctly failed, paired result.json present, the Rust fork + # actually ran (sender_kind=rust), and the NAK-on (candidate) run carries a + # NON-ZERO reverse_wire_bytes — the reverse channel is metered (the B3/O1 deliverable). + if [[ "$CONTROL_PASS" == "false" && -s "$cand" && -s "$base" ]] \ + && jq -e '.config.sender_kind == "rust"' "$cand" >/dev/null 2>&1 \ + && jq -e '.metrics.reverse_wire_amp != null' "$base" >/dev/null 2>&1 \ + && jq -e '(.metrics.reverse_wire_bytes // 0) > 0' "$cand" >/dev/null 2>&1; then + log "SMOKE OK (control pass:false; reverse channel metered; sender=rust)"; exit 0 + fi + log "SMOKE FAILED (control not falsified, missing paired result, non-rust sender, or zero reverse_wire_bytes)"; exit 1 + ;; + + stage) + [[ "$STAGE" == "screen" || "$STAGE" == "deep" ]] || die "--stage must be screen|deep (got '$STAGE')" + mkdir -p "$GAIN_OUT" + # --plan: compute and print the cell set with NO privilege and NO execution. + # The deep plan lists the sentinel cells for EVERY family — including ones the + # screen rejected — proving the anti-false-NULL rescue is wired (Oracle O4). + if [[ "$PLAN" == "1" ]]; then + if [[ "$STAGE" == "screen" ]]; then + print_screen_plan + else + plan_deep "${GAIN_OUT}/deep" + log "" + log " dry plan only — no cells run. deep-manifest.json: ${GAIN_OUT}/deep/deep-manifest.json" + fi + exit 0 + fi + # Real run needs CAP_NET_ADMIN + a resolvable srtla-send-rs (else SKIP exit 77). + # A self-test hook skips the privilege gate so the falsifiability ABORT (exit 2) + # is verifiable without netem. Each stage runs the control FIRST and ABORTS if it + # is not falsifiable, before any candidate cell is measured. + control_selftest_hook || require_run_env + if [[ "$STAGE" == "screen" ]]; then do_screen; else do_deep; fi + exit 0 + ;; + + analyze) + # §2 decision-rule statistics over ALREADY-MEASURED paired evidence. This COMPUTES + # a verdict from supplied per-rep results — it is the stats layer only, not the + # measurement driver, needs no CAP_NET_ADMIN and no sender, and is stdlib-only + # (scipy is absent on this box). + [[ -e "$ANALYZE_PATH" ]] || die "--analyze: path not found: ${ANALYZE_PATH}" + python3 - "$ANALYZE_PATH" <<'PY' +import json, math, os, sys, glob +from collections import Counter + +ALPHA = 0.05 +GAIN_GOODPUT_RATIO = 1.03 # real gain: median goodput(C) >= 1.03x median goodput(B) +GAIN_LATEDROP_RATIO = 0.80 # real gain: median pkt_rcv_drop(C) <= 0.80x median(B) +GOODPUT_FLOOR = 0.99 # no-regression + late-drop-win non-inferiority floor +WIRE_AMP_CEIL = 1.10 +REVERSE_AMP_CEIL = 1.10 + +def median(xs): + xs = sorted(xs); n = len(xs) + if n == 0: + return 0.0 + mid = n // 2 + return float(xs[mid]) if n % 2 else (xs[mid - 1] + xs[mid]) / 2.0 + +def p95(xs): + xs = sorted(xs); n = len(xs) + if n == 0: + return 0.0 + idx = min(n - 1, max(0, math.ceil(0.95 * n) - 1)) + return float(xs[idx]) + +def midranks(values): + # 1-based ranks with ties resolved to the average rank (midrank) of the tie group. + order = sorted(range(len(values)), key=lambda i: values[i]) + ranks = [0.0] * len(values) + n = len(values); i = 0 + while i < n: + j = i + while j + 1 < n and values[order[j + 1]] == values[order[i]]: + j += 1 + avg = (i + j + 2) / 2.0 + for k in range(i, j + 1): + ranks[order[k]] = avg + i = j + 1 + return ranks + +def _comb(a, b): + return math.comb(a, b) + +def mann_whitney_u_exact(x, y): + """Exact Mann-Whitney U statistic and two-sided p-value for small n. + + Uses the exact permutation rank-sum distribution for m,n <= 20: a subset-sum + DP over the (doubled, tie-aware) ranks counts, for every way to assign m of the + m+n ranks to x, the resulting rank sum, giving the exact null distribution of U. + Two-sided p = P(U <= min(U, m*n - U)) * 2 (clamped to 1). For m or n > 20 it + falls back to the normal approximation with tie correction (reserved for future + deeper sweeps; the campaign's n=10 always takes the exact path). Returns (U, p) + with U the candidate-arm statistic. + """ + m = len(x); n = len(y) + if m == 0 or n == 0: + return 0.0, 1.0 + combined = list(x) + list(y) + ranks = midranks(combined) + if m > 20 or n > 20: + return _mann_whitney_u_normal(m, n, ranks, combined) + # Double the midranks so every value is an integer (midranks are k/2 multiples). + dr = [int(round(2 * r)) for r in ranks] + Rx2 = sum(dr[:m]) + Ux2 = Rx2 - m * (m + 1) # doubled U for the candidate arm + Uy2 = 2 * m * n - Ux2 + Umin2 = min(Ux2, Uy2) + Ux = Ux2 / 2.0 + total_sum = sum(dr) + # dp[k][s] = number of size-k rank subsets summing to s (doubled units). + dp = [[0] * (total_sum + 1) for _ in range(m + 1)] + dp[0][0] = 1 + for r in dr: + for k in range(m, 0, -1): + row_k = dp[k]; row_k1 = dp[k - 1] + for s in range(total_sum, r - 1, -1): + c = row_k1[s - r] + if c: + row_k[s] += c + total = _comb(m + n, m) + # U <= Umin <=> R2 <= Umin2 + m(m+1) + thr = Umin2 + m * (m + 1) + count = sum(dp[m][s] for s in range(0, min(thr, total_sum) + 1)) + p = min(1.0, 2.0 * count / total) + return Ux, p + +def _mann_whitney_u_normal(m, n, ranks, combined): + N = m + n + Rx = sum(ranks[:m]) + Ux = Rx - m * (m + 1) / 2.0 + mu = m * n / 2.0 + tie = sum(t ** 3 - t for t in Counter(combined).values()) + var = (m * n / 12.0) * ((N + 1) - tie / (N * (N - 1.0))) + if var <= 0: + return Ux, 1.0 + z = (abs(Ux - mu) - 0.5) / math.sqrt(var) + if z < 0: + z = 0.0 + p = 2.0 * (1.0 - 0.5 * (1.0 + math.erf(z / math.sqrt(2.0)))) + return Ux, min(1.0, p) + +def rep_field(d, key, default): + if key in d: + return d[key] + m = d.get("metrics") or {} + if key in m: + return m[key] + s = d.get("sink") or {} + if key in s: + return s[key] + return default + +def norm_rep(d): + return { + "goodput_bps": float(rep_field(d, "goodput_bps", 0) or 0), + "pkt_rcv_drop": float(rep_field(d, "pkt_rcv_drop", 0) or 0), + "ts_sync_errors": int(rep_field(d, "ts_sync_errors", 0) or 0), + "ts_cc_errors": int(rep_field(d, "ts_cc_errors", 0) or 0), + "wire_amp": float(rep_field(d, "wire_amp", 0) or 0), + "reverse_wire_amp": float(rep_field(d, "reverse_wire_amp", 0) or 0), + "disconnects": int(rep_field(d, "disconnects", 0) or 0), + } + +def load_reps_dir(d): + out = [] + for f in sorted(glob.glob(os.path.join(d, "rep-*.json"))): + try: + with open(f) as fh: + out.append(norm_rep(json.load(fh))) + except (OSError, ValueError): + out.append(norm_rep({})) + return out + +def load_cells(path): + if os.path.isfile(path): + with open(path) as fh: + doc = json.load(fh) + cid = doc.get("candidate_id", "candidate") + if "cells" in doc: + cells = {name: {"candidate": [norm_rep(r) for r in c.get("candidate", [])], + "baseline": [norm_rep(r) for r in c.get("baseline", [])]} + for name, c in doc["cells"].items()} + else: + cells = {"cell": {"candidate": [norm_rep(r) for r in doc.get("candidate", [])], + "baseline": [norm_rep(r) for r in doc.get("baseline", [])]}} + return cid, cells + cand = os.path.join(path, "candidate"); base = os.path.join(path, "baseline") + if os.path.isdir(cand) and os.path.isdir(base): + return os.path.basename(os.path.normpath(path)) or "candidate", \ + {"cell": {"candidate": load_reps_dir(cand), "baseline": load_reps_dir(base)}} + cells = {} + for name in sorted(os.listdir(path)): + sub = os.path.join(path, name) + if os.path.isdir(os.path.join(sub, "candidate")) and os.path.isdir(os.path.join(sub, "baseline")): + cells[name] = {"candidate": load_reps_dir(os.path.join(sub, "candidate")), + "baseline": load_reps_dir(os.path.join(sub, "baseline"))} + return "candidate", cells + +def analyze_cell(arms): + C = arms["candidate"]; B = arms["baseline"] + cg = [r["goodput_bps"] for r in C]; bg = [r["goodput_bps"] for r in B] + cd = [r["pkt_rcv_drop"] for r in C]; bd = [r["pkt_rcv_drop"] for r in B] + mcg = median(cg); mbg = median(bg) + mcd = median(cd); mbd = median(bd) + goodput_win = mbg > 0 and mcg >= GAIN_GOODPUT_RATIO * mbg + latedrop_win = (mbd > 0 and mcd <= GAIN_LATEDROP_RATIO * mbd + and (mbg <= 0 or mcg >= GOODPUT_FLOOR * mbg)) + if goodput_win: + win_metric = "goodput_bps"; U, p = mann_whitney_u_exact(cg, bg) + elif latedrop_win: + win_metric = "pkt_rcv_drop"; U, p = mann_whitney_u_exact(cd, bd) + else: + win_metric = "goodput_bps"; U, p = mann_whitney_u_exact(cg, bg) + win = goodput_win or latedrop_win + + mc_wire = median([r["wire_amp"] for r in C]); mb_wire = median([r["wire_amp"] for r in B]) + mc_rev = median([r["reverse_wire_amp"] for r in C]); mb_rev = median([r["reverse_wire_amp"] for r in B]) + g = { + "disconnects_zero": all(r["disconnects"] == 0 for r in C), + "ts_sync_zero": all(r["ts_sync_errors"] == 0 for r in C), + "ts_cc_le_baseline": median([r["ts_cc_errors"] for r in C]) <= median([r["ts_cc_errors"] for r in B]), + "goodput_ge_99pct": mcg >= GOODPUT_FLOOR * mbg if mbg > 0 else True, + "wire_amp_le_110pct": (mc_wire <= WIRE_AMP_CEIL * mb_wire) if mb_wire > 0 else (mc_wire == 0), + "reverse_wire_amp_le_110pct": (mc_rev <= REVERSE_AMP_CEIL * mb_rev) if mb_rev > 0 else (mc_rev == 0), + "p95_late_drop_le_baseline": p95(cd) <= p95(bd), + } + tripped = [k for k, ok in g.items() if not ok] + return { + "n_candidate": len(C), "n_baseline": len(B), + "gain": {"goodput": goodput_win, "late_drop": latedrop_win, "win": win, + "win_metric": win_metric}, + "mwu": {"metric": win_metric, "U": U, "p": p}, + "guardrails": g, "tripped_guardrails": tripped, + "no_regression": not tripped, + "medians": {"goodput_c": mcg, "goodput_b": mbg, "pkt_rcv_drop_c": mcd, + "pkt_rcv_drop_b": mbd, "wire_amp_c": mc_wire, "wire_amp_b": mb_wire, + "reverse_wire_amp_c": mc_rev, "reverse_wire_amp_b": mb_rev}, + } + +def holm(pmap): + # Holm-Bonferroni step-down over the WHOLE family (every cell in the set). + order = sorted(pmap.items(), key=lambda kv: kv[1]) + k = len(order); adj = {}; running = 0.0 + for i, (name, p) in enumerate(order): + running = max(running, (k - i) * p) + adj[name] = min(1.0, running) + return adj + +path = sys.argv[1] +candidate_id, cells = load_cells(path) +if not cells or all((not c["candidate"] or not c["baseline"]) for c in cells.values()): + sys.stderr.write("gain-hunt-matrix --analyze: no paired evidence found at %s\n" % path) + print(json.dumps({"verdict": "error", "promoted": False, "winner": "none", + "reason": "no_paired_evidence", "path": path})) + sys.exit(2) + +cell_results = {name: analyze_cell(arms) for name, arms in cells.items()} +pmap = {name: r["mwu"]["p"] for name, r in cell_results.items()} +adj = holm(pmap) +for name, r in cell_results.items(): + r["holm_adjusted_p"] = adj[name] + r["holm_significant"] = adj[name] < ALPHA + +real_gain_cells = [name for name, r in cell_results.items() + if r["gain"]["win"] and r["holm_significant"] and r["no_regression"]] +all_no_regression = all(r["no_regression"] for r in cell_results.values()) +promoted = bool(real_gain_cells) and all_no_regression + +if promoted: + reason = "real_gain_in_%d_cell(s)_no_regression" % len(real_gain_cells) +elif not all_no_regression: + reason = "regression_in_>=1_cell" +elif not any(r["gain"]["win"] for r in cell_results.values()): + reason = "no_real_gain" +else: + reason = "gain_not_significant_after_holm" + +regression_cells = {name: r["tripped_guardrails"] + for name, r in cell_results.items() if r["tripped_guardrails"]} + +verdict = { + "candidate_id": candidate_id, + "verdict": "promoted" if promoted else "not_promoted", + "promoted": promoted, + "winner": candidate_id if promoted else "none", + "reason": reason, + "alpha": ALPHA, + "n_cells": len(cell_results), + "real_gain_cells": real_gain_cells, + "regression_cells": regression_cells, + "holm_adjusted_p": adj, + "cells": cell_results, +} +print(json.dumps(verdict, indent=2, sort_keys=True)) + +sys.stderr.write("\n========== gain-hunt --analyze (%s) ==========\n" % candidate_id) +for name in sorted(cell_results): + r = cell_results[name] + sys.stderr.write( + " %-18s win=%-5s U=%.1f p=%.3e holm=%.3e no_regr=%-5s tripped=%s\n" % ( + name, str(r["gain"]["win"]), r["mwu"]["U"], r["mwu"]["p"], + r["holm_adjusted_p"], str(r["no_regression"]), + ",".join(r["tripped_guardrails"]) or "-")) +sys.stderr.write("VERDICT: %s (winner=%s; %s)\n" % ( + verdict["verdict"], verdict["winner"], reason)) +sys.stderr.write("================================================\n") +sys.exit(0 if promoted else 1) +PY + exit $? + ;; + + aggregate-deep) + [[ -d "$AGGREGATE_DEEP_DIR" ]] || die "--aggregate-deep: not a directory: ${AGGREGATE_DEEP_DIR}" + agg_out="$(cd -- "$(dirname -- "$AGGREGATE_DEEP_DIR")" && pwd)" + if [[ ! -f "${AGGREGATE_DEEP_DIR}/analyze-verdict.json" ]]; then + log " no analyze-verdict.json — recomputing §2 stats from existing rep evidence" + bash "${BASH_SOURCE[0]}" --analyze "$AGGREGATE_DEEP_DIR" \ + > "${AGGREGATE_DEEP_DIR}/analyze-verdict.json" 2> "${AGGREGATE_DEEP_DIR}/analyze.log" || true + fi + write_deep_results "$AGGREGATE_DEEP_DIR" "$agg_out" + log " deep-results (aggregated): ${agg_out}/deep-results.json" exit 0 ;; claim-gain) - # Falsifiability anchor: a gain CANNOT be claimed by running this stub. If the + # Falsifiability anchor: a gain CANNOT be claimed by running this script. If the # required decision-rule inputs are missing, say which; if present, still refuse - # because no measured evidence exists (the campaign is unimplemented). + # because no aggregated campaign verdict is produced here (the §2 stats are T-A6). missing=() [[ -n "$CANDIDATE" ]] || missing+=("--candidate ") [[ -n "$BASELINE" ]] || missing+=("--baseline ") @@ -187,11 +1171,11 @@ case "$MODE" in exit 3 fi log "gain-hunt-matrix: REFUSED to claim a gain for candidate '${CANDIDATE}'." - log " Inputs were supplied, but the gain-hunt campaign is an UNIMPLEMENTED" - log " R&D track: this stub holds no measured candidate-vs-baseline evidence," - log " so the 'real gain + no regression' rule cannot be evaluated and NO gain" - log " may be asserted. Wire the campaign (docs/GAIN-HUNT-PROTOCOL.md) and run" - log " it under CAP_NET_ADMIN before any mixture is added to the catalog." + log " This driver runs the screen cells but does NOT compute the cross-cell" + log " 'real gain + no regression' verdict (the Holm-Bonferroni stats layer is the" + log " T-A6 deep-stage seam). A gain may be asserted ONLY by that aggregation over" + log " measured paired evidence — never by invoking this script. Run --stage deep" + log " under CAP_NET_ADMIN and let the stats layer decide before adding any recipe." exit 3 ;; esac diff --git a/tests/compat/scenarios/reorder-stress.sh b/tests/compat/scenarios/reorder-stress.sh index a282a3f..44c96e0 100755 --- a/tests/compat/scenarios/reorder-stress.sh +++ b/tests/compat/scenarios/reorder-stress.sh @@ -80,6 +80,43 @@ # the cross-link skew past the built-in 50/150ms band # (slow delay = 150 + RTT_SPREAD_MS). Empty = 150ms. # +# FEC caller passthrough (FEC arms of the gain hunt). Empty default = today's +# ffmpeg-direct SRT caller, byte-identical (Rule E): +# CALLER_PACKETFILTER SRT FEC packet-filter config (must match `^fec,`). When +# set, ffmpeg becomes the MPEG-TS generator and the SRT +# caller is srt-live-transmit carrying `&packetfilter=` +# — because ffmpeg's libsrt wrapper has a fixed option +# allow-list with NO `packetfilter` (appending it makes +# ffmpeg HARD-FAIL "Option not found"), whereas +# srt-live-transmit (libsrt 1.5.5) accepts it. Pair with a +# FEC-accepting sink via SINK_EXTRA_ARGS="--packetfilter fec". +# Requires srt-live-transmit on PATH (absent => SKIP, exit +# 77). Pure FEC (arq:never) is refused — FEC is always an +# arq:onreq hybrid. The negotiated filter the sink accepted +# is echoed to result.json as sink.negotiated_packetfilter. +# +# Sender selection (ADR-003; the campaign's PRIMARY sender is the Rust fork). Empty +# default = today's C srtla_send from the build dir, byte-identical (Rule E): +# SRTLA_SEND_RS_BIN path to the srtla-send-rs binary (the Rust fork sender, +# CLI-compatible: srtla_send ). +# When set+executable it REPLACES the C srtla_send as the +# sender (production-representative arm). Falls back to a +# `srtla_send_rs` on PATH when this is empty. result.json +# records which sender ran as config.sender_kind (c|rust). +# REQUIRE_RS_SENDER 0|1 -> when 1 and no srtla-send-rs is resolvable, SKIP +# cleanly (exit 77) instead of measuring the C srtla_send as +# if it were production. The gain-hunt orchestrator sets this +# so a missing fork binary SKIPs like a missing compat pair +# rather than silently falsifying the campaign. Default unset +# = the C sender is used (existing callers unaffected, Rule E). +# +# Reverse-channel metric (receiver->sender egress, e.g. periodic-NAK cost). The +# veth peer inside the netns has no countable root qdisc by default (noqueue), so a +# prio root is installed inside the netns on $PEERIF and its Sent byte counter is +# read after the run, emitting metrics.reverse_wire_bytes and metrics.reverse_wire_amp +# (reverse_wire_bytes / bytes_received). This makes the NAK-on reverse cost visible +# so a NAK-on profile cannot false-promote on forward-wire amplification alone. +# # The recipe-shorthand maps to the 4 non-FEC receive profiles like so (see the # A/B driver scenarios/profile-validation-matrix.sh): freeze+NAK (Balanced / # Low-Latency / Resilient) = REORDERFREEZE=1 NAKREPORT=1; freeze+NAK-off @@ -137,6 +174,12 @@ PORT_MISMATCH="${PORT_MISMATCH:-}" STEADY_LOSS_PCT="${STEADY_LOSS_PCT:-}" BURST_LOSS_PCT="${BURST_LOSS_PCT:-}" RTT_SPREAD_MS="${RTT_SPREAD_MS:-}" +CALLER_PACKETFILTER="${CALLER_PACKETFILTER:-}" +SRTLA_SEND_RS_BIN="${SRTLA_SEND_RS_BIN:-}" +REQUIRE_RS_SENDER="${REQUIRE_RS_SENDER:-}" +if [[ -z "$SRTLA_SEND_RS_BIN" ]] && command -v srtla_send_rs >/dev/null 2>&1; then + SRTLA_SEND_RS_BIN="$(command -v srtla_send_rs)" +fi while [[ $# -gt 0 ]]; do case "$1" in @@ -155,14 +198,33 @@ done [[ -z "$LOSSMAXTTL" || "$LOSSMAXTTL" =~ ^[0-9]+$ ]] || die "LOSSMAXTTL must be a non-negative integer" [[ -z "$NETEM_SEED" || "$NETEM_SEED" =~ ^[0-9]+$ ]] || die "NETEM_SEED must be a non-negative integer" [[ -z "$PORT_MISMATCH" || "$PORT_MISMATCH" =~ ^[01]$ ]] || die "PORT_MISMATCH must be 0 or 1" +[[ -z "$REQUIRE_RS_SENDER" || "$REQUIRE_RS_SENDER" =~ ^[01]$ ]] || die "REQUIRE_RS_SENDER must be 0 or 1" [[ -z "$STEADY_LOSS_PCT" || "$STEADY_LOSS_PCT" =~ ^[0-9]+(\.[0-9]+)?$ ]] || die "STEADY_LOSS_PCT must be a non-negative number" [[ -z "$BURST_LOSS_PCT" || "$BURST_LOSS_PCT" =~ ^[0-9]+(\.[0-9]+)?$ ]] || die "BURST_LOSS_PCT must be a non-negative number" [[ -z "$RTT_SPREAD_MS" || "$RTT_SPREAD_MS" =~ ^[0-9]+$ ]] || die "RTT_SPREAD_MS must be a non-negative integer" +# CALLER_PACKETFILTER: empty = ffmpeg-direct caller (today's path). Set => FEC arm, +# must be an SRT FEC packet-filter config (`fec,...`); pure FEC (arq:never) is BANNED +# (FEC is always an arq:onreq hybrid here, per docs/RECEIVER-RECONCILIATION.md). +if [[ -n "$CALLER_PACKETFILTER" ]]; then + [[ "$CALLER_PACKETFILTER" =~ ^fec, ]] || die "CALLER_PACKETFILTER must start with 'fec,' (got '$CALLER_PACKETFILTER')" + [[ "$CALLER_PACKETFILTER" == *arq:never* ]] && die "CALLER_PACKETFILTER must not use arq:never (pure FEC is BANNED)" +fi for tool in ffmpeg jq; do command -v "$tool" >/dev/null 2>&1 || die "required tool '$tool' not found in PATH" done +# FEC caller needs srt-live-transmit (ffmpeg's libsrt wrapper has no packetfilter +# option). SKIP-cleanly (exit 77, this scenario's SKIP convention) when the FEC +# arm is requested but the transmitter is absent — falsifiable, never best-effort. +if [[ -n "$CALLER_PACKETFILTER" ]] && ! command -v srt-live-transmit >/dev/null 2>&1; then + log "SKIP reorder-stress: CALLER_PACKETFILTER set but srt-live-transmit not in PATH (FEC caller needs it)" + mkdir -p "$RESULTS_DIR" + printf '{"scenario":"reorder-stress","skipped":true,"reason":"CALLER_PACKETFILTER set but srt-live-transmit absent"}\n' \ + > "${RESULTS_DIR}/result.json" + exit 77 +fi + # Capability gate — SKIP-PRIVILEGED (exit 77, the code netem_require returns and # the sibling netem scenarios use) if we cannot shape the network. if ! netem_require; then @@ -194,6 +256,24 @@ SRT_SINK="${BUILD_DIR}/tests/compat/srt-sink/srt-sink" SRTLA_REC="${BUILD_DIR}/srtla_rec" SRTLA_SEND="${BUILD_DIR}/srtla_send" +# Sender selection (ADR-003). The fork srtla-send-rs is the production sender and +# is CLI-identical to the C srtla_send, so the same invocation drives either. When +# SRTLA_SEND_RS_BIN resolves it becomes the sender; REQUIRE_RS_SENDER=1 makes a +# missing fork a clean SKIP rather than a silent C-sender measurement. +SENDER_BIN="$SRTLA_SEND" +SENDER_KIND="c" +if [[ -n "$SRTLA_SEND_RS_BIN" ]]; then + [[ -x "$SRTLA_SEND_RS_BIN" ]] || die "SRTLA_SEND_RS_BIN '$SRTLA_SEND_RS_BIN' is not executable" + SENDER_BIN="$SRTLA_SEND_RS_BIN" + SENDER_KIND="rust" +elif [[ "$REQUIRE_RS_SENDER" == "1" ]]; then + log "SKIP reorder-stress: REQUIRE_RS_SENDER=1 but no srtla-send-rs resolvable (set SRTLA_SEND_RS_BIN); refusing to measure C srtla_send as production" + mkdir -p "$RESULTS_DIR" + printf '{"scenario":"reorder-stress","skipped":true,"reason":"REQUIRE_RS_SENDER=1 but SRTLA_SEND_RS_BIN unset"}\n' \ + > "${RESULTS_DIR}/result.json" + exit 77 +fi + # Absolutise the loader path so it resolves regardless of srt-sink's cwd (the # caller may pass it relative, e.g. test-results/.../lib). if [[ -n "$SINK_LD_LIBRARY_PATH" ]]; then @@ -251,6 +331,7 @@ rm -rf "$RESULTS_DIR"; mkdir -p "$RESULTS_DIR" RX_LOG="${RESULTS_DIR}/receiver.log" TX_LOG="${RESULTS_DIR}/sender.log" FF_LOG="${RESULTS_DIR}/ffmpeg.log" +SLT_LOG="${RESULTS_DIR}/srt-live-transmit.log" SINK_LOG="${RESULTS_DIR}/sink.log" SINK_JSON="${RESULTS_DIR}/sink.json" IPS_FILE="${RESULTS_DIR}/ips.txt" @@ -309,6 +390,17 @@ qdisc_sent_bytes() { # dev handle(e.g. "1:") -> integer printf '%s' "${n:-0}" } +# Same Sent-bytes read but INSIDE the receiver netns — the reverse channel +# (srtla_rec/srt-sink -> sender: ACK/NAK/keepalive) leaves over $PEERIF's egress, +# which the host-side counter never sees. Reads the prio root installed on $PEERIF. +ns_qdisc_sent_bytes() { # dev handle(e.g. "1:") -> integer + local n + n="$(ip netns exec "$NS" tc -s qdisc show dev "$1" 2>/dev/null | awk -v h="$2" ' + $1=="qdisc" && $3==h {f=1; next} + f && $1=="Sent" {print $2; exit}')" + printf '%s' "${n:-0}" +} + setup_topology() { ip link set lo up 2>/dev/null || true # host-side loopback (needed under a fresh netns / unshare) @@ -324,6 +416,12 @@ setup_topology() { ip netns exec "$NS" ip link set "$PEERIF" up || die "peer link up failed" ip netns exec "$NS" ip link set lo up || true + # Countable root qdisc on the receiver-side egress so the reverse channel + # (ACK/NAK/keepalive back to the sender) has a Sent-bytes counter; veth peers + # default to noqueue with no usable stats. All bands fold to 1:1 (priomap all 0). + ip netns exec "$NS" tc qdisc add dev "$PEERIF" root handle 1: prio bands 3 \ + priomap 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 || die "reverse qdisc failed" + # Per-source egress shaping: classful prio + u32(src) -> two netem bands. # priomap routes all unclassified IP to band 1:3 (passthrough), so only the # two bonded sources are delayed; ARP/control stays fast. @@ -379,7 +477,8 @@ wait_for_marker "$RX_LOG" "srtla_rec is now running" 5 || die "receiver never ca printf '%s\n%s\n' "$SRC_A" "$SRC_B" > "$IPS_FILE" TARGET_SRTLA_PORT="$SRTLA_PORT" [[ "$PORT_MISMATCH" == "1" ]] && TARGET_SRTLA_PORT=$(( SRTLA_PORT + 1 )) # falsifiability control: wrong port => no bytes => MUST fail -RUST_LOG="${RUST_LOG:-info}" "$SRTLA_SEND" "$LOCAL_SRT_PORT" "$RX_IP" "$TARGET_SRTLA_PORT" "$IPS_FILE" >"$TX_LOG" 2>&1 & +log " sender: ${SENDER_KIND} (${SENDER_BIN})" +RUST_LOG="${RUST_LOG:-info}" "$SENDER_BIN" "$LOCAL_SRT_PORT" "$RX_IP" "$TARGET_SRTLA_PORT" "$IPS_FILE" >"$TX_LOG" 2>&1 & TX_PID=$!; track "$TX_PID" sleep 0.6 @@ -392,11 +491,30 @@ SRT_OPTS="mode=caller&transtype=live&latency=${SRT_LATENCY_US}&peerlatency=${SRT # source so the mpeg2video encoder actually emits the requested multi-Mbps target. FF_SIZE=320x240; FF_RATE=25 if [[ "$BITRATE_KBPS" -ge 4000 ]]; then FF_SIZE=1280x720; FF_RATE=30; fi +if [[ -z "$CALLER_PACKETFILTER" ]]; then +# Unset path: ffmpeg is the SRT caller directly. Kept byte-identical to the +# pre-FEC form (Rule E) — see test-results/a2-rule-e-diff.txt. ffmpeg -hide_banner -loglevel warning -re \ -f lavfi -i "testsrc2=size=${FF_SIZE}:rate=${FF_RATE}" -c:v mpeg2video -b:v "${BITRATE_KBPS}k" -f mpegts \ "srt://127.0.0.1:${LOCAL_SRT_PORT}?${SRT_OPTS}" \ >"$FF_LOG" 2>&1 & FF_PID=$!; track "$FF_PID" +else + # FEC path: ffmpeg becomes the MPEG-TS generator (stdout); srt-live-transmit is + # the SRT caller carrying &packetfilter (libsrt 1.5.5, FEC-capable). $! is the + # tail of the pipe (srt-live-transmit); killing it SIGPIPEs ffmpeg on teardown. + # srt-live-transmit URI latency is in MILLISECONDS (ffmpeg's was microseconds), + # and `timeout` is an ffmpeg-only libsrt option — reusing $SRT_OPTS here would + # set an 800s buffer and deliver zero bytes, so build SRT options afresh in ms. + SLT_OPTS="mode=caller&transtype=live&latency=${SRT_LATENCY_MS}&peerlatency=${SRT_LATENCY_MS}&sndbuf=24000000" + ffmpeg -hide_banner -loglevel warning -re \ + -f lavfi -i "testsrc2=size=${FF_SIZE}:rate=${FF_RATE}" -c:v mpeg2video -b:v "${BITRATE_KBPS}k" -f mpegts - \ + 2>"$FF_LOG" \ + | srt-live-transmit -chunk:1316 "file://con" \ + "srt://127.0.0.1:${LOCAL_SRT_PORT}?${SLT_OPTS}&packetfilter=${CALLER_PACKETFILTER}" \ + >"$SLT_LOG" 2>&1 & + FF_PID=$!; track "$FF_PID" +fi handshake=false wait_for_marker "$RX_LOG" "Group registered" 10 && handshake=true @@ -440,12 +558,15 @@ sleep "$PHASE_SEC" reorder_p1="$(qdisc_sent_pkts "$HOSTIF" "10:")" # ... after band_b_pkts="$(qdisc_sent_pkts "$HOSTIF" "20:")" # slow-link total (asymmetry proof) wire_bytes="$(qdisc_sent_bytes "$HOSTIF" "1:")" # total bonded forward-wire egress (root prio) -{ printf '=== after phase ii (phase ii end) ===\n'; tc -s qdisc show dev "$HOSTIF"; } >>"$TC_LOG" 2>&1 +reverse_wire_bytes="$(ns_qdisc_sent_bytes "$PEERIF" "1:")" # receiver->sender egress (reverse channel) +{ printf '=== after phase ii (phase ii end) ===\n'; tc -s qdisc show dev "$HOSTIF"; \ + printf '=== reverse channel (peer egress, in netns) ===\n'; ip netns exec "$NS" tc -s qdisc show dev "$PEERIF"; } >>"$TC_LOG" 2>&1 [[ "$reorder_p0" =~ ^[0-9]+$ ]] || reorder_p0=0 [[ "$reorder_p1" =~ ^[0-9]+$ ]] || reorder_p1=0 [[ "$band_b_pkts" =~ ^[0-9]+$ ]] || band_b_pkts=0 [[ "$wire_bytes" =~ ^[0-9]+$ ]] || wire_bytes=0 +[[ "$reverse_wire_bytes" =~ ^[0-9]+$ ]] || reverse_wire_bytes=0 reorder_pkts=$(( reorder_p1 - reorder_p0 )) [[ "$reorder_pkts" -lt 0 ]] && reorder_pkts=0 @@ -485,11 +606,16 @@ for v in ts_pkts pkt_loss pkt_drop pkt_retr; do [[ "${!v}" =~ ^[0-9]+$ ]] || pri # the same two quantities ADR-002's pre-registered "equal" rule compares. goodput=0; [[ "$duration_s" -gt 0 ]] && goodput=$(( bytes / duration_s )) wire_amp="$(awk -v w="$wire_bytes" -v b="$bytes" 'BEGIN{ printf "%.4f", (b>0 ? w/b : 0) }')" +reverse_wire_amp="$(awk -v w="$reverse_wire_bytes" -v b="$bytes" 'BEGIN{ printf "%.4f", (b>0 ? w/b : 0) }')" # libsrt version straight from the Task-4 srt-sink banner (proves which build ran). libsrt_ver="$(sed -n 's/^srt-sink: libsrt version \([0-9.]*\).*/\1/p' "$SINK_LOG" 2>/dev/null | head -1)" [[ -n "$libsrt_ver" ]] || libsrt_ver="unknown" +# Negotiated SRT packet-filter the sink accepted (non-empty => FEC was negotiated +# end-to-end on the FEC arm; "" when the caller sent plain or the sink cleared it). +negotiated_pf="$(jq -r '.packetfilter // ""' "$SINK_JSON" 2>/dev/null || echo "")" + # --------------------------------------------------------------------------- # # Verdict (DEFAULT condition). A/B/C analysis is NOT decided here (Task 16). # # --------------------------------------------------------------------------- # @@ -523,11 +649,15 @@ jq -n \ --arg reorderfreeze "${REORDERFREEZE:-default}" --arg netem_seed "${NETEM_SEED:-none}" \ --arg steady_loss_pct "${STEADY_LOSS_PCT:-none}" --arg burst_loss_pct "${BURST_LOSS_PCT:-none}" \ --arg rtt_spread_ms "${RTT_SPREAD_MS:-none}" \ + --arg caller_packetfilter "${CALLER_PACKETFILTER:-none}" \ + --arg negotiated_packetfilter "$negotiated_pf" \ --argjson ts_sync_errors "$ts_sync" --argjson ts_cc_errors "$ts_cc" \ --argjson ts_packets "$ts_pkts" --argjson pkt_rcv_loss "$pkt_loss" \ --argjson pkt_rcv_drop "$pkt_drop" --argjson pkt_retrans "$pkt_retr" \ --argjson goodput_bps "$goodput" --argjson wire_bytes "$wire_bytes" \ --argjson wire_amp "$wire_amp" \ + --argjson reverse_wire_bytes "$reverse_wire_bytes" --argjson reverse_wire_amp "$reverse_wire_amp" \ + --arg sender_kind "$SENDER_KIND" --arg sender_bin "$SENDER_BIN" \ --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ '{ scenario:"reorder-stress", pass:$pass, profile:$profile, @@ -539,11 +669,13 @@ jq -n \ nakreport:$nakreport, lossmaxttl:$lossmaxttl, reorderfreeze:$reorderfreeze, netem_seed:$netem_seed, steady_loss_pct:$steady_loss_pct, burst_loss_pct:$burst_loss_pct, - rtt_spread_ms:$rtt_spread_ms}, + rtt_spread_ms:$rtt_spread_ms, caller_packetfilter:$caller_packetfilter, + sender_kind:$sender_kind, sender_bin:$sender_bin}, sink:{bytes_received:$bytes, disconnects:$disconnects, duration_s:$duration_s, libsrt_version:$libsrt, libsrt_path:$sink_libsrt_path, - extra_args:$sink_extra_args}, + extra_args:$sink_extra_args, negotiated_packetfilter:$negotiated_packetfilter}, metrics:{goodput_bps:$goodput_bps, wire_bytes:$wire_bytes, wire_amp:$wire_amp, + reverse_wire_bytes:$reverse_wire_bytes, reverse_wire_amp:$reverse_wire_amp, ts_packets:$ts_packets, ts_sync_errors:$ts_sync_errors, ts_cc_errors:$ts_cc_errors, pkt_rcv_loss:$pkt_rcv_loss, pkt_rcv_drop:$pkt_rcv_drop, pkt_retrans:$pkt_retrans}, @@ -558,7 +690,7 @@ jq -n \ # so `grep -E 'bytes_received=[0-9]+ disconnects=0'` matches on a PASS run). log "" log "reorder-stress: bytes_received=${bytes} disconnects=${disc} duration=${duration_s}s libsrt=${libsrt_ver} reorder_pkts=${reorder_pkts} slow_link_pkts=${band_b_pkts}" -log "reorder-stress[ab]: profile=${PROFILE_LABEL} goodput_bps=${goodput} wire_amp=${wire_amp} ts_sync_errors=${ts_sync} ts_cc_errors=${ts_cc} pkt_rcv_drop=${pkt_drop} pkt_retrans=${pkt_retr}" +log "reorder-stress[ab]: profile=${PROFILE_LABEL} sender=${SENDER_KIND} goodput_bps=${goodput} wire_amp=${wire_amp} reverse_wire_amp=${reverse_wire_amp} ts_sync_errors=${ts_sync} ts_cc_errors=${ts_cc} pkt_rcv_drop=${pkt_drop} pkt_retrans=${pkt_retr}" log "" log "================ reorder-stress summary ================" log " profile=${PROFILE_LABEL} bitrate=${BITRATE_KBPS}k rx_latency=${RX_LATENCY_MS}ms" @@ -568,7 +700,8 @@ log " handshake_ok=${handshake_ok} (both_links_added=${both_links_added})" log " bytes_ok=${bytes_ok} (bytes=${bytes} >= 5000) disc_ok=${disc_ok} (disc=${disc})" log " duration_ok=${duration_ok} (duration=${duration_s}s >= 30)" log " reorder_active=${reorder_active} (configured=${reorder_configured} phase_ii_pkts=${reorder_pkts})" -log " equal-gate signal: goodput_bps=${goodput} wire_amp=${wire_amp} ts_sync=${ts_sync} ts_cc=${ts_cc} pkt_drop=${pkt_drop} pkt_retrans=${pkt_retr}" +log " equal-gate signal: goodput_bps=${goodput} wire_amp=${wire_amp} reverse_wire_amp=${reverse_wire_amp} ts_sync=${ts_sync} ts_cc=${ts_cc} pkt_drop=${pkt_drop} pkt_retrans=${pkt_retr}" +log " reverse channel: reverse_wire_bytes=${reverse_wire_bytes} reverse_wire_amp=${reverse_wire_amp} | sender=${SENDER_KIND}" log " libsrt=${libsrt_ver} loader=${SINK_RESOLVED:-}" log " result: ${RESULT_JSON}" log "=======================================================" diff --git a/tests/compat/srt-sink/main.cpp b/tests/compat/srt-sink/main.cpp index b797192..240517a 100644 --- a/tests/compat/srt-sink/main.cpp +++ b/tests/compat/srt-sink/main.cpp @@ -8,7 +8,9 @@ // {"bytes_received": N, "first_byte_ms": M, "disconnects": D, "duration_ms": T, // "ts_packets": P, "ts_sync_errors": S, "ts_cc_errors": C, // "pkt_rcv_loss": L, "pkt_rcv_drop": D2, "pkt_retrans": R, -// "packetfilter": ""} +// "packetfilter": "", +// "nakreport_readback": NR, "lossmaxttl_readback": LT, +// "reorderfreeze_readback": RF} // // bytes_received total SRT payload bytes delivered by srt_recv // first_byte_ms ms from sink start to the first delivered byte (-1 if none) @@ -100,6 +102,12 @@ struct Result { uint64_t pkt_rcv_drop = 0; uint64_t pkt_retrans = 0; std::string packetfilter; + // Negotiated policy read off the accepted socket (srt_getsockflag), NOT the + // requested values the banner echoes. -1 = the option could not be read (e.g. + // opt id 120 on a stock libsrt), kept distinct from a real 0. + int nakreport_readback = -1; + int lossmaxttl_readback = -1; + int reorderfreeze_readback = -1; }; void usage(const char *argv0) { @@ -217,10 +225,14 @@ bool write_result(const std::string &path, const Result &r) { ", \"ts_cc_errors\": %" PRIu64 ", \"pkt_rcv_loss\": %" PRIu64 ", \"pkt_rcv_drop\": %" PRIu64 ", \"pkt_retrans\": %" PRIu64 ", \"packetfilter\": \"%s\"" + ", \"nakreport_readback\": %d, \"lossmaxttl_readback\": %d" + ", \"reorderfreeze_readback\": %d" "}\n", r.bytes_received, r.first_byte_ms, r.disconnects, r.duration_ms, r.ts_packets, r.ts_sync_errors, r.ts_cc_errors, r.pkt_rcv_loss, - r.pkt_rcv_drop, r.pkt_retrans, r.packetfilter.c_str()); + r.pkt_rcv_drop, r.pkt_retrans, r.packetfilter.c_str(), + r.nakreport_readback, r.lossmaxttl_readback, + r.reorderfreeze_readback); std::fflush(f); std::fclose(f); if (std::rename(tmp.c_str(), path.c_str()) != 0) { @@ -427,8 +439,27 @@ int main(int argc, char **argv) { pflen > 0) { res.packetfilter.assign(pfbuf, static_cast(pflen)); } - std::fprintf(stderr, "srt-sink: accepted SRT caller (packetfilter='%s')\n", - res.packetfilter.c_str()); + + int rb = 0; + int rblen = static_cast(sizeof(rb)); + if (srt_getsockflag(client, SRTO_NAKREPORT, &rb, &rblen) == 0) { + res.nakreport_readback = rb; + } + rblen = static_cast(sizeof(rb)); + if (srt_getsockflag(client, SRTO_LOSSMAXTTL, &rb, &rblen) == 0) { + res.lossmaxttl_readback = rb; + } + rblen = static_cast(sizeof(rb)); + if (srt_getsockflag(client, static_cast(kSrtoReorderFreeze), + &rb, &rblen) == 0) { + res.reorderfreeze_readback = rb; + } + std::fprintf(stderr, + "srt-sink: accepted SRT caller (packetfilter='%s' " + "nakreport_readback=%d lossmaxttl_readback=%d " + "reorderfreeze_readback=%d)\n", + res.packetfilter.c_str(), res.nakreport_readback, + res.lossmaxttl_readback, res.reorderfreeze_readback); continue; }