test(compat): extended receiver FEC×NAK×FREEZE evaluation harness - #39
Merged
Conversation
reorder-stress.sh gains an optional CALLER_PACKETFILTER env for the FEC arms of the gain hunt. ffmpeg's libsrt wrapper has a fixed SRT-option allow-list with no packetfilter (appending it hard-fails Option not found), so the FEC arm routes ffmpeg's MPEG-TS into srt-live-transmit (libsrt 1.5.5, FEC-capable) as the SRT caller carrying the filter -- the same caller as fec-connect-matrix. - Unset (every non-FEC arm): ffmpeg-direct SRT caller, byte-identical to pre-change (Rule E). - Set (must match ^fec,, arq:never refused): ffmpeg | srt-live-transmit with a fresh ms-based SRT options string (srt-live-transmit latency is ms, not ffmpeg's microseconds; timeout is ffmpeg-only). - srt-live-transmit absent + filter set => SKIP exit 77 (this scenario's convention). - result.json records config.caller_packetfilter and the read-back sink.negotiated_packetfilter.
The --stage deep code path wrote deep-results.json as only the manifest
({stage,out_dir,deep_set}); the per-cell U-stat, Holm-corrected p, and
guardrail outcomes lived solely in the co-located deep/analyze-verdict.json,
and no aggregate recorded each cell's NETEM_SEED. T-B3 requires deep-results.json
itself to list every deep cell (survivors, top-K, sentinel) with its per-cell
U-stat + Holm p + guardrails, each recording its NETEM_SEED.
Add write_deep_results(), invoked by do_deep after emit_verdict, merging the
deep manifest, the analyze-verdict stats, and each cell's per-rep NETEM_SEED into
the canonical deep-results.json. analyze-verdict.json is preserved unchanged.
Add --aggregate-deep <dir> to regenerate the aggregate from already-measured
evidence with no privilege and no campaign re-run (deterministic; verdict/stats
unchanged).
…lemented + ran) Three doc surfaces still described the two-stage campaign orchestrator as an unimplemented stub / follow-up seam, which is stale: T-A6 landed the screen->deep structure and Wave B RAN the campaign to a NULL verdict. - gain-hunt-matrix.sh header: drop "filled in by a follow-up effort (T-A6)"; state the two-stage screen->deep structure is implemented and the campaign RAN (NULL). - GAIN-HUNT-PROTOCOL.md: the "Harness scaffold" bullet said orchestrator **stub**... **does not run the campaign** — now describes the fully-wired orchestrator that has run the campaign (verdict NULL), aligned with the Status block below it. - AGENTS.md: "Gain-hunt scaffold (Task 2, pending)" -> "Task 2, DONE", campaign ran with a NULL verdict; points to the up-to-date gain-hunt section above. Also reword the --analyze mode comment to describe it as the stats layer (not the measurement driver) without the ambiguous 'does not run the campaign' phrasing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extended receiver FEC×NAK×FREEZE evaluation harness for the SRTLA compat suite:
tests/compat/lib/probe-fec-capability.sh) — detects whether the receiver's libsrt build can negotiate an FEC packet filter.srt-sinksockopt read-back — the mock SRT receiver now reads back the negotiatedSRTO_NAKREPORT/SRTO_LOSSMAXTTL/SRTO_REORDERFREEZEoff the accepted socket, plus a proxy-fidelity assertion againstirl-srt-server's L1/L2 profile tuples.srt-live-transmit(CALLER_PACKETFILTER), since ffmpeg's libsrt wrapper has nopacketfilteroption.reverse_wire_bytes/reverse_wire_amp) so a recipe cannot false-promote on forwardwire_ampalone.gain-hunt-matrix.sh --analyze) — pure-stdlib exact Mann-Whitney U + Holm-Bonferroni, no scipy.PORT_MISMATCHfalsifiability control.python3/awkdo.Why
Empirically settle whether any FEC / NAK / FREEZE receiver recipe beats the Classic baseline, under a pre-registered decision rule (real gain + no regression), so the operator-facing receiver-capability catalog is populated by evidence rather than intuition. The campaign result was NULL — no combination cleared the gate — and that NULL is recorded as a first-class, falsifiable outcome.
How to verify
cmake -B build && cmake --build build && ctest --test-dir build→ 187/187 greentests/compat/scenarios/gain-hunt-analyze-test.sh→ 4/4 golden fixtures (gain→promoted, regression→rejected, reverse-spam→rejected, tie→none)tests/compat/scenarios/gain-hunt-geometry-lint.sh→ geometry budget lint passes and discriminates (cols:16PASS,cols:10/cols:8REJECT)Risks
Test-harness-only changes — no product or receiver-binary code is touched. The
gain-hunt-matrixscenario is registeredtier: informationalinmatrix.yaml(non-blocking; not iterated byrun-matrix.sh --tier). The campaign's NULL verdict is documented indocs/GAIN-HUNT-PROTOCOL.md.