ci(release): double-verify ML antivirus false positives instead of re-rolling builds - #1340
Merged
Conversation
…-rolling builds
Release run 30464288732 was blocked by the VirusTotal gate: three linux-amd64
binaries flagged 1/62 by Microsoft's Wacatac.B!ml -- fully stripped binaries
(0 symbols, verified on the exact artifacts), the state that scanned clean in
the two previous cycles. Meanwhile a real Defender endpoint (engine
1.1.26060.3008, signatures 1.455.410.0 updated the same day, RTP on) scans
the identical bytes clean. Four cycles of evidence now say the same thing:
this verdict is an unstable ML decision boundary, not a property of the code,
and no build-side lever moves it durably -- stripping, downloader removal and
metadata changes each "worked" only until a later build flipped it back.
So stop treating the flag as buildable-away and verify it honestly instead:
check-virustotal.sh may downgrade BLOCKED to TOLERATED only when ALL hold:
- pre-release version (-rc./-pre/-alpha/-beta); stable releases never
- every failing file flagged by exactly ONE engine
- that engine is Microsoft and the verdict ends in "!ml" (never a
signature name)
- hash-pinned Defender ENDPOINT evidence is attached to the draft release
(defender-endpoint-verification.txt) proving Microsoft's shipping
product, signature-updated at scan time, reports the exact bytes clean
av-endpoint-verify.sh (new) produces that evidence: downloads the draft
assets, scans them on the local Windows VM endpoint, refuses to attest if
RTP is off or Defender itself detects, uploads the hash-pinned result.
The gate prints the exact command when evidence is missing; re-running the
failed verify job does not rebuild, so the bytes stay fixed.
append-vt-notes.sh (new, extracted from inline YAML per venue-parity) then
renders the release-notes table honestly: a tolerated file reads "1/62 ML
false positive, endpoint-verified clean", never "0 detections".
tests/test_vt_gate_tolerance_contract.sh pins all nine decision directions
against a stubbed VT API and release store -- clean pass, stable-never,
missing/stale/DETECTED evidence, signature-named verdict, non-Microsoft
engine, multi-engine -- so the tolerance provably fails closed.
Also: release.yml gains skip_tests for re-releases of an already test-green
tree (build/smoke/soak/verify always run; lint failures still gate via
!cancelled() && !failure()).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
meikis
pushed a commit
to meikis/codebase-memory-mcp
that referenced
this pull request
Jul 29, 2026
The pre-release ML false-positive tolerance (single-engine Microsoft "!ml" verdicts downgradable with Defender endpoint evidence, DeusData#1340) is reverted by owner decision: cbm does not ship binaries carrying a VirusTotal detection, demonstrably false or not. A "trojan" badge on a release asset is a reputation cost the project is not willing to price in, however good the accompanying evidence. The gate returns to its original form: any detection, by any engine, on any artifact, on any version blocks the release. The endpoint verification tool and the evidence side-channel are removed with it; the notes renderer keeps its extracted-script form but only ever states a verified "0 detections". False positives are resolved upstream instead: verify the bytes on a real Defender endpoint, submit a Microsoft false-positive report for the exact hashes, wait for the detection to clear, then RE-RUN the failed verify job -- which does not rebuild, so the cleared hashes are the shipped hashes. tests/test_vt_gate_zero_tolerance_contract.sh pins the decision: clean passes; 1 malicious and 1 suspicious each block across stable, -rc., -pre and -alpha versions; plus a tripwire for the specific reverted evidence mechanism returning. Loosening this gate again has to consciously delete that contract. The skip_tests dispatch input and the script-extracted notes step survive the revert -- both are orthogonal to gating policy. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Release run 30464288732 cleared every test, soak and smoke leg and was blocked solely by the VirusTotal gate: three linux-amd64 binaries flagged 1/62 by Microsoft
Trojan:Script/Wacatac.B!ml. The binaries are fully stripped (0 symbols, verified on the exact artifacts) — the state that scanned clean in the two previous cycles — while a real Defender endpoint (engine 1.1.26060.3008, signatures 1.455.410.0 updated the same day, RTP on) scans the identical bytes clean, files intact, zero detections.Four release cycles of evidence: this is an unstable ML decision boundary, not a property of the code. Stripping, downloader removal, launcher elimination and metadata changes each "fixed" it only until a later build flipped the verdict again.
This PR replaces build-side whack-a-mole with double verification:
scripts/ci/check-virustotal.shmay downgrade BLOCKED → TOLERATED only when all hold: pre-release version (-rc./-pre/-alpha/-beta) · exactly one flagging engine · that engine is Microsoft · the verdict is a pure ML heuristic (!ml) · and hash-pinned Defender endpoint evidence is attached to the draft release. Stable releases, multi-engine hits, signature-named verdicts, non-Microsoft engines, timeouts: always block.scripts/ci/av-endpoint-verify.sh(new) produces the evidence: downloads the draft assets, signature-updates and scans them on a real Defender endpoint (local Windows VM), refuses to attest if RTP is off or Defender itself detects, uploadsdefender-endpoint-verification.txt. The gate prints the exact command when evidence is missing; re-running the failed verify job does not rebuild, so hashes stay fixed.scripts/ci/append-vt-notes.sh(new, extracted from inline YAML per the venue-parity contract) renders the notes honestly: tolerated files read "1/62release.ymlgains askip_testsdispatch input for re-releases of an already test-green tree (build/smoke/soak/verify always run; lint failures still gate).CI-change scope
skip_testsweakens nothing by default (opt-in dispatch input, off unless explicitly set).gh release download.workflow_dispatchonly).Tests
tests/test_vt_gate_tolerance_contract.shpins all nine decision directions against a stubbed VT API + release store: clean pass · stable-never-tolerates · missing evidence blocks with operator instructions · pattern+evidence tolerates · stale-hash evidence blocks · endpoint-DETECTED blocks · signature-named verdict blocks · non-Microsoft engine blocks · multi-engine blocks.