feat(testing): bake order-sensitive determinism check into fill - #906
Merged
tcoratger merged 1 commit intoJun 10, 2026
Merged
Conversation
The order_sensitive marker docstring claimed the determinism check "generates this vector twice and diffs the output", but no such logic existed in the plugin. The only real gate lived in a justfile recipe invoked by CI, so a plain `uv run fill` never verified determinism and contributors relied on CI. Bake the two-seed check into the fill command itself. After a successful fill, regenerate the order_sensitive subset under PYTHONHASHSEED=1 and =2 in throwaway directories and byte-diff them. The mocked prover is forced so proof bytes stay deterministic, and a single process pins each seed cleanly. A difference fails the command and lists the offending fixtures. Add --no-check-determinism to opt out for fast local iteration. Drop the now-redundant CI step: `just fill-ci` runs fill, which performs the identical subset check by default. Repurpose the fill-determinism recipe as the standalone, wide-scope audit the baked-in check cannot give: it runs without a full fill and can cover the whole tree to catch a filler that should be marked but is not. Pass --no-check-determinism in the recipe so its own fill calls do not nest the per-fill gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The
order_sensitivemarker docstring claimed "the determinism check generates this vector twice and diffs the output", but no such logic existed in the pytest plugin — it only registered the marker. The only real gate lived in thefill-determinismjustfile recipe, invoked once by CI.Consequences:
uv run fillnever verified determinism.The hazard being guarded: Python randomizes
str/byteshashing per process (PYTHONHASHSEED), soset/dict-iteration order varies run-to-run. A vector whose emitted bytes depend on that order is not reproducible across clients.What this does
Bake the two-seed check into
fill. After a successful fill, the command regenerates theorder_sensitivesubset underPYTHONHASHSEED=1and=2into throwaway directories and byte-diffs them. A difference fails the command and lists the offending fixtures.PYTHONHASHSEEDis frozen per interpreter).--no-check-determinismopts out for fast local iteration.Drop the redundant CI step.
just fill-cirunsfill, which now performs the identical subset check by default.Repurpose the
fill-determinismrecipe as the standalone, wide-scope audit the baked-in check cannot provide: it runs without a full fill and can cover the whole tree (just fill-determinism tests/consensus) to catch a filler that should be markedorder_sensitivebut isn't — the safety net for the opt-in marker model's blind spot. Itsfillcalls now pass--no-check-determinismso the recipe (which is the check) doesn't nest the per-fill gate.Net result
fillcheckjust fill-determinismThe determinism gate no longer depends on CI. No emitted vectors change; the existing emission paths were already deterministic by construction (the coverage picker tie-breaks by encoded bytes; the builders use insertion-ordered
dict/list).Testing
just checkpasses (lint, format, ty, codespell, mdformat).uv run fill ... -m order_sensitiveruns the main fill plus two seeded re-runs, all byte-identical; the check reports pass.--no-check-determinismskips the check and the repurposed recipe runs exactly twofillinvocations (no nesting).🤖 Generated with Claude Code