Skip to content

Gate failures must not be dismissed as environmental without base-ref proof #156

Description

@luisherranz

Goal

No agent that runs guardrail gates treats a non-zero gate as passed on the theory that the failure is environmental or pre-existing. A failing gate is resolved one of two ways: fix the underlying cause, or — when the failure stems from a genuinely broken input or environment — raise a blocker. Classifying a failure as ambient (pre-existing or environmental, rather than caused by the current work) requires reproducing the identical failure on the run's base ref; without that evidence the gate counts as failed. This binds every gate-running agent — writers must not commit around such a failure, reviewers must not approve around it.

Context

Observed in the autonomous run that removed the iAPI Mini-Cart feature flag. The code-reviewer ran the e2e gate (35 pass / 17 fail), classified all 17 failures as environmental, and approved — masking a real regression (the Mini-Cart drawer lost its fixed-position overlay CSS and rendered inline at the bottom of the page). The reviewer leaned on an inverted heuristic — "all the failing tests are ones this batch didn't modify → environmental" — which is backwards: a regression is by definition a previously-passing test that now fails.

Although it surfaced at the reviewer, the same fallacy threatens any gate-running agent. Gates are run by code-writer-tdd, code-writer-e2e, code-reviewer, docs-writer, and docs-reviewer (guardrails convention). A writer faces the mirror-image temptation: commit around a failure it believes was "already broken / not my change." Today agents/code-reviewer.md says a non-zero gate is a rejection finding but doesn't forbid the "environmental" rationalization or define proof of "ambient"; the writer profiles (e.g. agents/code-writer-e2e.md) treat every non-zero gate as the writer's own work to fix and don't contemplate a genuinely pre-existing/ambient failure at all. The e2e-writer in this same run did reproduce the original blocker on the base ref — the technique already exists in the project and should become the standard for everyone.

Related to #155 (reviewers should not approve around an unverified hedge): the "these failures are ambient" dismissal is the same family — an unverified hedge on a load-bearing claim. This issue is the concrete, gate-level case, and it spans writers as well as reviewers.

Assumptions / directions to explore

Open — for later phases to confirm or revise, not requirements.

  • The rule must be self-contained in each gate-running agent's profile, since profiles can't reference skill files — duplicated across the relevant profiles (code-writer-tdd, code-writer-e2e, code-reviewer, docs-writer, docs-reviewer). The terminal action differs per role (writer: don't commit; reviewer: don't approve), so the principle is stated and each profile states its own action.
  • Name and forbid the inverted "a failing test the work didn't touch → ambient" heuristic.
  • Define base-ref reproduction (identical failure) as the only accepted evidence for an "ambient" classification, with a blocker as the fallback when that evidence isn't obtained.
  • For a reviewer, base-ref reproduction is an escape hatch, not a mandatory step — the safe default for a suspect failure is a blocker, so the env cost never forces a pass.
  • Standing up the base-ref environment cheaply depends on the worktree + shared-services env work tracked in Initialize worktree #65; this issue stays actionable meanwhile via the blocker fallback.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions