Zero-result pages can pass as success (YC batch=W24 renders empty)

[QRex-v0]

[CLAUDE-OPUS-5]

Gap

A page can render normally, return HTTP 200, and contain zero results. The verifier (§3.6) checks "required row counts" only when the task states a count. Several eval tasks state none, and for those an empty result is currently indistinguishable from a correct one.

Evidence

The Y Combinator directory URL ycombinator.com/companies?batch=W24 renders a complete, healthy page whose result pane reads "Sorry, no matching companies found". The facet value is no longer W24; the working value is Winter 2024, which returns results normally. Nothing about the failed page looks like a failure.

This is the single most dangerous pattern in the eval set. It produces a well-formed, empty, confidently wrong CSV.

A related trap on the same site: filters apply asynchronously after the initial render, so an extraction that runs too early sees zero rows on a URL that would have worked.

Scope

1. Add a zero-result detector that runs before extraction: known empty-state strings, result-count elements, and a comparison against any total the page reports.
2. Treat zero rows as blocked by default, never success, unless the task explicitly anticipates an empty set.
3. When a filter is expressed as a URL parameter, verify the applied filter chip in the rendered page matches the requested value before extracting.
4. Add a settle-and-recheck step so an early read cannot be mistaken for an empty result.

Acceptance

• The batch=W24 URL produces blocked with a reason naming the unmatched filter value.
• A task with no stated row count still cannot report success on zero rows.

[QRex-v0]

[CLAUDE-OPUS-5] Correction after reading the code.

Filed against the plan before commit 1636f7c. The blunt version of the claim is wrong.

Already done. verifier.py has a required_rows check that downgrades to partial when a CSV was requested and zero rows were extracted. parse.py:42 sets expect_rows whenever a CSV is among the requested artifacts. Zero rows cannot silently reach success on that path.

Still open, and this is the sharper form of the finding.

The Y Combinator failure is not an empty page. It is a page that would have returned rows for the intended query and returned none for a stale filter value. Row-count checking cannot tell those apart, because both look like zero.

• No check compares the requested filter against the filter the page actually applied. The rendered chip reads W24 while the facet the site now accepts is Winter 2024.
• expect_rows is derived from whether a CSV was requested, so a screenshot-only or prose task carries no row expectation at all and no empty-state detection.
• The applied-filter problem generalises beyond emptiness. A filter that silently resolves to something different produces the wrong rows, not zero rows, and nothing would catch that either.

Revised acceptance: verify the applied filter state in the rendered page against the requested value before extracting, independently of row count, and independently of whether the output is a CSV.