[CLAUDE-OPUS-5]
Gap
The output schema has no way to distinguish a value that is genuinely absent at the source from a value the extractor failed to find. IMPLEMENTATION_PLAN.md §2 states "A wrong answer is worse than no answer", which is the right instinct, but there is no field-level contract implementing it.
Evidence
Two eval tasks request columns that are largely or entirely empty at the source.
Task 6 asks for a reviewer per merged pull request. Sampling the ten most recently merged pull requests on openclaw/openclaw via /pulls/{n}/reviews:
| Pull request |
Reviews returned |
| 143292 |
0 |
| 143097 |
0 |
| 143235 |
0 |
| 143300 |
0 |
| 140914 |
0 |
| 142742 |
0 |
| 143298 |
0 |
Every sampled pull request merged with zero reviews and zero requested reviewers. The correct reviewer column is empty for all ten rows, and that emptiness is itself the audit finding.
Task 8 asks for a LinkedIn URL per contributor. Of the top 30 contributors, three are bot accounts and only seven publish a LinkedIn on their GitHub profile.
An agent that silently emits blank cells is indistinguishable from one that crashed mid-column. An agent that fills them by guessing produces exactly the wrong answer §2 forbids.
Scope
- Add a per-cell state to the provenance record in §3.5: observed, absent-at-source, not-found-after-search, or extraction-failed.
- Require that absent-at-source carries evidence of the absence, such as the URL and the empty region inspected.
- Teach the verifier (§3.6) to accept a column of absent-at-source cells as complete, and to reject a column of extraction-failed cells.
- Define the CSV rendering for each state so a reviewer reading the file alone can tell them apart.
Acceptance
- Task 6 completes with an empty reviewer column and passes verification.
- A column that failed to extract cannot reach
success.
[CLAUDE-OPUS-5]
Gap
The output schema has no way to distinguish a value that is genuinely absent at the source from a value the extractor failed to find.
IMPLEMENTATION_PLAN.md§2 states "A wrong answer is worse than no answer", which is the right instinct, but there is no field-level contract implementing it.Evidence
Two eval tasks request columns that are largely or entirely empty at the source.
Task 6 asks for a reviewer per merged pull request. Sampling the ten most recently merged pull requests on
openclaw/openclawvia/pulls/{n}/reviews:Every sampled pull request merged with zero reviews and zero requested reviewers. The correct reviewer column is empty for all ten rows, and that emptiness is itself the audit finding.
Task 8 asks for a LinkedIn URL per contributor. Of the top 30 contributors, three are bot accounts and only seven publish a LinkedIn on their GitHub profile.
An agent that silently emits blank cells is indistinguishable from one that crashed mid-column. An agent that fills them by guessing produces exactly the wrong answer §2 forbids.
Scope
Acceptance
success.