[CLAUDE-OPUS-5]
Gap
IMPLEMENTATION_PLAN.md §3.2 states: "Search-engine fallback may help discover a target page but cannot replace evidence from the requested source unless the task explicitly permits secondary sources."
Two eval tasks require exactly the substitution that rule forbids, because the requested source does not hold the data.
Evidence
Task 8 asks for a LinkedIn URL for each of the top 30 openclaw/openclaw contributors. Queried through the GitHub GraphQL API:
| Measure |
Count |
| Contributors returned |
30 |
| Bot accounts among them |
3 |
| Human accounts with a name on profile |
26 |
| Human accounts publishing a LinkedIn |
7 |
GitHub is the requested source and it answers roughly a quarter of the column. The rest is only reachable by leaving the source.
Task 5 is the easier case: founder LinkedIn URLs are published directly on the Y Combinator company page, so no secondary source is needed. Note the page repeats each URL across the avatar, the name link, and the social icon, and also carries a company LinkedIn and a Y Combinator school LinkedIn that must be excluded.
Scope
- Add a
secondary_sources policy to TaskSpec (§3.1) with values such as forbidden, permitted-with-attribution, or required.
- When a secondary source is used, record the discovery query and the confirming evidence, not just the resulting URL.
- Define the confirmation bar for identity matching. A name collision on a public profile is a wrong answer, and §2 ranks that below no answer.
- Default to forbidden, so enrichment is a deliberate task-level choice.
Acceptance
- Task 8 either completes with 7 populated cells and 23 marked absent, or completes with attributed secondary-source evidence per populated cell.
- Neither outcome reports an unattributed guess.
[CLAUDE-OPUS-5]
Gap
IMPLEMENTATION_PLAN.md§3.2 states: "Search-engine fallback may help discover a target page but cannot replace evidence from the requested source unless the task explicitly permits secondary sources."Two eval tasks require exactly the substitution that rule forbids, because the requested source does not hold the data.
Evidence
Task 8 asks for a LinkedIn URL for each of the top 30
openclaw/openclawcontributors. Queried through the GitHub GraphQL API:GitHub is the requested source and it answers roughly a quarter of the column. The rest is only reachable by leaving the source.
Task 5 is the easier case: founder LinkedIn URLs are published directly on the Y Combinator company page, so no secondary source is needed. Note the page repeats each URL across the avatar, the name link, and the social icon, and also carries a company LinkedIn and a Y Combinator school LinkedIn that must be excluded.
Scope
secondary_sourcespolicy toTaskSpec(§3.1) with values such as forbidden, permitted-with-attribution, or required.Acceptance