Skip to content

Cross-source enrichment policy conflicts with tasks 5 and 8 #11

Description

@QRex-v0

[CLAUDE-OPUS-5]

Gap

IMPLEMENTATION_PLAN.md §3.2 states: "Search-engine fallback may help discover a target page but cannot replace evidence from the requested source unless the task explicitly permits secondary sources."

Two eval tasks require exactly the substitution that rule forbids, because the requested source does not hold the data.

Evidence

Task 8 asks for a LinkedIn URL for each of the top 30 openclaw/openclaw contributors. Queried through the GitHub GraphQL API:

Measure Count
Contributors returned 30
Bot accounts among them 3
Human accounts with a name on profile 26
Human accounts publishing a LinkedIn 7

GitHub is the requested source and it answers roughly a quarter of the column. The rest is only reachable by leaving the source.

Task 5 is the easier case: founder LinkedIn URLs are published directly on the Y Combinator company page, so no secondary source is needed. Note the page repeats each URL across the avatar, the name link, and the social icon, and also carries a company LinkedIn and a Y Combinator school LinkedIn that must be excluded.

Scope

  1. Add a secondary_sources policy to TaskSpec (§3.1) with values such as forbidden, permitted-with-attribution, or required.
  2. When a secondary source is used, record the discovery query and the confirming evidence, not just the resulting URL.
  3. Define the confirmation bar for identity matching. A name collision on a public profile is a wrong answer, and §2 ranks that below no answer.
  4. Default to forbidden, so enrichment is a deliberate task-level choice.

Acceptance

  • Task 8 either completes with 7 populated cells and 23 marked absent, or completes with attributed secondary-source evidence per populated cell.
  • Neither outcome reports an unattributed guess.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    eval-blockerBlocks an eval task from passingreconFound by live site reconnaissance

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions