Add transfer-evidence cards to separate reusable improvements from task-specific residue

[sunghunkwag]

Motivation

SIA already has a clear self-improvement loop: run a target agent, evaluate it, write results.json, inject execution/evaluation context into the feedback prompt, and ask the feedback agent to produce the next generation.

The current architecture is strong, but I think one boundary is worth making explicit: separate reusable improvements from task-specific residue before the next generation consumes feedback.

Relevant paths:

• sia/orchestrator.py::run_generation(...) runs the target agent, runs evaluation, adds generation context, then builds the feedback context for the next generation.
• sia/orchestrator.py::_build_feedback_context(...) injects execution logs and raw results.json into the feedback prompt.
• sia/context_manager.py::add_generation(...) appends generation metrics, insights from improvement.md, and LLM summaries to context.md.
• sia/prompts.py tells the feedback agent to build on successful patterns, avoid failed approaches, and avoid task-specific optimization.
• tests/test_feedback_context_golden.py and the prompt golden files already lock this feedback/context surface, which makes it a good place to add a structured contract.

The prompt currently asks the feedback agent to infer what is reusable. That can work, but it leaves a self-improvement failure mode: the next generation may consume local reward gains as if they were generalizable agent improvements.

Failure Mode

A generation can improve the benchmark score while still teaching the wrong lesson:

• an improvement exploits a task-specific artifact in data/public/;
• results.json improves because formatting changed, not because reasoning improved;
• a trajectory-level fix works for one task family but should not be carried into other task types;
• improvement.md describes a broad lesson even though the evidence only supports a narrow claim.

SIA already protects private evaluation data and tells the feedback agent not to overfit. The missing piece is a machine-readable transfer/residue record that the feedback agent must consume.

Proposal: Transfer Evidence Card

After each generation, write a compact transfer_evidence.json next to results.json:

@dataclass
class TransferEvidenceCard:
    generation: int
    accepted_for_reuse: bool
    score_delta: float | None
    evaluator_status: Literal["passed", "failed", "missing", "error"]
    reusable_changes: list[str]
    task_specific_residue: list[str]
    unsupported_claims: list[str]
    negative_probe_hits: int
    claim_boundary: str

Then _build_feedback_context(...) can inject a concise section:

TRANSFER EVIDENCE:
- Accepted reusable changes: [...]
- Task-specific residue to avoid carrying forward: [...]
- Unsupported claims: [...]
- Claim boundary: [...]

This would give the feedback agent an explicit distinction between "build on this" and "do not promote this as general capability."

Local Prototype Pattern

A local self-improvement prototype I use applies this gate:

{
  "claim_boundary": "heterogeneous feature-level self-improvement over list/string/grid/record tasks; not unbounded real-world general intelligence",
  "cold_frontier": {"validated": 0, "total": 8},
  "warm_frontier": {"validated": 8, "total": 8},
  "unsupported_probe": {"validated": 0, "total": 2},
  "meta_gate": {"accepted": true, "delta": 8}
}

The useful rule is:

score gain alone is not enough
  -> positive transfer must improve
  -> unsupported probes must remain rejected
  -> the next generation receives only bounded reusable claims

For SIA, the equivalent would be: a score-improving generation can still write task_specific_residue and unsupported_claims, and the feedback prompt should treat those as negative evidence rather than positive evolutionary material.

Minimal Implementation Path

• Add a small helper that reads results.json, improvement.md, and previous metrics to produce transfer_evidence.json.
• Start with a conservative heuristic version, not a new judge:
  • score delta improved or not;
  • results file present or missing;
  • extract bullets from improvement.md;
  • mark claims as "candidate reusable" unless they mention task/data-specific terms;
  • allow custom tasks to emit their own transfer evidence later.
• Update _build_feedback_context(...) to include a transfer evidence section when present.
• Add golden tests for the new feedback section, similar to test_feedback_context_golden.py.
• Add one fixture where results.json improves but transfer_evidence.json contains task_specific_residue, and verify the feedback prompt displays it as something not to carry forward.

This keeps SIA's loop aggressive while making the self-improvement signal cleaner: benchmark gains can still drive evolution, but only bounded, evidence-backed lessons get promoted into the next generation's reusable context.