Epic: Optional AgentV evaluation integration for MetaSkills adopters

[smota]

Epic

Incorporate AgentV as an optional evaluation tool for adopting metaskills projects.

Product intent

MetaSkills should help downstream projects build portable agents/skills and optionally evaluate them locally with versioned evals. AgentV should be positioned as an optional evaluation layer, not as required runtime infrastructure.

Recommended framing:

MetaSkills can generate and package evaluation specs; adopting projects can optionally run them with AgentV.

Avoid framing:

MetaSkills manages eval infrastructure.

Current context

• Repo: metaskills
• Graph artifact: GRAPH_REPORT.md
• Graph freshness note: report built at 4b2c72e6; repo was 61d74b5e during analysis. Run graphify update . before implementation plan freeze.
• Existing eval concepts:
  • skills/agent-builder/templates/agent-evals.md
  • skills/skill-creator/templates/skill-eval-spec.md
• Repo architecture favors local-first, portable, Git-backed assets under skills/, agents/, toolsets/, and optional components/ submodules.
• AgentV fit: local-first CLI, Git-backed YAML evals/results, multi-agent/CLI target support.

AgentV facts used in analysis

From AgentV docs/site:

• CLI-first evaluation framework.
• Install: npm install -g agentv.
• Run: agentv eval ./evals/example.yaml.
• Local-first; no server/signup for eval infrastructure.
• YAML specs.
• Multi-objective scoring: correctness, latency, cost, safety.
• Judges: deterministic script graders, llm-rubric, custom Python/TypeScript graders.
• Targets include direct LLM providers and agent CLIs such as Claude Code, Codex, Pi, Copilot, OpenCode.
• Results saved under .agentv/results/<run_id>/ with portable JSON/JSONL/YAML artifacts.
• Supports compare/regression detection for CI/CD gates.

Desired product capabilities

1. Optional AgentV eval pack per skill/agent.
2. AgentV provider/target setup guidance for adopting projects.
3. Mapping from current MetaSkills eval templates to AgentV YAML.
4. CI recipe for regression gates and result artifact upload.
5. Adoption maturity ladder:
   • Level 0: Markdown eval checklist only
   • Level 1: AgentV YAML cases
   • Level 2: deterministic graders
   • Level 3: LLM rubric graders
   • Level 4: CI regression gate
   • Level 5: tracked improvement loop

Proposed repo shape

toolsets/
  agentv/
    README.md
    templates/
      providers.yaml
      eval.eval.yaml
      github-actions-agentv.yml
    mappings/
      metaskills-eval-to-agentv.md
    examples/
      skill-eval.eval.yaml
      agent-eval.eval.yaml

Acceptance criteria

• AgentV remains optional; no root runtime dependency unless explicitly justified later.
• toolsets/agentv/ contains adopter-ready guidance, examples, and templates.
• Existing agent/skill eval templates mention optional AgentV export/run path.
• CI recipe documented and usable by adopting repos.
• Validation command documented.
• Security guidance covers .env, API keys, and committed result artifacts.
• Follow-up decision recorded if/when adding components/agentv submodule.

Related implementation issues

• Add optional AgentV toolset docs and templates #19
• Map MetaSkills eval templates to AgentV YAML specs #20
• Update agent and skill eval templates with optional AgentV path #21
• Add downstream GitHub Actions recipe for AgentV evals #22
• Document AgentV eval adoption maturity ladder #23
• Decide whether AgentV should become a component submodule #24