Skip to content

feat(testing-plugin): ship test-analyze classify-and-act harness template #2170

Description

@laurigates

Tier A #4. Zero refutations — the only design in the set that prices itself.

Source: docs/plans/dynamic-workflow-migration.md §4 (merged in #2162).

Why it wins

The repo's one clean classify-and-act fit: a fixed category→agent-type table that is currently restated twice in prose. Every other classify candidate turned out to have a shell script already doing the routing (list-actionable-prs.sh, list-components.sh TYPES=, blueprint-autorun.sh), which is why they were cut.

It also carries its own cost guard — if (failures.length < 5) return {mode:'inline'} plus a structural cap of 8 categories.

Two corrections baked in

  1. Classify is merged into Parse. The 8-row routing table plus a 4-value severity enum is a lookup the parse agent can do while it already holds the failures — a separate classifier agent is pure overhead.
  2. AGENT_FOR is keyed off agent names that actually exist. The table currently in SKILL.md names agent types that agents-plugin/agents/*.md does not ship. Fixing those stale names is a bug fix independent of this harness and should land regardless.

Sketch

phase('Parse + classify');
const parsed = await agent(
  `Read every result file under ${args.resultsPath}. Per failure emit
   {id, test, file, message, stack_head, category, severity} using the enums below.
   Anything fitting no category goes to unroutable[] with a reason.`,
  { label:'parse', schema: RoutedFailuresSchema, model:'opus' });

if (parsed.failures.length < 5) {            // HARD FLOOR, not a knob
  log(`only ${parsed.failures.length} failures — use the inline path in SKILL.md`);
  return { mode:'inline', failures: parsed.failures };
}

// Real agent names from agents-plugin/agents/*.md — NOT the stale table in SKILL.md.
const AGENT_FOR = { accessibility:'review', security:'security-audit', performance:'performance',
                    quality:'refactor', flaky:'test', integration:'debug', ci:'ci', docs:'docs' };
const groups = Object.values(parsed.failures.reduce((acc, f) => {   // group by AGENT TYPE, not category
  const at = AGENT_FOR[f.category]; (acc[at] ??= { at, items: [] }).items.push(f); return acc }, {}));

phase('Act');
const plans = await parallel(groups.map(g => () => agent(
  `You own these failures ONLY. Per failure: objective, success criteria, depends_on (by failure id),
   verification command. ${JSON.stringify(g.items)}`,
  { label:`plan:${g.at}`, phase:'plan', schema: PlanSchema, agentType: g.at, model:'opus' })));
// BARRIER: sequencing resolves dependencies ACROSS groups.

phase('Synthesize');
return await agent(`Merge into ONE ordered fix plan; topologically sequence the depends_on edges the
  group agents emitted (mcp__pal__planner is NOT available here — the edges are inferred, not planned).
  Unroutable: ${JSON.stringify(parsed.unroutable)}\n${JSON.stringify(plans.filter(Boolean))}`,
  { label:'synthesize', schema: FixPlanSchema, model:'opus' });

SKILL.md delta

Fix the stale agent-type names in the body first — that is a bug independent of any workflow. Collapse the routing table's two copies into one under ## Subagent Selection Logic, annotated:

this table is the contract the workflow's category enum and AGENT_FOR map encode — edit them together.

Add to the new workflow section:

the harness surrenders mcp__pal__planner; a run that needs PAL planning should stay inline

context: fork is pinned at plugin-compliance-check.sh:826 and stays; the parallel() width is capped at the fixed agent-type set precisely so it does not become the wide fan-out skill-fork-context.md warns about.

Acceptance

  • Stale agent-type names fixed (independently landable)
  • Routing table collapsed to one copy, annotated as the shared contract
  • workflows/test-analyze.workflow.js with the <5 floor and the 8-category cap
  • context: fork resolution stated inline

Blocked by #2165 (the context: fork vs skill-fork-context.md conflict — this is the skill it turns on) and #2163.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions