Skip to content

content-type-checker: add classify-proposal mode#68

Merged
theletterf merged 3 commits into
elastic:mainfrom
florent-leborgne:add-classify-proposal-mode-content-type-checker
May 21, 2026
Merged

content-type-checker: add classify-proposal mode#68
theletterf merged 3 commits into
elastic:mainfrom
florent-leborgne:add-classify-proposal-mode-content-type-checker

Conversation

@florent-leborgne

Copy link
Copy Markdown
Member

Summary

  • Adds a classify mode to docs-content-type-checker so callers can submit a short description of a planned page and get back the best-fit content type, confidence level, required elements still to draft, and a recommendation. Existing validate-mode flow is unchanged.
  • 4 new evals exercise the new mode across high/medium/low-confidence cases.
  • Skill version: 2.1.1 → 2.2.0 (MINOR). Plugin version: 1.0.3 → 1.0.4.

Why

Today the skill only validates files or pasted content. Other skills that need to decide which content type a hypothetical page should use — for example, the docs-quest-scanner skill, when gating a "create new page" recommendation against the content-type definitions — currently bypass this skill and fetch the guideline pages directly. That duplicates per-type definitions in two places. Adding classify mode keeps the canonical rules here and lets downstream skills delegate.

What changed

SKILL.md

  • Top of body restated as two modes (validate, classify) with a Detect the mode section before Step 1
  • New Classify mode steps section with four sub-steps: read the proposal → match against type definitions → assign confidence → generate a classification report
  • Confidence ladder: high (clear required elements invoked) / medium (best fit but missing pieces, suggest outlining first) / low (no clean fit, recommend assembly check rather than a new page)
  • Output template explicitly distinct from the validate-mode report
  • Argument hint widened: <file-or-directory><file-or-directory-or-proposal>
  • Description widened to mention classify-mode triggers

evals.json

4 new evals (ids 6–9):

  • 6 — clear how-to proposal (sequential steps + prerequisites) → expects high-confidence how-to, lists Before-you-begin / steps / checkpoint as remaining elements
  • 7 — clear overview proposal (concept-led "what / why / how it works") → expects high-confidence overview, warns that procedural steps would be an anti-pattern
  • 8 — error-fix proposal that should be troubleshooting, not how-to → tests the problem-first vs task-first distinction
  • 9 — ambiguous reference-flavored proposal (a CLI flag with a one-line example) → expects low confidence on all types and an assembly-check recommendation, not an invented fit

Test plan

  • python3 -m json.tool evals/evals.json — valid
  • Local replay of validate-skills.yml Python checks — 17 skills validated, 0 errors
  • Manual review: validate-mode wording is preserved; existing evals 1–5 are unchanged
  • CI validate-skills.yml and skill-eval-test.md runs on PR
  • Eval grading on the 4 new cases

Notes for reviewers

  • Detection of classify vs validate mode relies on intent cues in the prompt (described content vs. existing content). Edge cases like "here is the rough outline I'm going to write…" with bullet points may straddle both modes; the SKILL.md instructs the model to ask one focused clarifying question rather than guess.
  • For the low-confidence path, the recommendation deliberately points to an assembly check (look for an existing page) rather than picking a content type. This is the behavior docs-quest-scanner was missing in its stage-2 validation step.

Adds a second mode that lets the skill classify an *unwritten* page
idea against the content types, complementing the existing
"validate-an-existing-page" mode. Useful for callers that need to
decide which content type a planned page should use before drafting,
or that need to gate a "create new page" recommendation on whether
the proposed content actually fits a content-type definition.

Changes:
- SKILL.md: adds Mode-detection guidance and a new "Classify mode
  steps" section (read proposal, match against type definitions,
  assign confidence, generate classification report). Existing
  validate-mode flow is unchanged.
- evals.json: 4 new evals (ids 6–9) covering a clear how-to proposal,
  a clear overview, a problem-first proposal that should be
  troubleshooting (not how-to), and an ambiguous reference-flavored
  proposal that should trigger an assembly-check recommendation
  rather than a confident classification.
- Skill version bumped 2.1.1 → 2.2.0 (MINOR — new capability, no
  breaking change).
- Plugin version bumped 1.0.3 → 1.0.4 per validate-skills CI.

Motivation: the docs-quest-scanner skill currently does its content-type
"would this fit?" check by directly fetching the guideline pages
because the existing skill is file-only. Adding classify mode here
keeps the canonical rules in one place and lets downstream skills
delegate cleanly.
theletterf and others added 2 commits May 21, 2026 10:04
Trivial conflict: — escape vs literal — em dash in description.
Kept the literal character form from main.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@theletterf theletterf merged commit c667ae7 into elastic:main May 21, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants