pharaoh:retrofit — Help users introduce sphinx-needs and traceability into existing codebases

[patdhlk]

Problem

Teams with existing documentation or codebases want to adopt sphinx-needs for requirements traceability but face a chicken-and-egg problem: Pharaoh's skills assume sphinx-needs is already set up with types, links, and needs in place. There's no guided path for:

1. Introducing sphinx-needs from scratch into an existing Sphinx (or non-Sphinx) documentation project
2. Incrementally adding traceability without a big-bang rewrite — start with a few requirement types, grow over time
3. Converting existing content (markdown, plain RST, Word docs, Jira tickets) into sphinx-needs directives
4. Handling the markdown/RST boundary — projects using MyST-Parser can define needs in .md files, but some sphinx-needs directives require RST syntax, creating a conversion decision point

Scope: New pharaoh:retrofit skill

A guided workflow that helps users introduce sphinx-needs into an existing project at whatever pace makes sense — from "add your first requirement" to "convert an entire doc tree to traced needs."

Key capabilities

1. Project assessment

• Detect current documentation format (RST, Markdown/MyST, mixed)
• Identify existing content that could become needs (headings that look like requirements, TODO lists, feature descriptions)
• Assess project size and recommend an introduction strategy (incremental vs. bulk)

2. sphinx-needs bootstrapping

• Guide ubproject.toml creation with appropriate need types for the project's domain
• Recommend a minimal starting set of types (don't start with 10 types — start with 2-3)
• Configure [needs.fields], [needs.links], and [[needs.types]] using modern TOML syntax:

  [[needs.types]]
  directive = "req"
  title = "Requirement"
  prefix = "REQ_"
  
  [[needs.types]]
  directive = "spec"
  title = "Specification"
  prefix = "SPEC_"
  
  [needs.links.implements]
  outgoing = "implements"
  incoming = "is implemented by"

• Set up ID schemes (id_required, id_regex) appropriate for the project
• Optionally configure schema validation via schema_definitions_from_json

3. Content conversion

• Convert identified content into sphinx-needs directives with proper IDs, types, and initial links
• Handle the Markdown ↔ RST decision:
  • MyST-Parser projects: needs can be defined in .md using MyST directive syntax ({req}, {spec})
  • When MyST doesn't support a needed directive or option: guide selective conversion of specific .md files to .rst
  • Never force a full markdown→RST migration — convert only files that need sphinx-needs directives that MyST can't handle
• Preserve existing content structure and cross-references during conversion

4. Multi-level introduction (avoid redundancy)

• Level 1: Requirements only — add req type, capture top-level requirements from existing docs
• Level 2: + Specifications — add spec type with :implements: links to requirements
• Level 3: + Implementation/Test — add impl and test types, link to specs
• Level 4: + Custom types — domain-specific types (e.g., decision, risk, constraint)
• At each level, validate that new needs link to existing ones — no orphans from the start
• Use needtable and needflow to visualize coverage at each level

5. Expert guidance

• Warn about common mistakes: too many types too early, IDs that don't scale, missing link types
• Recommend ubc CLI installation for indexing and validation
• Suggest pharaoh:mece after each batch of conversions to check for gaps
• Guide users toward pharaoh:verify once implementations exist

Integration with existing skills

pharaoh:retrofit [NEW]
  |-- pharaoh:setup        (project detection, ubproject.toml generation)
  |-- pharaoh:author       (write new need directives)
  |-- pharaoh:mece         (validate coverage after each batch)
  '-- pharaoh:trace        (verify links are connected)

pharaoh:retrofit orchestrates the introduction workflow by calling existing skills for the actual work. It adds the migration-specific intelligence: what to convert, in what order, at what pace.

What this is NOT

• Not an automated migration tool — the AI guides and assists, the user decides what becomes a need
• Not a one-shot converter — designed for incremental adoption over multiple sessions
• Not a replacement for pharaoh:setup — retrofit calls setup internally, adds the "what to do after setup" guidance

Acceptance criteria

• User with a plain Sphinx project (no sphinx-needs) can run pharaoh:retrofit and get a working ubproject.toml with appropriate types
• User with existing markdown docs gets guidance on MyST vs RST for needs directives
• Multi-level introduction works: user can stop after Level 1 and come back later for Level 2
• Each level ends with a pharaoh:mece check showing current coverage
• No redundant needs are created — the skill checks existing content before suggesting new needs