Establish a formal language-theoretic schema for IDD specification formats

[slusset]

Summary

The IDD spec format currently evolves through documentation, examples, and validator behavior, but it does not yet have a single normative schema that defines which artifact forms are legal and how they compose.

This creates ambiguity when introducing richer representations such as:

• annotated journey references
• lifecycle/state-machine shorthand
• aggregate-root model documents
• catalog/registry documents
• shared value-object bundles

Problem

Today, format evaluation is operational rather than fully formal:

• docs/idd/front-matter-spec.md defines part of the shape
• validators enforce parts of the format
• examples demonstrate accepted forms
• certification verifies traceability and evidence

That is useful, but it means the canonical language is partly implicit in tool behavior. As the spec surface expands, that increases the risk of drift between documentation, examples, and validators.

Desired outcome

Define a formal language-theoretic schema for IDD specifications so that:

• each artifact kind has a normative grammar or meta-schema
• allowed document families are explicit rather than inferred
• validator behavior is derived from the documented schema
• examples act as acceptance fixtures rather than de facto standards
• new spec forms can be evaluated against a published extension model

Scope

This issue should establish the schema approach, not necessarily complete every validator rewrite in one pass.

In scope:

• decide the formalism to use for each artifact class
• document canonical forms for narrative, model, contract, and traceability metadata
• define how extensions are introduced without breaking compatibility
• identify which current formats are normative, legacy-compatible, or experimental
• define conformance levels for docs, examples, validators, and certification

Out of scope:

• redesigning the entire IDD methodology
• changing downstream implementation skills unless needed for schema conformance

Acceptance criteria

• A normative schema document exists for IDD artifact formats.
• The schema distinguishes canonical, legacy-compatible, and experimental forms.
• Model documents explicitly cover entity, value object, aggregate, catalog, bundle, and lifecycle/state-machine forms.
• Traceability metadata rules explicitly cover annotated references where supported.
• Validator responsibilities are mapped to the schema rather than relying on undocumented assumptions.
• Example fixtures are identified as conformance fixtures for the schema.
• A migration strategy exists for evolving formats without breaking existing adopters.

Why this matters

Without a formal schema, tool behavior becomes the effective standard. That slows evolution, makes validator bugs harder to classify, and makes it unclear whether a new representation is invalid, unsupported, or simply undocumented.

A formal schema would let IDD evolve intentionally instead of extemporaneously.

[slusset]

Sequencing note from explore/methodology-attribution

The methodology-attribution exploration (branch explore/methodology-attribution) is investigating six candidate primitives (shared memory, action budgets, observation frames, hypothesis lifecycle, execution traces, handoff protocols) that may need to become first-class artifact kinds if they earn provisional status.

Recommendation: defer the full schema formalization until the exploration settles.

Formalizing the current artifact surface now risks either:

• constraining evolution (new primitives must fit a schema designed before we knew they existed), or
• immediate rework (schema published, then extended within weeks as primitives are promoted)

What #24 already has coverage for today:

• The methodology-change-process (docs/idd/methodology-change-process.md) already establishes conformance levels (exploratory → provisional → canonical), which partially addresses the "canonical vs legacy-compatible vs experimental" distinction this issue calls for.
• Front-matter spec and validators are functional for the current artifact surface.

What to do when the exploration resolves:

• Include promoted primitives as artifact kinds in the schema from the start
• Use the exploration's eval metrics (M1–M5) to validate that schema conformance actually correlates with methodology value — not just structural compliance
• The schema should formalize a methodology whose primitives have been tested, not one that's about to change

This issue remains valuable — just better sequenced after the exploration produces evidence.