[Schema Consistency] Error Documentation Consistency Check - 2026-01-27

[github-actions[bot]]

This automated analysis examined error message consistency across schema definitions, parser/compiler code, and documentation to identify gaps where users might encounter undocumented errors.

Analysis Overview

Strategy: Error Path & Exception Handling Consistency Analysis (NEW)
Approach: Radically different from previous 18 strategies
Date: 2026-01-27
Workflow Run: §21418811357

Key Metrics

• Total error messages in codebase: 530 (343 workflow + 187 parser)
• Schema constraints requiring validation: 247 constraints
• Documented errors in errors.md: ~40 error patterns
• Undocumented user-facing errors: 22+
• Findings: 8 inconsistencies (3 critical, 5 moderate)

---

Critical Issues

1. dispatch-workflow Errors Completely Missing

Severity: Critical - Security-related feature

The dispatch-workflow safe-output has comprehensive validation with 3 security-critical error messages, but NONE are documented:

dispatch-workflow: self-reference not allowed (workflow '%s' cannot dispatch itself)
dispatch-workflow: workflow '%s' must be compiled first (run 'gh aw compile %s')
dispatch-workflow: invalid workflow name '%s' (path traversal not allowed)
```

**Location:** `pkg/workflow/dispatch_workflow.go` (implemented), missing from `docs/src/content/docs/troubleshooting/errors.md`

**Impact:** Users encountering infinite loop prevention, compilation dependencies, or path traversal security checks will have no documentation to reference.

**Recommendation:** Add "dispatch-workflow Safe Output Errors" section to errors.md with all three errors documented.

---

### 2. Template + Import Interaction Error Not Documented

**Error message:**
```
import directives cannot be used inside template regions ({{#if...}}{{/if}}): found '%s' at line %d within template block

Status: Validation implemented in pkg/parser/import_processor.go but not documented

Impact: Users attempting to use imports within Handlebars template blocks will encounter this error with no documentation.

Use case: This prevents incorrect patterns like:

imports:
{{#if condition}}
  - shared/tools.md
{{/if}}

Recommendation: Add to "Compilation Errors" section in errors.md with explanation of the limitation and workarounds.

---

3. Cron Expression Validation Errors Not Documented

Error messages:

• fuzzy cron expression '%s' must be scattered to proper cron format before compilation
• invalid cron expression '%s': must have exactly 5 fields (minute hour day-of-month month day-of-week)

Status: Validation exists in pkg/workflow/schedule.go but not documented

Impact: Users with invalid schedule cron expressions won't find troubleshooting guidance.

Recommendation: Add "Schedule Configuration Errors" section with cron expression format requirements and fuzzy expression documentation.

---

Moderate Issues

4. slash_command Error Documentation Incomplete

Current documentation: Only covers command field conflict error
Missing: slash_command field conflict error

Error messages:

• cannot use 'command' with '%s' in the same workflow ✓ Documented
• cannot use 'slash_command' with '%s' in the same workflow ✗ NOT Documented

Impact: Users using the newer slash_command field (recommended) won't find their error variant in documentation.

Recommendation: Update existing error entry to include both field names or add note that error applies to both command and slash_command.

---

5. GitHub Actions Mutual Exclusivity Errors Missing

Error messages:

• %s event cannot specify both 'branches' and 'branches-ignore'...
• %s event cannot specify both 'paths' and 'paths-ignore'...

Status: Runtime validation in pkg/workflow/compiler_events.go but not documented

Impact: Users violating GitHub Actions mutual exclusivity constraints won't find guidance in error reference.

Recommendation: Add "Trigger Filter Errors" section documenting mutual exclusivity requirements.

---

6. Action SHA Format Validation Error

Error message: invalid SHA format for %s@%s: %s

Status: Validation in action pinning code but not documented

Recommendation: Add to action pinning troubleshooting section with SHA format requirements.

---

7. Engine Validation Error

Error message: invalid engine: %s. Valid engines are: copilot, claude, codex, custom. Example: engine: copilot

Status: Self-explanatory error but not in documentation

Recommendation: Add to engine configuration errors section for completeness.

---

8. Generic Range Validation Error Format

Error pattern: %s must be between %d and %d, got %d

Applied to: Multiple fields (port, retention-days, etc.)

Status: Generic error used across many fields but specific constraints not documented

Recommendation: Document common range-validated fields with their specific constraints.

---

Positive Findings

View Excellent Implementation Examples

✓ tracker-id Validation (EXCELLENT)

All three schema constraints enforced with clear error messages:

• ✓ minLength: 8 → "tracker-id must be at least 8 characters long (got %d)"
• ✓ maxLength: 128 → Validated via string length check
• ✓ pattern: ^[a-zA-Z0-9_-]+$ → "tracker-id contains invalid character at position %d: '%c' (only alphanumeric, hyphens, and underscores allowed)"

Location: pkg/workflow/frontmatter_extraction_metadata.go:63-91

✓ branch-prefix Validation (EXCELLENT)

Complete constraint enforcement:

• ✓ minLength: 4 → "branch-prefix must be at least 4 characters long, got %d"
• ✓ maxLength: 32 → "branch-prefix must be at most 32 characters long, got %d"
• ✓ Reserved keyword rejection → "branch-prefix cannot be 'copilot' (reserved)"
• ✓ Pattern validation → Alphanumeric, hyphens, underscores enforced

✓ Strict Mode Error Documentation (EXCELLENT)

All 10 strict mode errors comprehensively documented in errors.md with:

• Clear error message
• Root cause explanation
• Actionable solution
• Links to relevant documentation pages
• Security context and rationale

✓ Error Message Quality (EXCELLENT)

Error messages consistently follow best practices:

• Explain the constraint violated
• Show actual value vs expected value
• Provide examples of correct usage
• Include links to relevant documentation
• Suggest specific fixes

---

Architectural Insights

Error Handling Architecture

The codebase follows a clear separation:

1. Parser errors (187): Schema validation, YAML parsing, import resolution, frontmatter extraction
2. Workflow compiler errors (343): Business logic validation, feature validation, security checks
3. Documentation coverage: ~40 documented patterns out of 530 total errors

Documentation Gap Patterns

Most undocumented errors fall into these categories:

Category	Examples	Count
Edge case validations	template+import, self-reference prevention	3
New features	dispatch-workflow safe-output	3
GitHub Actions constraints	branches/paths mutual exclusivity	2
Format validations	SHA format, cron expression format	2

Schema Constraint Coverage

Schema defines 247 constraints:

• 17 minLength
• 5 maxLength
• 24 pattern
• 59 minimum
• 33 maximum
• 19 minItems
• 19 maxItems
• 71 enum

Runtime validation approach:

• ✓ String length: Well-validated (tracker-id, branch-prefix)
• ✓ Numeric ranges: Well-validated (retention-days, port)
• ~ Pattern validation: Mixed (some validated, some schema-only)
• ~ Enum validation: Mostly schema-only
• ~ minItems/maxItems: Mostly schema-only

This selective approach is intentional - schema validation catches most issues, runtime validates critical security and business logic constraints.

---

Recommendations

High Priority (Documentation Gaps)

1. Document dispatch-workflow errors - Add dedicated section for all 3 validation errors
2. Document template + import error - Add to compilation errors with workarounds
3. Update slash_command documentation - Ensure both command variants covered

Medium Priority (Completeness)

4. Add mutual exclusivity errors - Document branches/paths validation
5. Add cron expression errors - Document format requirements and fuzzy expressions
6. Add SHA format error - Document action pinning format requirements

Low Priority (Enhancement)

7. Expand error coverage - Document additional 15-20 user-facing errors
8. Add error message tests - Validate documentation matches actual errors
9. Field-specific constraint docs - Document range constraints for all validated fields

Architectural (Future Improvement)

10. Error message registry - Consider centralizing error messages for consistency
11. Error code system - Consider adding error codes for easier documentation lookup

---

Strategy Performance

Effectiveness

Rating: Very High - Found 8 inconsistencies in first run

Unique value:

• First strategy to systematically analyze error message consistency
• Discovered documentation gaps in new features (dispatch-workflow)
• Found partial documentation (command vs slash_command variants)
• Validated constraint enforcement coverage
• Traced error flow from schema → code → documentation

Methodology

This analysis:

1. Extracted 530 error messages from Go code (fmt.Errorf, errors.New)
2. Extracted 247 schema constraints from JSON Schema
3. Parsed error documentation from docs/src/content/docs/troubleshooting/errors.md
4. Cross-referenced code errors with documented errors
5. Validated that schema constraints have error handling
6. Analyzed error message patterns and quality

Complementary Strategies

Works well with:

• Strategy-003: Required field enforcement (validates constraints exist)
• Strategy-005: Conditional logic analysis (validates complex rules)
• Strategy-008: Schema metadata completeness (validates descriptions)

Recommendation

Use every 5-6 analyses to validate error documentation stays current as new features and validations are added.

Trigger after:

• Adding new safe-outputs
• Adding new validation logic
• Adding new schema constraints
• Major feature releases

---

Next Steps

• Add dispatch-workflow error documentation (3 errors)
• Document template + import interaction limitation
• Document cron expression validation requirements
• Update command/slash_command error documentation
• Add mutual exclusivity error examples
• Consider error documentation CI test

References:

• §21418811357 - This workflow run
• [Error Reference Documentation]((redacted)
• Strategy Cache Database

AI generated by Schema Consistency Checker

• expires on Feb 3, 2026, 11:54 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-03T23:54:30.872Z.