docs: add CLI reference and improve help output

[sjnims]

Description

Add comprehensive CLI documentation and improve help output for all commands with option groups and usage examples.

Type of Change

• Documentation update (improvements to README, CLAUDE.md, or inline docs)

Component(s) Affected

Core Infrastructure

• CLI (src/cli/)

Other

• Documentation (CLAUDE.md, README.md)

Motivation and Context

The CLI help output needed improvements for better discoverability and usability. This change:

• Creates a comprehensive CLI reference document
• Organizes options into logical groups (Input, Execution Mode, Output, Testing, etc.)
• Adds practical usage examples to every command
• Improves option descriptions for clarity

How Has This Been Tested?

Test Configuration:

• Node.js version: 22.x
• OS: macOS

Test Steps:

1. npm run build && npm run lint && npm run typecheck && npm test - all pass
2. npm run format && npm run knip && npm run madge - all pass
3. Verified help output for all commands:
   • cc-plugin-eval --help
   • cc-plugin-eval run --help
   • cc-plugin-eval resume --help
   • cc-plugin-eval report --help
   • cc-plugin-eval list --help
   • cc-plugin-eval analyze --help
   • cc-plugin-eval generate --help
   • cc-plugin-eval execute --help

Checklist

General

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• My changes generate no new warnings or errors

TypeScript / Code Quality

• Strict TypeScript checks pass (npm run typecheck)
• ESM import/export patterns used correctly

Documentation

• I have updated inline JSDoc comments where applicable

Linting

• I have run npm run lint and fixed all issues
• I have run npm run format:check

Testing

• I have run npm test and all tests pass
• Test coverage meets thresholds

Example Output

Usage: cc-plugin-eval run [options]

Run full evaluation pipeline

Input Options:
  -p, --plugin <path>     Path to plugin directory
  -c, --config <path>     Path to config file (default: config.yaml)
  --marketplace <path>    Evaluate all plugins in marketplace

Execution Mode:
  --dr, --dry-run         Generate scenarios without execution
  --fast                  Only run previously failed scenarios
  ...

Examples:
  $ cc-plugin-eval run -p ./my-plugin
  $ cc-plugin-eval run -p ./my-plugin --dry-run
  $ cc-plugin-eval run -p ./my-plugin --estimate

Additional Notes

Files changed:

• docs/cli.md - New comprehensive CLI reference (428 lines)
• src/cli/commands/*.ts - Added option groups and examples to all 7 commands

🤖 Generated with Claude Code

[claude]

Claude finished @sjnims's task in 2m 22s —— View job

---

PR Review: docs: add CLI reference and improve help output

Verdict: ✅ Looks Good

This PR adds comprehensive CLI documentation and improves help output with well-organized option groups and practical examples.

---

Quality Checks ✅

All automated checks pass:

• ✅ npm run typecheck - No type errors
• ✅ npm run lint - No linting issues
• ✅ npm run format:check - All files properly formatted
• ✅ npm run knip - No dead code detected

---

What's Good

Documentation Quality:

• Excellent comprehensive CLI reference in docs/cli.md (437 new lines)
• Clear command overview table with all 7 commands
• Well-organized option groups (Input, Execution Mode, Output, Testing, Filter, Identification)
• Practical examples for every command showing common usage patterns
• Coverage of all output formats (JSON, YAML, JUnit XML, TAP) with examples
• Helpful workflow sections (Development, CI/CD, Iterative Testing, Cost Estimation)

Code Quality:

• Minimal, focused changes to 7 command files in src/cli/commands/
• Consistent use of .optionsGroup() and .addHelpText() across all commands
• Improved option descriptions (e.g., --semantic now says "Generate prompt variations to test robustness" instead of just "Enable semantic variation testing")
• Better clarity on defaults (e.g., "default: config.yaml")

Consistency:

• All commands follow the same pattern for organizing options
• Examples use consistent formatting and demonstrate realistic scenarios
• Option groups are logically categorized and easy to scan

---

Suggestions

1. Documentation - Verbose Option Placement (Non-blocking)

In docs/cli.md:279, the execute command lists --verbose under "Input Options":

**Input Options:**

| Option               | Description                               |
| -------------------- | ----------------------------------------- |
| `-p, --plugin <path>`| Path to plugin directory                  |
| `-c, --config <path>`| Path to config file (default: config.yaml)|
| `--verbose`          | Detailed progress output                  |

For consistency with the run command (which has --verbose under "Output Options"), consider moving it to its own group or under "Output Options" if you update the docs/code in the future.

Note: The actual implementation in execute.ts:46 doesn't use .optionsGroup() before --verbose, so it's ungrouped in the help output. This is fine but slightly inconsistent with the documentation.

---

Summary

Excellent work on improving the CLI documentation and help output! The comprehensive reference guide will significantly improve discoverability and usability. The changes are well-tested, properly formatted, and follow the project's patterns.

The only minor observation is about --verbose categorization in the execute command, but this doesn't block approval—it's a minor consistency detail that could be addressed in future refinements if desired.

Recommended action: Merge when ready. ✅

---

[sjnims]

Fixed in 0573ec5 - moved --verbose to its own "Output Options" group in both the execute command code and docs for consistency with the run command.

[claude]

Claude finished @sjnims's task in 2m 47s —— View job

---

PR Review: docs: add CLI reference and improve help output

Verdict: ✅ Approved - Excellent Work

This PR successfully adds comprehensive CLI documentation and improves help output with well-organized option groups and practical examples. The follow-up fix in commit 0573ec5 properly addressed the previous review feedback.

---

Quality Checks ✅

All automated checks pass:

• ✅ npm run typecheck - No type errors
• ✅ npm run lint - No linting issues
• ✅ npm run format:check - All files properly formatted
• ✅ npm run knip - No dead code detected

---

What's Good

Documentation Quality:

• Comprehensive CLI reference in docs/cli.md (442 new lines)
• Clear command overview table listing all 7 commands
• Well-organized option groups (Input, Execution Mode, Output, Testing, Filter, Identification)
• Practical examples for every command demonstrating real-world usage
• Complete coverage of all output formats (JSON, YAML, JUnit XML, TAP) with examples
• Helpful workflow sections (Development, CI/CD, Iterative Testing, Cost Estimation)
• Good configuration precedence explanation (CLI flags → env vars → config.yaml → defaults)

Code Quality:

• Focused, minimal changes to exactly 7 command files in src/cli/commands/
• Consistent use of .optionsGroup() and .addHelpText() patterns
• Improved option descriptions throughout:
  • --semantic: "Generate prompt variations to test robustness" (much clearer than before)
  • --samples: "Multi-sample judgment count (improves confidence)" (explains purpose)
  • --reps: "Repeat each scenario N times (measures variance)" (explains purpose)
• Better defaults documentation (e.g., "default: config.yaml")
• No TypeScript issues, no any types, proper ESM imports with .js extensions

Fix Verification:

• Commit 0573ec5 successfully addressed the previous review feedback
• --verbose now correctly placed under "Output Options" in both code and docs for the execute command
• Consistent with the run command's organization

---

Suggestions

1. Minor Consistency Note (Non-blocking)

I posted an inline comment on generate.ts:35. The --verbose option is grouped under "Testing Options" in the generate command, but under "Output Options" in run and execute commands. This is minor and can be addressed in a future cleanup PR if desired.

2. Documentation Enhancement Opportunities (Future consideration)

These are excellent additions that could be considered for future PRs:

• Add a "Global Options" section showing options inherited from the main program (if any)
• Consider adding return code documentation (what exit codes mean)
• Could add a troubleshooting section for common issues

---

Summary

Outstanding work! This PR significantly improves the CLI user experience through:

1. Comprehensive reference documentation that covers all commands thoroughly
2. Improved help output with logical grouping and clear descriptions
3. Practical examples that show real-world usage patterns
4. Complete coverage of output formats and workflows

The fix in commit 0573ec5 properly addressed the previous review feedback about --verbose placement. The only minor observation is about consistency in the generate command (noted in inline comment), but this doesn't block approval.

Recommendation: Merge when ready. This is a solid documentation improvement that will help users discover and use the CLI effectively. ✅

---