feat: comprehensive CLI UX improvements - #24
Merged
Conversation
added 2 commits
July 10, 2026 17:41
- Create docs/examples/index.md with tutorial overview - Add Examples tab to mkdocs.yml navigation - Restore Providers navigation section (lost in merge)
- Add global error handler for OpenAgentEvalError with friendly messages - Add global flags: --quiet, --json, --no-color, --verbose - Add config auto-discovery (config.yaml/oaeval.yaml in cwd, OAEVAL_CONFIG env var) - Add validate command to check config without running evaluation - Add dry-run mode (--dry-run flag on run command) - Add shell completion support for bash, zsh, and fish - Enhance doctor command with API connectivity tests - Add delete command for removing old reports - Enhance list command with sorting (date/score/cost) and search filtering - Enhance init command with interactive wizard for provider/model selection - Add comprehensive test suite for all new features Closes #123
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| report_id = r.get("report_id", "unknown") | ||
| created_at = r.get("created_at", "unknown") | ||
| config_name = "unknown" | ||
| status = "unknown" |
| elif sort_by == "cost": | ||
| # Get total cost | ||
| return data.get("cost", {}).get("total", 0.0) | ||
| except (FileNotFoundError, KeyError): |
| else: | ||
| size_str = f"{size / (1024 * 1024):.1f} MB" | ||
| console.print(f" [dim]Size: {size_str}[/dim]") | ||
| except OSError: |
| from __future__ import annotations | ||
|
|
||
| import json | ||
| from pathlib import Path |
|
|
||
| import json | ||
| from pathlib import Path | ||
| from unittest.mock import MagicMock, patch |
| from pathlib import Path | ||
| from unittest.mock import MagicMock, patch | ||
|
|
||
| import pytest |
|
|
||
| from __future__ import annotations | ||
|
|
||
| import json |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements comprehensive CLI UX improvements for OpenAgent Eval, addressing user experience gaps and adding powerful new features.
Changes
New Commands
oaeval validate [config]- Validate configuration without running evaluationoaeval delete <id>- Delete evaluation reports (single or all)oaeval completion bash/zsh/fish- Generate shell completion scriptsEnhanced Commands
oaeval init- Interactive wizard with provider/model selectionoaeval run- Dry-run mode, metrics override, better progress, JSON outputoaeval doctor- API connectivity tests, recommendationsoaeval list- Sorting (date/score/cost), search filtering, JSON outputoaeval report- JSON output supportoaeval compare- JSON output supportGlobal Features
OpenAgentEvalErrorsubclasses with friendly Rich output--quiet,--json,--no-color,--verboseon all commandsconfig.yaml/oaeval.yamlautomatically, supportsOAEVAL_CONFIGenv varFiles Changed
New Files
openagent_eval/cli/context.py- Global CLI context for managing flagsopenagent_eval/cli/utils/discovery.py- Config auto-discovery utilityopenagent_eval/cli/commands/validate.py- New validate commandopenagent_eval/cli/commands/delete.py- New delete commandtests/unit/test_cli/test_cli_improvements.py- 33 new testsModified Files
openagent_eval/cli/main.py- Global error handler, global flags, shell completion commandopenagent_eval/cli/commands/run.py- Dry-run mode, metrics override, better progress, JSON outputopenagent_eval/cli/commands/init.py- Interactive wizard with provider/model selectionopenagent_eval/cli/commands/doctor.py- API connectivity tests, recommendationsopenagent_eval/cli/commands/list_evaluations.py- Sorting, search filtering, JSON outputopenagent_eval/cli/commands/report.py- JSON output supportopenagent_eval/cli/commands/compare.py- JSON output supportopenagent_eval/cli/utils/helpers.py- Fixed error chaining, added docstringsopenagent_eval/cli/utils/display.py- Added format_file parameterTesting
Usage Examples
Validate configuration
Dry-run mode
Generate shell completion
Delete reports
List with sorting
JSON output