-
Notifications
You must be signed in to change notification settings - Fork 34
Description
Bug Description
The JSON schema (assessment-schema.json) rejects valid assessments generated with --exclude flags because it hardcodes attributes_total as exactly 25 and findings array length as exactly 25.
To Reproduce
- Run
agentready assess . --exclude architecture_decisions --exclude openapi_specs - Submit the assessment to the leaderboard via PR
- CI fails with:
attributes_total: 25 was expectedandfindings: [...] is too short
See PR #301 for a real example.
Expected Behavior
Assessments using --exclude should either:
- Validate successfully (if exclusions are allowed for leaderboard)
- Fail with a clear message at assessment time (if exclusions are disallowed)
Actual Behavior
The assessment generates successfully, but schema validation fails at submission time with a confusing error.
Environment
- agentready: 2.28.0
- Python: 3.12
- CI: GitHub Actions (Ubuntu 24.04)
Possible Solution
Option A (minimal change): Add a leaderboard-specific check in the workflow that rejects exclusions early with a clear message, keeping the schema strict:
- name: Reject excluded assessments
run: |
TOTAL=$(jq -r '.attributes_total' "$ASSESSMENT_FILE")
if [ "$TOTAL" -ne 25 ]; then
echo "::error::Leaderboard requires all 25 attributes. Found $TOTAL (did you use --exclude?)."
exit 1
fiThis preserves the intent that leaderboard submissions must be comparable (all 25 attributes), while giving users a clear error instead of a schema validation failure.
Option B: Relax the schema to allow attributes_total <= 25 — simpler but allows non-comparable submissions.