Skip to content

[Bug] JSON schema validation against the Assessment submission which has less than 25 attributes fails hard in the Leaderboard CI #309

@kami619

Description

@kami619

Bug Description

The JSON schema (assessment-schema.json) rejects valid assessments generated with --exclude flags because it hardcodes attributes_total as exactly 25 and findings array length as exactly 25.

To Reproduce

  1. Run agentready assess . --exclude architecture_decisions --exclude openapi_specs
  2. Submit the assessment to the leaderboard via PR
  3. CI fails with: attributes_total: 25 was expected and findings: [...] is too short

See PR #301 for a real example.

Expected Behavior

Assessments using --exclude should either:

  • Validate successfully (if exclusions are allowed for leaderboard)
  • Fail with a clear message at assessment time (if exclusions are disallowed)

Actual Behavior

The assessment generates successfully, but schema validation fails at submission time with a confusing error.

Environment

  • agentready: 2.28.0
  • Python: 3.12
  • CI: GitHub Actions (Ubuntu 24.04)

Possible Solution

Option A (minimal change): Add a leaderboard-specific check in the workflow that rejects exclusions early with a clear message, keeping the schema strict:

- name: Reject excluded assessments
  run: |
    TOTAL=$(jq -r '.attributes_total' "$ASSESSMENT_FILE")
    if [ "$TOTAL" -ne 25 ]; then
      echo "::error::Leaderboard requires all 25 attributes. Found $TOTAL (did you use --exclude?)."
      exit 1
    fi

This preserves the intent that leaderboard submissions must be comparable (all 25 attributes), while giving users a clear error instead of a schema validation failure.

Option B: Relax the schema to allow attributes_total <= 25 — simpler but allows non-comparable submissions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions