fix(schema): allow assessments with excluded attributes

[kami619]

Description

Relaxes the JSON schema constraints to accept assessments with 10-25 attributes instead of exactly 25. This enables users who run agentready assess --exclude to submit valid assessments to the leaderboard.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• Performance improvement
• Test coverage improvement

Related Issues

Fixes #309

Changes Made

• attributes_total: changed from const: 25 to minimum: 10, maximum: 25
• findings: changed from minItems: 25, maxItems: 25 to minItems: 10, maxItems: 25

Added regression tests to verify:

• Assessments with 10 attributes validate (minimum boundary)
• Assessments with 22 attributes validate (PR Leaderboard: dbasunag/opendatahub-tests (69.9/100 - Silver) #301 case)
• Assessments with fewer than 10 attributes are rejected

Testing

• Unit tests pass (pytest)
• Integration tests pass
• Manual testing performed
• No new warnings or errors

Checklist

• My code follows the project's code style
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published

[github-actions]

📈 Test Coverage Report

Branch	Coverage
This PR	66.1%
Main	66.1%
Diff	✅ +0%

---

Coverage calculated from unit tests only

[github-actions]

AgentReady Code Review — PR #312

Reviewer: Claude Code (AgentReady Review Agent)
Date: 2026-02-19
Score Impact: Neutral on scoring; fixes a leaderboard submission blocker for users of --exclude

---

Summary

This PR addresses issue #309: users running agentready assess --exclude <attrs> generate valid assessments with fewer than 25 findings, but the JSON schema validation rejected them because attributes_total was const: 25 and findings required exactly 25 items. The fix relaxes both constraints to a range of 10-25. Three regression tests are added.

This is a well-motivated, targeted fix with good test coverage.

---

Strengths

• Directly fixes the reported issue: Changing const: 25 to minimum/maximum is the correct schema primitive to use.
• Minimum floor of 10: Prevents trivially small assessments from passing as valid reports, which is a sensible guard.
• Three meaningful regression tests: Boundary test (10 attributes), common case (22 attributes from PR Leaderboard: dbasunag/opendatahub-tests (69.9/100 - Silver) #301), and rejection test (9 attributes). These directly encode the requirements.
• Clear description: The PR body explains the problem, the fix, and the test cases clearly.

---

Issues

Medium

1. attributes_assessed and attributes_skipped maximums are not updated

The schema currently allows attributes_assessed: max 25 and attributes_skipped: max 25 even when attributes_total is 10. A report with attributes_total: 10 and attributes_assessed: 25 is logically impossible but passes schema validation. JSON Schema does not natively enforce cross-field sum constraints, but this inconsistency should at least be documented or the maximums lowered.

2. findings count is not required to match attributes_total

A report with attributes_total: 15 and 22 findings is technically valid under the new schema. If strict counts are important for leaderboard integrity, consider whether this is acceptable or whether application-level validation should enforce the invariant.

Minor

3. Schema version not bumped

The schema title is still "AgentReady Assessment v1.0.0". This is a backwards-incompatible relaxation — old strict validators will now accept documents they previously rejected. Bumping to 1.1.0 and noting the change in contracts/research-report-schema.md would signal the intent.

4. Minimum of 10 is undocumented

The description field added to attributes_total says the value may be less than 25, but does not explain why 10 is the floor. A brief rationale (e.g., "minimum to ensure a meaningful assessment") aids future maintainers.

5. Fixture assumption in regression tests

The tests mutate valid_report_data["findings"] using slice syntax (e.g., [:10]). This relies on the fixture always having at least 25 entries. Consider asserting len(valid_report_data["findings"]) == 25 at the start of each boundary test to make the dependency explicit.

---

Security / Data Integrity

No security concerns. Schema relaxation is unidirectional (more permissive). The risk of accepting malformed data is low since assessments are user-generated for their own repos.

---

AgentReady Attribute Compliance

Attribute	Status	Notes
test_coverage	Good	Three regression tests; all meaningful
code_quality	Pass	Clean schema JSON, well-structured tests
documentation	Partial	Schema description field added, but version not bumped

---

Recommendation

Approve with suggestions: The core fix is correct and the tests are solid. The schema version bump and a comment on the 10-attribute floor are low-effort improvements worth doing before merge. The cross-field invariant gaps are pre-existing schema limitations not introduced by this PR and are acceptable to defer.

[github-actions]

AgentReady Code Review

PR: fix(schema): allow assessments with excluded attributes
Reviewer: Claude (automated review)

---

Summary

This PR fixes a real user-facing issue (#309): assessments generated with agentready assess --exclude were rejected by the JSON schema because it hardcoded attributes_total: const: 25. The fix relaxes the constraint to minimum: 10, maximum: 25 and ships three regression tests.

Overall: Correct fix for a genuine bug. A few schema consistency concerns to address before moving out of DRAFT.

---

AgentReady Attribute Compliance

Attribute	Status	Notes
Test Coverage	✅ Pass	3 regression tests added, covers boundary cases
Schema Versioning	⚠️ Concern	Schema changed but schema_version not bumped
Documentation	⚠️ Partial	PR checklist largely unchecked
Conventional Commits	✅ Pass	Commit message follows fix(schema): convention
Code Style	✅ Pass	Consistent with existing schema style

---

Issues

🔴 Critical

None.

🟡 Important

1. Schema change without version bump

The schema file has "title": "AgentReady Assessment v1.0.0" but the schema_version field in the schema is validated by pattern only (^\d+\.\d+\.\d+$). This is a relaxing change (previously-rejected assessments now pass), but callers caching validation behavior could be affected. CLAUDE.md notes: "Don't change Assessment model without schema version bump". Consider bumping the schema title to v1.1.0 and documenting the change in a changelog entry.

2. No cross-field validation between attributes_total and findings.length

The schema now allows a submission with attributes_total: 22 but only 10 findings items — these are inconsistent but the schema won't catch it. JSON Schema Draft-07 doesn't support cross-field constraints natively, but this could be enforced in application-level validation in SchemaValidator.validate_report():

if data.get("attributes_total") \!= len(data.get("findings", [])):
    errors.append("attributes_total must equal number of findings")

3. Minimum of 10 is undocumented

The minimum was chosen to "ensure meaningful assessments" but this rationale isn't captured anywhere. Why 10 and not 8 or 15? The description field added to attributes_total is a good start but doesn't explain the reasoning. A comment in the linked issue or PR description would help future contributors.

🟢 Suggestions

4. Test fixture assumption

In test_validate_partial_exclusion_issue_309:

valid_report_data["findings"] = valid_report_data["findings"][:22]

This silently produces fewer than 22 findings if the fixture has fewer items. Add a guard:

assert len(valid_report_data["findings"]) >= 22, "Fixture must have >= 22 findings for this test"
valid_report_data["findings"] = valid_report_data["findings"][:22]

5. DRAFT status

Once the issues above are addressed, promote to "Ready for Review" so it can be merged before more --exclude users hit #309.

---

Strengths

• Clear problem statement linking to issue [Bug] JSON schema validation against the Assessment submission which has less than 25 attributes fails hard in the Leaderboard CI #309 with a concrete reproduction case (PR Leaderboard: dbasunag/opendatahub-tests (69.9/100 - Silver) #301).
• The description field added to attributes_total improves schema self-documentation.
• Regression tests cover minimum boundary (10), a real-world case (22), and the below-minimum rejection path (5) — exactly the right test set.
• The fix is minimal and surgical — no unnecessary changes.
• Test docstrings clearly explain the scenario being tested.

---

Verdict

Approve with suggestions — the core fix is correct and the tests are good. The schema version bump (item 1) and cross-field validation (item 2) are worth addressing; the rest are suggestions. This is close to merge-ready once out of DRAFT.

[github-actions]

AgentReady Code Review — PR #312

Reviewed by: Claude Code (AgentReady Review Agent)
Date: 2026-02-19

---

Summary

This PR relaxes the assessment schema's attributes_total constraint from const: 25 to minimum: 10, maximum: 25, and adjusts findings minItems to match. This unblocks users of agentready assess --exclude from submitting leaderboard entries (fixes #309). The change is accompanied by three regression tests.

---

AgentReady Attribute Impact

Attribute	Tier	Direction	Notes
conventional_commits	2	Neutral	Unrelated
Schema contract	—	Breaking relaxation	The leaderboard now accepts partial assessments — see fairness note below

---

Code Quality

Strengths:

• Schema change is minimal and surgical — touches exactly the two fields that need relaxing
• description field added to attributes_total is helpful documentation
• Test for boundary rejection (fewer than 10 attributes) is important — good that it is included
• The regression test for the PR Leaderboard: dbasunag/opendatahub-tests (69.9/100 - Silver) #301 case (22 attributes) documents a concrete real-world scenario

Issues:

1. The minimum of 10 is undocumented and arbitrary (medium)

The PR description says 'at least 10 attributes to ensure meaningful assessments' but there is no rationale in the schema, tests, or any spec file for why 10 is the right floor rather than, say, 15 or 5. This is a business logic decision that should be explicitly justified in the PR description with reasoning, possibly driven by a minimum coverage per tier.

2. Leaderboard fairness — score inflation risk (medium)

This is the most important concern for the long-term health of the leaderboard.

When a user excludes attributes, they are more likely to exclude ones their repo fails. A repo that excludes 15 attributes and passes 10 easy ones could achieve a higher certification level than a repo honestly assessed on all 25.

Suggested mitigations:

• Show (partial: N/25 attributes) on the leaderboard for excluded assessments
• Normalize scores against the full 25-attribute baseline
• Or: cap excluded-attribute assessments from leaderboard submission entirely

This PR does not have to solve all of this, but a follow-up GitHub issue should track the behavior as intentional.

3. Cross-field consistency not enforced (low)

The schema allows inconsistency between fields. Because findings now has minItems: 10 but attributes_total still allows up to 25, JSON Schema draft-07 does not easily express cross-field numeric constraints, so this is a known limitation worth documenting.

---

Test Quality

The three regression tests are well-constructed and cover:

• Boundary minimum (10 attributes) — validates correctly
• Real-world PR Leaderboard: dbasunag/opendatahub-tests (69.9/100 - Silver) #301 case (22 attributes) — validates correctly
• Below-minimum rejection (5 attributes) — correctly rejected

Missing test: What happens when attributes_total=10 but findings has 25 items? This cross-field mismatch should be either rejected or explicitly documented as acceptable.

---

Security

No security concerns. This is a pure JSON Schema change with no code execution path changes.

---

Best Practices

• Fixes a real bug with a minimal schema change
• Adds regression tests for the specific issue case
• Backwards-compatible (existing 25-attribute assessments still validate)
• Minimum value of 10 needs documented justification
• Leaderboard fairness implications need to be tracked as a follow-on issue
• Missing test: cross-field inconsistency between attributes_total and findings count

---

Suggested Follow-up Issue

Create a GitHub issue to track: 'Leaderboard fairness for partial assessments (--exclude)' — document whether partial assessments should be scored, ranked, or labeled differently from full 25-attribute assessments.

---

Verdict

Approve with follow-up — The schema fix is correct and the tests are solid. The core change is safe and backwards-compatible. The leaderboard fairness concern is real but does not need to block this merge — it should be tracked as a follow-up. Add a brief comment in the PR or linked issue explaining the rationale for the minimum of 10.

[kami619]

@dbasunag WDYT ?

[github-actions]

AgentReady Code Review — PR #312

fix(schema): allow assessments with excluded attributes
Fixes #309 | specs/001-agentready-scorer/contracts/assessment-schema.json + tests

---

Summary

This is a clean, minimal fix that directly addresses the schema rejection of valid --exclude assessments. The approach is sound and the test coverage is good. A few issues need attention before merge.

---

Issues

Critical

No consistency constraint between attributes_total and findings count

After this change, the schema accepts attributes_total: 10 with 25 findings, or attributes_total: 22 with only 10 findings. The two fields are entirely decoupled. An attacker could exploit this on the leaderboard — submitting 10-attribute findings but claiming a 25-attribute score context.

Suggestion: add a cross-field validation in agentready validate-report (not just the JSON schema, which cannot enforce cross-field equality natively in draft-07).

Medium

Minimum of 10 is arbitrary and undocumented

The PR says "ensure meaningful assessments" but does not explain why 10 is the threshold. With 25 total attributes, excluding 15 (60%) seems like a very permissive floor. Document the rationale in the schema's description field.

attributes_assessed.maximum still hardcoded at 25

The attributes_assessed property still has "maximum": 25. An assessment with attributes_total: 10 and attributes_assessed: 10 is logically valid, but the existing maximum does not cause a problem today — it will if the overall maximum is ever raised beyond 25.

Conflicts with PR #315

Both PRs solve issue #309 with different approaches:

• This PR: relaxes schema constraints to minimum: 10, maximum: 25
• PR Fix: read attributes_total from the assessment*.json to set the max attribute expectations for schema validation #315: dynamically patches schema in CI to const == attributes_total

They cannot both be merged as-is. PR #315's jq command attempts to re-set .const which this PR removed. One approach must be selected before either is merged.

Minor

PR checklist has multiple unchecked items

"My code follows the project's code style", "I have performed a self-review", "I have commented my code", and "I have made corresponding changes to the documentation" are unchecked.

Test fixture fragility

valid_report_data["findings"][:10] assumes the fixture has >=25 findings. Assert the precondition explicitly:

assert len(valid_report_data["findings"]) >= 25, "Fixture needs at least 25 findings for this test"

---

What's Good

• Schema change is minimal and precisely targeted
• description field added to attributes_total aids tooling clarity
• Three tests covering: min boundary (10), partial exclusion (22), too-few rejection (5)
• Conventional commit message format
• No breaking changes to existing valid 25-attribute assessments

---

Attribute Impact

Attribute	Impact
contract_driven_development	Positive — schema becomes inclusive of real-world --exclude usage
test_coverage	Neutral — good new tests but missing cross-field consistency coverage

---

Verdict: Needs revision. The critical gap is the lack of consistency enforcement between attributes_total and findings count. Recommend coordinating with PR #315 to agree on a single approach before merging either.