Leaderboard: redhat/rhel-ai AIPCC productization repos (13 repos)#347
Merged
kami619 merged 1 commit intoambient-code:mainfrom Mar 26, 2026
Conversation
- aipcc-claudio: 44.5/100 (Bronze) - aipcc-infrastructure: 27.6/100 (Needs Improvement) - aipcc-product-management: 72.2/100 (Silver) - aipcc-product-management-configs: 49.3/100 (Bronze) - central-linter: 45.9/100 (Bronze) - dashboard: 58.5/100 (Bronze) - docs: 60.9/100 (Silver) - toolbox: 24.8/100 (Needs Improvement) - bootc: 34.1/100 (Needs Improvement) - bootc-test: 26.7/100 (Needs Improvement) - disk-image-test: 26.7/100 (Needs Improvement) - konflux-data: 15.3/100 (Needs Improvement) - renovate-config: 24.1/100 (Needs Improvement) Repos are hosted on GitLab (gitlab.com/redhat/rhel-ai/...) assessed using the bug bash standard config. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Jakub Rusz <jrusz@redhat.com>
|
Warning
|
| Cohort / File(s) | Summary |
|---|---|
Assessment Reports submissions/redhat/aipcc-claudio/2026-03-24T17-51-33-assessment.json, submissions/redhat/aipcc-infrastructure/2026-03-24T17-58-24-assessment.json, submissions/redhat/aipcc-product-management-configs/2026-03-24T17-58-25-assessment.json, submissions/redhat/aipcc-product-management/2026-03-24T17-58-26-assessment.json, submissions/redhat/bootc-test/2026-03-24T17-58-26-assessment.json, submissions/redhat/bootc/2026-03-24T17-58-24-assessment.json, submissions/redhat/central-linter/2026-03-24T17-58-23-assessment.json, submissions/redhat/dashboard/2026-03-24T17-58-23-assessment.json, submissions/redhat/disk-image-test/2026-03-24T17-58-27-assessment.json, submissions/redhat/docs/2026-03-24T17-58-26-assessment.json, submissions/redhat/konflux-data/2026-03-24T17-58-24-assessment.json, submissions/redhat/renovate-config/2026-03-24T17-58-23-assessment.json, submissions/redhat/toolbox/2026-03-24T17-58-26-assessment.json |
New assessment report JSON files capturing agentready evaluation results. Each contains schema version, execution metadata, repository details, overall scoring/certification level, and a findings array documenting per-attribute pass/fail/not\_applicable status with measured values, thresholds, evidence, and remediation guidance. |
Estimated code review effort
🎯 1 (Trivial) | ⏱️ ~3 minutes
🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
| Check name | Status | Explanation |
|---|---|---|
| Title check | ✅ Passed | The title accurately reflects the main change—submitting 13 Red Hat RHEL AI AIPCC productization repository assessments to the leaderboard with scores and tiers. |
| Description check | ✅ Passed | The description directly relates to the changeset by documenting the leaderboard submission of 13 GitLab repositories with their assessment scores, organization context, and validation details. |
| Docstring Coverage | ✅ Passed | No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. |
✏️ Tip: You can configure your own custom pre-merge checks in the settings.
✨ Finishing Touches
🧪 Generate unit tests (beta)
- Create PR with unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands and usage tips.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 24
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@submissions/redhat/aipcc-infrastructure/2026-03-24T17-58-24-assessment.json`:
- Around line 260-318: The dependency_security attribute ("id":
"dependency_security") is being marked as status: "pass" while score: 35 but
threshold: "≥60 points" — fix the scoring logic in the assessment pipeline
(functions like determineAttributeStatus or calculateAttributeScore) so the
pass/fail decision uses the numeric threshold parsed from the
attribute.threshold (e.g., parse "≥60" to 60) and sets status = "pass" only when
score >= thresholdNumeric; alternatively, if the intended pass mark is 35,
update the attribute.threshold string to reflect the actual numeric threshold
and ensure both calculateAttributeScore and determineAttributeStatus use the
same source of truth for threshold values.
In
`@submissions/redhat/aipcc-product-management/2026-03-24T17-58-26-assessment.json`:
- Around line 218-266: The JSON entries have mismatched semantics: fields
"status", "score", and "threshold" are inconsistent (e.g., measured_value
"Security tools configured: Renovate" shows score 35 but status "pass" while
threshold is "≥60 points"); update the logic that populates these report items
so "status" is derived from comparing numeric score >= numeric threshold (parse
the "threshold" string like "≥60 points" to 60) and set status to "pass" only
when score >= threshold otherwise "fail" (or an agreed alternate like "warn");
ensure "remediation" is present only when status is not passing; apply the same
fix to all analogous entries (fields "status", "score", "threshold",
"remediation", e.g., the Renovate/dependency-security item and the
container-setup item referenced in the comment).
- Around line 8-10: When serializing/publishing the assessment JSON, redact or
normalize environment-identifying fields so they do not expose runner details:
replace executed_by with a generic identifier (e.g., "anonymous" or null),
normalize command by stripping absolute binary paths and local workspace paths
(remove "/usr/local/bin/" and "/repo" or replace them with "<tool>" and
"<workspace>"), and remove or set working_directory to a normalized value (e.g.,
"<workspace>"). Locate the code that writes or commits the report and update the
serializer/mapper for the keys "executed_by", "command", and "working_directory"
to apply these transformations before saving or committing the report.
- Around line 229-233: The remediation text currently emits GitHub-specific
suggestions such as "Enable Dependabot alerts in GitHub repository settings (or
configure Renovate: add renovate.json to repository root)" and "Add CodeQL
scanning workflow for SAST" even for GitLab targets; update the remediation
generation logic (e.g., the function that emits remediation lists, such as
generateRemediations/buildRemediationList) to detect the repository host from
the repository URL (gitlab.com vs github.com) and emit platform-appropriate
guidance: for GitLab replace GitHub items with GitLab equivalents (enable
Security & Compliance/Dependency Scanning, add GitLab CI templates in
.gitlab-ci.yml, use GitLab Secret Detection/SAST, or reference Renovate/GitLab
integration docs), and ensure the same change is applied wherever the same
GitHub-only strings appear (the entries matching the four GitHub strings shown).
In `@submissions/redhat/bootc/2026-03-24T17-58-24-assessment.json`:
- Line 698: The duration_seconds field is being set to 0.0; update the code that
populates "duration_seconds" to compute a real runtime instead of writing a
constant zero by calculating the difference between the assessment start and end
timestamps (e.g., parse start_time and end_time or equivalent timestamp fields)
or by using the measured execution timer used by the assessment runner, then
serialize that computed difference (in seconds) into duration_seconds; ensure
this logic runs for all code paths that emit the assessment JSON so
duration_seconds reflects actual elapsed time rather than 0.0.
- Line 8: The metadata field executed_by contains a personal identifier and must
be redacted or pseudonymized; update the JSON so executed_by no longer contains
an email or user-specific token (e.g., replace the value with a neutral string
like "REDACTED", an anonymized id such as "operator-<hash>", or a CI/system
account), ensure the change is applied where executed_by appears, and commit the
sanitized value so no personal identifiers remain in the artifact.
- Around line 263-283: The guidance is GitHub-specific (mentions "Enable
Dependabot alerts", "CodeQL", ".github/dependabot.yml", "gh repo edit
--enable-security", etc.) but the repo is on GitLab; replace those
GitHub-specific remediation steps with GitLab-native equivalents: recommend
GitLab Managed SAST and Dependency Scanning, Secret Detection, and CI-based
scanners by referencing GitLab CI templates (include
Security/SAST.gitlab-ci.yml, Security/Dependency-Scanning.gitlab-ci.yml),
explain enabling Security & Compliance features in project settings and using
project CI/CD variables for secrets, update the "tools" list (e.g., GitLab SAST,
GitLab Dependency Scanning, GitLab Secret Detection, detect-secrets/gitleaks as
CI jobs) and replace "commands" and "examples" (swap .github/dependabot.yml
sample for a .gitlab-ci.yml include example and show how to enable scanning via
the GitLab UI/API); apply the same replacement wherever the GitHub-specific
items (Dependabot, CodeQL, .github, gh commands) appear.
- Around line 252-256: The assessment entry shows an inconsistent state:
"status" is "pass" while "score": 35 is below the declared "threshold": "≥60",
so update the entry to a correct outcome by setting "status" to "fail" (or
increase the numeric score to meet the threshold if that is the intended
change); specifically modify the JSON object containing the fields "status",
"score", "threshold", and "measured_value" so that the boolean/pass-fail status
aligns with the numeric comparison (score >= 60) and ensure downstream
aggregation logic uses the corrected "status" field.
In `@submissions/redhat/central-linter/2026-03-24T17-58-23-assessment.json`:
- Around line 14-15: The JSON uses a generic "name": "repo" which should be
replaced with the canonical repository name to ensure traceability; update the
"name" field value from "repo" to "central-linter" (use the repo indicated by
the "url" field) so records are attributable and maintainable.
- Around line 287-291: The JSON entry is inconsistent: "status" is "pass" while
"score": 35 is below the declared "threshold": "≥60 points"; update the record
so status reflects the numeric check (e.g., set "status" to "fail" when score <
60) and ensure the fields "status", "score", "measured_value", and "threshold"
in this object are mutually consistent (or recalculate score/threshold if
intended), using the keys "status", "score", "measured_value", and "threshold"
to locate and fix the entry.
- Around line 731-735: The JSON block has an inconsistent outcome: "status" is
"pass" while "score": 40 is below the declared "threshold": "≥70 points"; update
the JSON so status reflects the numeric comparison—either raise "score" to meet
the threshold or set "status" to "fail" (and adjust any related summary fields)
so that "status", "score", and "threshold" are consistent; locate the object
containing the keys "status", "score", and "threshold" in the submission record
and make the change there.
- Around line 8-10: The published artifact contains sensitive runtime metadata:
remove or redact the "executed_by" value and avoid embedding local paths in
"command" and "working_directory"; specifically, in the JSON that includes the
"executed_by", "command", and "working_directory" fields, replace the executor
identity with a generic token (e.g., "executor": "redacted" or null) and rewrite
the "command" and "working_directory" values to non-identifying, portable
placeholders (e.g., "command": "agentready assess /repo --config
/agentready-config.yml --output-dir /reports" without absolute local paths or
use "<redacted-path>") or remove those keys entirely if not required for
consumers. Ensure any change preserves valid JSON shape and update any consumer
code expecting these keys to handle redacted/absent values.
In `@submissions/redhat/dashboard/2026-03-24T17-58-23-assessment.json`:
- Around line 8-10: The report contains sensitive environment/user metadata —
specifically the JSON fields "executed_by", "command", and "working_directory"
(and any internal clone URL strings) — which must be redacted before committing;
update the report-generation or post-processing step that writes the assessment
JSON so it strips or masks these keys (e.g., replace values with
null/"[REDACTED]" or remove the keys) for "executed_by", "command",
"working_directory" and any clone URL patterns, and ensure the sanitizer runs on
files like the assessment JSON(s) prior to saving/committing.
- Around line 258-262: The dependency_security result is inconsistent: the JSON
object has "status": "pass" but "score": 35 which is below the "threshold": "≥60
points"; update the object so status matches the numeric threshold—either raise
"score" to meet or exceed 60 (and adjust "measured_value"/"evidence"
accordingly) or change "status" to "fail" (and optionally add an explanation in
"evidence"); locate the dependency_security assessment JSON block (keys
"status", "score", "threshold", "measured_value", "evidence") and make the
status/score consistent.
- Around line 479-483: The JSON entry for inline_documentation has inconsistent
fields: "status" is "pass" while "measured_value" ("66.2%") is below the
"threshold" ("≥80%"); update the inline_documentation object so the semantic
state matches the metric—either set "status" to "fail" (preferred) or correct
"measured_value"/"threshold" if the numbers were wrong; locate the
inline_documentation record (fields "status", "measured_value", "threshold",
"evidence") and make the change so the status accurately reflects the comparison
between measured_value and threshold.
In `@submissions/redhat/disk-image-test/2026-03-24T17-58-27-assessment.json`:
- Around line 260-263: The remediation text uses GitHub-specific instructions
(e.g., "Enable Dependabot", ".github/dependabot.yml", "gh repo edit", "gh
workflow view") but the assessed repository is on GitLab; update the remediation
generator to detect Git provider and emit GitLab-equivalent guidance: replace
".github/*" paths with GitLab file locations (e.g., configure Dependency
Scanning via .gitlab-ci.yml or use GitLab's Dependency Scanning/Dependency List
features), swap "Enable Dependabot" / GitHub Actions suggestions for
GitLab-native options (use GitLab Dependency Scanning, SAST via GitLab CI, or
recommend Renovate which supports GitLab using renovate.json and GitLab
integration), and remove or replace any "gh" CLI commands with GitLab API/CLI or
UI instructions; apply the same changes for the other occurrences flagged (the
other remediation blocks containing the same GitHub-only strings).
- Around line 13-15: The repository metadata uses a generic name ("name":
"repo"); update the repository.name field to a descriptive, unique identifier
derived from the URL (e.g., "disk-image-test") so downstream aggregation and
leaderboards can reliably trace artifacts; locate the JSON block containing the
"path", "name", and "url" keys and replace the value of "name" with the
repository-specific identifier matching the "url".
- Around line 8-10: Sanitize or remove runtime identity and environment fields
in the assessment JSON by redacting the "executed_by", "command", and
"working_directory" values (or omitting these keys) before committing; replace
them with generic placeholders (e.g., "redacted" or null) or remove the keys
entirely, and ensure any code that writes the assessment (the logic producing
the JSON containing executed_by/command/working_directory — e.g., the agentready
invocation metadata generator) applies this redaction step so operator/runtime
details are never persisted.
In `@submissions/redhat/docs/2026-03-24T17-58-26-assessment.json`:
- Around line 118-134: The "standard_layout" rule is being applied to
Markdown-only repos causing false failures; update the assessment logic that
evaluates the "standard_layout" rule (rule id "standard_layout") to first detect
Markdown-only repositories (e.g., repository file-type scan shows only
.md/.markdown and docs/) and short-circuit the check for such repos by marking
the rule as "skipped" (or clear pass) with an explanatory evidence message
instead of failing; ensure the output updates the "status", "score", and
"evidence" fields for "standard_layout" to reflect the skip and include the
detection reason so markdown-only repos are not penalized.
- Around line 203-207: Several assessment entries show "status": "pass" while
"score" (e.g., 35) is below the stated "threshold" (≥60), so update the
assessment semantics by setting "status" to "fail" for any record where numeric
"score" < 60; locate the JSON objects with the fields "status", "score",
"threshold", and "measured_value" (like the example entry) and change "status":
"pass" to "status": "fail" for those items (apply the same fix across the 31+
affected assessment files). Ensure the change is applied consistently and only
for records where the parsed numeric threshold (60) is not met.
- Around line 8-10: Remove or sanitize sensitive runtime metadata currently
stored under the JSON keys "executed_by", "command", and "working_directory" in
the generated assessment report: stop writing executor identity and local
execution context into the report (or replace values with generic placeholders
like "REDACTED" or remove keys entirely) at the point where the report is
serialized; update the report-generation/serialization logic that emits those
keys so public artifacts no longer include user/host or local path information
and ensure any tests or consumers expect the redacted/absent fields.
In `@submissions/redhat/renovate-config/2026-03-24T17-58-23-assessment.json`:
- Around line 272-276: The JSON entry is internally inconsistent: "status" is
set to "pass" while "score": 35 is below the declared "threshold": "≥60"; update
the artifact so these fields align — set "status" to "fail" (or raise "score" to
meet the threshold if that is the true result), and ensure "threshold" and
"measured_value" remain accurate; specifically edit the "status", "score",
and/or "threshold" fields in the affected JSON object so that status reflects
the numeric comparison (e.g., keep "score": 35 and change "status" to "fail").
- Line 8: Remove the sensitive executor fingerprint by editing the JSON metadata
key "executed_by": either delete the "executed_by" field entirely or replace its
value with a non-identifying token (e.g., "REDACTED" or null) so no
user/host-style identifier (like "jrusz@a6bb48cac2f6") remains; ensure any code
that reads "executed_by" can handle its absence or the redacted value
gracefully.
In `@submissions/redhat/toolbox/2026-03-24T17-58-26-assessment.json`:
- Line 784: The assessment output shows "duration_seconds": 0.0; locate the code
that constructs the assessment object (the place that writes the
"duration_seconds" field, e.g., where start_time/end_time or
startTimestamp/endTimestamp are used) and compute duration as a floating-point
seconds value using high-resolution timing (process.hrtime or millisecond
timestamps via Date.now), e.g., (end - start)/1000.0, ensuring you don't
accidentally floor or cast to an integer; update the serialization so
"duration_seconds" is set from this computed float and add a fallback to null if
timestamps are missing rather than 0.0.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 1bb95f29-bbf9-46d0-b067-9faafb915a84
📒 Files selected for processing (13)
submissions/redhat/aipcc-claudio/2026-03-24T17-51-33-assessment.jsonsubmissions/redhat/aipcc-infrastructure/2026-03-24T17-58-24-assessment.jsonsubmissions/redhat/aipcc-product-management-configs/2026-03-24T17-58-25-assessment.jsonsubmissions/redhat/aipcc-product-management/2026-03-24T17-58-26-assessment.jsonsubmissions/redhat/bootc-test/2026-03-24T17-58-26-assessment.jsonsubmissions/redhat/bootc/2026-03-24T17-58-24-assessment.jsonsubmissions/redhat/central-linter/2026-03-24T17-58-23-assessment.jsonsubmissions/redhat/dashboard/2026-03-24T17-58-23-assessment.jsonsubmissions/redhat/disk-image-test/2026-03-24T17-58-27-assessment.jsonsubmissions/redhat/docs/2026-03-24T17-58-26-assessment.jsonsubmissions/redhat/konflux-data/2026-03-24T17-58-24-assessment.jsonsubmissions/redhat/renovate-config/2026-03-24T17-58-23-assessment.jsonsubmissions/redhat/toolbox/2026-03-24T17-58-26-assessment.json
Comment on lines
+260
to
+318
| { | ||
| "attribute": { | ||
| "id": "dependency_security", | ||
| "name": "Dependency Security & Vulnerability Scanning", | ||
| "category": "Security", | ||
| "tier": 1, | ||
| "description": "Security scanning tools configured for dependencies and code", | ||
| "criteria": "Dependabot, Renovate, CodeQL, or SAST tools configured; secret detection enabled", | ||
| "default_weight": 0.04 | ||
| }, | ||
| "status": "pass", | ||
| "score": 35, | ||
| "measured_value": "Security tools configured: Renovate", | ||
| "threshold": "\u226560 points (Dependabot/Renovate + SAST or multiple scanners)", | ||
| "evidence": [ | ||
| "\u2713 Renovate configured for dependency updates", | ||
| " Meaningful Renovate configuration detected" | ||
| ], | ||
| "remediation": { | ||
| "summary": "Add more security scanning tools for comprehensive coverage", | ||
| "steps": [ | ||
| "Enable Dependabot alerts in GitHub repository settings (or configure Renovate: add renovate.json to repository root)", | ||
| "Add CodeQL scanning workflow for SAST", | ||
| "Configure secret detection (detect-secrets, gitleaks)", | ||
| "Set up language-specific scanners (pip-audit, npm audit, Snyk)" | ||
| ], | ||
| "tools": [ | ||
| "Dependabot", | ||
| "Renovate", | ||
| "CodeQL", | ||
| "detect-secrets", | ||
| "pip-audit", | ||
| "npm audit" | ||
| ], | ||
| "commands": [ | ||
| "gh repo edit --enable-security", | ||
| "pip install detect-secrets # Python secret detection", | ||
| "npm audit # JavaScript dependency audit" | ||
| ], | ||
| "examples": [ | ||
| "# .github/dependabot.yml\nversion: 2\nupdates:\n - package-ecosystem: pip\n directory: /\n schedule:\n interval: weekly" | ||
| ], | ||
| "citations": [ | ||
| { | ||
| "source": "OWASP", | ||
| "title": "Dependency-Check Project", | ||
| "url": "https://owasp.org/www-project-dependency-check/", | ||
| "relevance": "Open-source tool for detecting known vulnerabilities in dependencies" | ||
| }, | ||
| { | ||
| "source": "GitHub", | ||
| "title": "Dependabot Documentation", | ||
| "url": "https://docs.github.com/en/code-security/dependabot", | ||
| "relevance": "Official guide for configuring automated dependency updates and security alerts" | ||
| } | ||
| ] | ||
| }, | ||
| "error_message": null | ||
| }, |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check if other assessment files have the same pattern
fd -e json . submissions/ --exec jq -r 'select(.findings[]? | select(.attribute.id == "dependency_security" and .status == "pass" and .score < 60)) | "File: \(.metadata.assessment_timestamp) Score: \(.findings[] | select(.attribute.id == "dependency_security") | .score)"'Repository: ambient-code/agentready
Length of output: 969
Fix scoring logic for dependency_security assessment attribute.
The dependency_security finding shows status: "pass" with a score of 35 across multiple assessment files, but the threshold field consistently states "≥60 points". This logical inconsistency appears in at least 21 assessment files and indicates a systemic issue in the assessment tool's scoring logic.
The pass threshold appears to be lower than documented (allowing scores as low as 35), or the threshold definition is incorrect. This misleads users about whether the repository actually meets the security criteria.
Investigate the scoring logic in the assessment tool to ensure the pass/fail classification aligns with the documented thresholds. Either adjust the scoring boundaries to match the threshold requirements (minimum 60 points for pass), or update the threshold documentation to reflect the actual pass threshold being used.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@submissions/redhat/aipcc-infrastructure/2026-03-24T17-58-24-assessment.json`
around lines 260 - 318, The dependency_security attribute ("id":
"dependency_security") is being marked as status: "pass" while score: 35 but
threshold: "≥60 points" — fix the scoring logic in the assessment pipeline
(functions like determineAttributeStatus or calculateAttributeScore) so the
pass/fail decision uses the numeric threshold parsed from the
attribute.threshold (e.g., parse "≥60" to 60) and sets status = "pass" only when
score >= thresholdNumeric; alternatively, if the intended pass mark is 35,
update the attribute.threshold string to reflect the actual numeric threshold
and ensure both calculateAttributeScore and determineAttributeStatus use the
same source of truth for threshold values.
Comment on lines
+8
to
+10
| "executed_by": "jrusz@729a3c51fb68", | ||
| "command": "/usr/local/bin/agentready assess /repo --config /agentready-config.yml --output-dir /reports", | ||
| "working_directory": "/tmp" |
There was a problem hiding this comment.
Remove environment-identifying metadata from published report artifacts.
executed_by, absolute command path, working directory, and local path (/repo) expose runner/environment details that are unnecessary for leaderboard output and increase disclosure risk. Redact or normalize these fields before committing reports.
Suggested redaction pattern
- "executed_by": "jrusz@729a3c51fb68",
- "command": "/usr/local/bin/agentready assess /repo --config /agentready-config.yml --output-dir /reports",
- "working_directory": "/tmp"
+ "executed_by": "redacted",
+ "command": "agentready assess <repo> --config <config> --output-dir <output>",
+ "working_directory": "redacted"
...
- "path": "/repo",
+ "path": "<repo>"As per coding guidelines, "-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity."
Also applies to: 13-15
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In
`@submissions/redhat/aipcc-product-management/2026-03-24T17-58-26-assessment.json`
around lines 8 - 10, When serializing/publishing the assessment JSON, redact or
normalize environment-identifying fields so they do not expose runner details:
replace executed_by with a generic identifier (e.g., "anonymous" or null),
normalize command by stripping absolute binary paths and local workspace paths
(remove "/usr/local/bin/" and "/repo" or replace them with "<tool>" and
"<workspace>"), and remove or set working_directory to a normalized value (e.g.,
"<workspace>"). Locate the code that writes or commits the report and update the
serializer/mapper for the keys "executed_by", "command", and "working_directory"
to apply these transformations before saving or committing the report.
Comment on lines
+218
to
+266
| "status": "pass", | ||
| "score": 35, | ||
| "measured_value": "Security tools configured: Renovate", | ||
| "threshold": "\u226560 points (Dependabot/Renovate + SAST or multiple scanners)", | ||
| "evidence": [ | ||
| "\u2713 Renovate configured for dependency updates", | ||
| " Meaningful Renovate configuration detected" | ||
| ], | ||
| "remediation": { | ||
| "summary": "Add more security scanning tools for comprehensive coverage", | ||
| "steps": [ | ||
| "Enable Dependabot alerts in GitHub repository settings (or configure Renovate: add renovate.json to repository root)", | ||
| "Add CodeQL scanning workflow for SAST", | ||
| "Configure secret detection (detect-secrets, gitleaks)", | ||
| "Set up language-specific scanners (pip-audit, npm audit, Snyk)" | ||
| ], | ||
| "tools": [ | ||
| "Dependabot", | ||
| "Renovate", | ||
| "CodeQL", | ||
| "detect-secrets", | ||
| "pip-audit", | ||
| "npm audit" | ||
| ], | ||
| "commands": [ | ||
| "gh repo edit --enable-security", | ||
| "pip install detect-secrets # Python secret detection", | ||
| "npm audit # JavaScript dependency audit" | ||
| ], | ||
| "examples": [ | ||
| "# .github/dependabot.yml\nversion: 2\nupdates:\n - package-ecosystem: pip\n directory: /\n schedule:\n interval: weekly" | ||
| ], | ||
| "citations": [ | ||
| { | ||
| "source": "OWASP", | ||
| "title": "Dependency-Check Project", | ||
| "url": "https://owasp.org/www-project-dependency-check/", | ||
| "relevance": "Open-source tool for detecting known vulnerabilities in dependencies" | ||
| }, | ||
| { | ||
| "source": "GitHub", | ||
| "title": "Dependabot Documentation", | ||
| "url": "https://docs.github.com/en/code-security/dependabot", | ||
| "relevance": "Official guide for configuring automated dependency updates and security alerts" | ||
| } | ||
| ] | ||
| }, | ||
| "error_message": null | ||
| }, |
There was a problem hiding this comment.
Fix inconsistent finding semantics (status vs score vs threshold).
At least two entries are marked status: "pass" while their scores are below threshold and remediation is present (e.g., dependency security: 35 vs threshold ≥60; container setup: 40 vs threshold ≥70). This inconsistency can break downstream aggregation and ranking logic.
Also applies to: 609-648
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In
`@submissions/redhat/aipcc-product-management/2026-03-24T17-58-26-assessment.json`
around lines 218 - 266, The JSON entries have mismatched semantics: fields
"status", "score", and "threshold" are inconsistent (e.g., measured_value
"Security tools configured: Renovate" shows score 35 but status "pass" while
threshold is "≥60 points"); update the logic that populates these report items
so "status" is derived from comparing numeric score >= numeric threshold (parse
the "threshold" string like "≥60 points" to 60) and set status to "pass" only
when score >= threshold otherwise "fail" (or an agreed alternate like "warn");
ensure "remediation" is present only when status is not passing; apply the same
fix to all analogous entries (fields "status", "score", "threshold",
"remediation", e.g., the Renovate/dependency-security item and the
container-setup item referenced in the comment).
Comment on lines
+229
to
+233
| "Enable Dependabot alerts in GitHub repository settings (or configure Renovate: add renovate.json to repository root)", | ||
| "Add CodeQL scanning workflow for SAST", | ||
| "Configure secret detection (detect-secrets, gitleaks)", | ||
| "Set up language-specific scanners (pip-audit, npm audit, Snyk)" | ||
| ], |
There was a problem hiding this comment.
Make remediation platform-aware for GitLab repositories.
This report targets a GitLab repo (gitlab.com/...), but remediation suggests GitHub-specific actions (gh repo edit, .github/workflows, Dependabot docs). That makes remediation partially non-actionable for this submission and lowers report maintainability/usefulness.
As per coding guidelines, "-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity."
Also applies to: 243-249, 495-525
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In
`@submissions/redhat/aipcc-product-management/2026-03-24T17-58-26-assessment.json`
around lines 229 - 233, The remediation text currently emits GitHub-specific
suggestions such as "Enable Dependabot alerts in GitHub repository settings (or
configure Renovate: add renovate.json to repository root)" and "Add CodeQL
scanning workflow for SAST" even for GitLab targets; update the remediation
generation logic (e.g., the function that emits remediation lists, such as
generateRemediations/buildRemediationList) to detect the repository host from
the repository URL (gitlab.com vs github.com) and emit platform-appropriate
guidance: for GitLab replace GitHub items with GitLab equivalents (enable
Security & Compliance/Dependency Scanning, add GitLab CI templates in
.gitlab-ci.yml, use GitLab Secret Detection/SAST, or reference Renovate/GitLab
integration docs), and ensure the same change is applied wherever the same
GitHub-only strings appear (the entries matching the four GitHub strings shown).
| "research_version": "1.0.1", | ||
| "assessment_timestamp": "2026-03-24T17:58:24.843049", | ||
| "assessment_timestamp_human": "March 24, 2026 at 5:58 PM", | ||
| "executed_by": "jrusz@1f7862a33abe", |
There was a problem hiding this comment.
Remove personal identifier from report metadata.
executed_by includes a user-identifying value (jrusz@...) in a committed artifact. Please redact or pseudonymize this field to avoid leaking operator identity in public submissions.
As per coding guidelines, this is a major security/compliance concern and should be addressed over stylistic issues.
Suggested fix
- "executed_by": "jrusz@1f7862a33abe",
+ "executed_by": "redacted",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| "executed_by": "jrusz@1f7862a33abe", | |
| "executed_by": "redacted", |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@submissions/redhat/bootc/2026-03-24T17-58-24-assessment.json` at line 8, The
metadata field executed_by contains a personal identifier and must be redacted
or pseudonymized; update the JSON so executed_by no longer contains an email or
user-specific token (e.g., replace the value with a neutral string like
"REDACTED", an anonymized id such as "operator-<hash>", or a CI/system account),
ensure the change is applied where executed_by appears, and commit the sanitized
value so no personal identifiers remain in the artifact.
Comment on lines
+118
to
+134
| "id": "standard_layout", | ||
| "name": "Standard Project Layouts", | ||
| "category": "Repository Structure", | ||
| "tier": 1, | ||
| "description": "Follows standard project structure for language", | ||
| "criteria": "Standard directories (src/ or project-named, tests/) present", | ||
| "default_weight": 0.1 | ||
| }, | ||
| "status": "fail", | ||
| "score": 0.0, | ||
| "measured_value": "0/2 directories", | ||
| "threshold": "2/2 directories", | ||
| "evidence": [ | ||
| "Found 0/2 standard directories", | ||
| "source directory: \u2717 (no src/ or project-named dir)", | ||
| "tests/: \u2717" | ||
| ], |
There was a problem hiding this comment.
Avoid applying code-layout checks to Markdown-only repositories.
Line 111 indicates a Markdown-only repo, but Line 126-Line 134 fail standard_layout for missing src//tests/. This is a rule-targeting defect and materially lowers scoring accuracy for documentation repos.
As per coding guidelines, "Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity."
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@submissions/redhat/docs/2026-03-24T17-58-26-assessment.json` around lines 118
- 134, The "standard_layout" rule is being applied to Markdown-only repos
causing false failures; update the assessment logic that evaluates the
"standard_layout" rule (rule id "standard_layout") to first detect Markdown-only
repositories (e.g., repository file-type scan shows only .md/.markdown and
docs/) and short-circuit the check for such repos by marking the rule as
"skipped" (or clear pass) with an explanatory evidence message instead of
failing; ensure the output updates the "status", "score", and "evidence" fields
for "standard_layout" to reflect the skip and include the detection reason so
markdown-only repos are not penalized.
Comment on lines
+203
to
+207
| "status": "pass", | ||
| "score": 35, | ||
| "measured_value": "Security tools configured: Renovate", | ||
| "threshold": "\u226560 points (Dependabot/Renovate + SAST or multiple scanners)", | ||
| "evidence": [ |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify inconsistent findings across submitted assessment reports:
# status=pass while score < numeric threshold in "threshold" text.
python - <<'PY'
import json, re, subprocess
files = subprocess.check_output(
["fd", "-t", "f", "assessment.json", "submissions"]
).decode().strip().splitlines()
bad = []
for f in files:
with open(f, "r", encoding="utf-8") as fh:
data = json.load(fh)
for idx, finding in enumerate(data.get("findings", []), 1):
status = finding.get("status")
score = finding.get("score")
threshold = finding.get("threshold")
if status != "pass" or not isinstance(score, (int, float)) or not isinstance(threshold, str):
continue
m = re.search(r'(\d+(?:\.\d+)?)', threshold.replace(',', ''))
if not m:
continue
threshold_num = float(m.group(1))
if score < threshold_num:
bad.append((f, idx, finding.get("attribute", {}).get("id"), score, threshold))
if not bad:
print("No pass/score/threshold inconsistencies found.")
else:
print("Found inconsistencies:")
for row in bad:
print(f"{row[0]} | finding #{row[1]} | {row[2]} | score={row[3]} | threshold={row[4]}")
PYRepository: ambient-code/agentready
Length of output: 5872
🏁 Script executed:
#!/bin/bash
# Verify the exact lines and content in the specific file mentioned
FILE="submissions/redhat/docs/2026-03-24T17-58-26-assessment.json"
# Check if file exists and get its size
if [ -f "$FILE" ]; then
echo "File found. Line count:"
wc -l "$FILE"
echo ""
echo "Lines 200-210:"
sed -n '200,210p' "$FILE"
echo ""
echo "Full finding `#6` context (searching for finding containing 'dependency_security'):"
python3 - <<'PY'
import json
with open("submissions/redhat/docs/2026-03-24T17-58-26-assessment.json", "r") as f:
data = json.load(f)
for idx, finding in enumerate(data.get("findings", []), 1):
if finding.get("attribute", {}).get("id") == "dependency_security":
print(f"Finding #{idx} (dependency_security):")
print(json.dumps(finding, indent=2))
PY
else
echo "File not found at: $FILE"
fiRepository: ambient-code/agentready
Length of output: 2917
Fix inconsistent finding semantics (pass with score below threshold).
Lines 203–207 mark this finding as pass, but the score (35) is below the stated threshold (≥60 points). This inconsistency appears across 31+ assessment files and can mislead leaderboard consumers. Correct the status to reflect actual threshold compliance.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@submissions/redhat/docs/2026-03-24T17-58-26-assessment.json` around lines 203
- 207, Several assessment entries show "status": "pass" while "score" (e.g., 35)
is below the stated "threshold" (≥60), so update the assessment semantics by
setting "status" to "fail" for any record where numeric "score" < 60; locate the
JSON objects with the fields "status", "score", "threshold", and
"measured_value" (like the example entry) and change "status": "pass" to
"status": "fail" for those items (apply the same fix across the 31+ affected
assessment files). Ensure the change is applied consistently and only for
records where the parsed numeric threshold (60) is not met.
| "research_version": "1.0.1", | ||
| "assessment_timestamp": "2026-03-24T17:58:23.905718", | ||
| "assessment_timestamp_human": "March 24, 2026 at 5:58 PM", | ||
| "executed_by": "jrusz@a6bb48cac2f6", |
There was a problem hiding this comment.
Remove executor fingerprint from committed assessment metadata.
executed_by currently exposes a user/host-style identifier (jrusz@a6bb48cac2f6). For public submission artifacts, this is unnecessary metadata and a privacy/security risk. Prefer a redacted or non-identifying value.
As per coding guidelines, "Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity."
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@submissions/redhat/renovate-config/2026-03-24T17-58-23-assessment.json` at
line 8, Remove the sensitive executor fingerprint by editing the JSON metadata
key "executed_by": either delete the "executed_by" field entirely or replace its
value with a non-identifying token (e.g., "REDACTED" or null) so no
user/host-style identifier (like "jrusz@a6bb48cac2f6") remains; ensure any code
that reads "executed_by" can handle its absence or the redacted value
gracefully.
Comment on lines
+272
to
+276
| "status": "pass", | ||
| "score": 35, | ||
| "measured_value": "Security tools configured: Renovate", | ||
| "threshold": "\u226560 points (Dependabot/Renovate + SAST or multiple scanners)", | ||
| "evidence": [ |
There was a problem hiding this comment.
Inconsistent pass/fail state for dependency_security must be corrected.
This finding is internally contradictory (status: pass with score: 35 against a ≥60 threshold). That breaks report integrity and can skew leaderboard aggregation. Please regenerate/fix this artifact so status, score, and threshold are logically consistent.
As per coding guidelines, "Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity."
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@submissions/redhat/renovate-config/2026-03-24T17-58-23-assessment.json`
around lines 272 - 276, The JSON entry is internally inconsistent: "status" is
set to "pass" while "score": 35 is below the declared "threshold": "≥60"; update
the artifact so these fields align — set "status" to "fail" (or raise "score" to
meet the threshold if that is the true result), and ensure "threshold" and
"measured_value" remain accurate; specifically edit the "status", "score",
and/or "threshold" fields in the affected JSON object so that status reflects
the numeric comparison (e.g., keep "score": 35 and change "status" to "fail").
| "report_theme": "default", | ||
| "custom_theme": null | ||
| }, | ||
| "duration_seconds": 0.0, |
There was a problem hiding this comment.
Suspicious zero-duration assessment.
The assessment reports duration_seconds: 0.0, which seems unlikely for a repository assessment involving file scanning and analysis. This could indicate a timing measurement bug in the assessment tool or extreme rounding. While this doesn't affect the assessment results themselves, it suggests a data quality issue worth investigating.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@submissions/redhat/toolbox/2026-03-24T17-58-26-assessment.json` at line 784,
The assessment output shows "duration_seconds": 0.0; locate the code that
constructs the assessment object (the place that writes the "duration_seconds"
field, e.g., where start_time/end_time or startTimestamp/endTimestamp are used)
and compute duration as a floating-point seconds value using high-resolution
timing (process.hrtime or millisecond timestamps via Date.now), e.g., (end -
start)/1000.0, ensuring you don't accidentally floor or cast to an integer;
update the serialization so "duration_seconds" is set from this computed float
and add a fallback to null if timestamps are missing rather than 0.0.
Contributor
📈 Test Coverage Report
Coverage calculated from unit tests only |
19 tasks
kami619
added a commit
that referenced
this pull request
Mar 26, 2026
…names (#350) * fix(leaderboard): add GitLab repository support for URLs and display names The leaderboard pipeline assumed all repositories were on GitHub, causing GitLab-hosted repos to display broken links (https://github.com/redhat/builder instead of https://gitlab.com/redhat/rhel-ai/wheels/builder) and truncated names. Changes: - Leaderboard generator now reads repository.url from assessment JSON and converts SSH URLs to HTTPS, with fallback to GitHub for backwards compat - Validation workflow detects GitHub vs GitLab and uses git ls-remote for non-GitHub repo verification - Submit CLI accepts GitLab SSH/HTTPS URLs with deep paths - Added 17 regression tests for URL conversion and GitLab support Affected entries: redhat/rhel-ai/wheels/builder (#2), redhat/rhel-ai/rhai/pipeline (#11) Unblocks PR #347 (13 more GitLab repos) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style: apply black formatting to pass CI quality checks Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style: remove extraneous f-string prefix to fix ruff F541 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
github-actions bot
pushed a commit
that referenced
this pull request
Mar 26, 2026
# [2.31.0](v2.30.1...v2.31.0) (2026-03-26) ### Bug Fixes * **assessors:** support all YAML file naming conventions in dbt assessors ([3ff475a](3ff475a)) * **leaderboard:** add GitLab repository support for URLs and display names ([#350](#350)) ([47d8e71](47d8e71)), closes [#2](#2) [#11](#11) [#347](#347) ### Features * add python-wheel-build/fromager to leaderboard ([#346](#346)) ([6a9fab1](6a9fab1)) * add redhat/builder to leaderboard ([#348](#348)) ([480a4a4](480a4a4)) * add redhat/rhai-pipeline to leaderboard ([#349](#349)) ([e305a0f](e305a0f)) * add redhat/rhel-ai AIPCC productization repos to leaderboard ([#347](#347)) ([9b07e37](9b07e37)) * **assessors:** add first-class dbt SQL repository support ([8660e6b](8660e6b))
Contributor
|
🎉 This PR is included in version 2.31.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Leaderboard Submission
Organization: Red Hat / RHEL AI AIPCC Productization
Repos: 13 GitLab repos (gitlab.com/redhat/rhel-ai/...)
Submitted by: @jrusz
Repos & Scores
Validation Checklist
Submitted as part of the AI-First Bug Bash 2026-03.