fix(leaderboard): add GitLab repository support for URLs and display names#350
Merged
kami619 merged 3 commits intoambient-code:mainfrom Mar 26, 2026
Merged
Conversation
…names The leaderboard pipeline assumed all repositories were on GitHub, causing GitLab-hosted repos to display broken links (https://github.com/redhat/builder instead of https://gitlab.com/redhat/rhel-ai/wheels/builder) and truncated names. Changes: - Leaderboard generator now reads repository.url from assessment JSON and converts SSH URLs to HTTPS, with fallback to GitHub for backwards compat - Validation workflow detects GitHub vs GitLab and uses git ls-remote for non-GitHub repo verification - Submit CLI accepts GitLab SSH/HTTPS URLs with deep paths - Added 17 regression tests for URL conversion and GitLab support Affected entries: redhat/rhel-ai/wheels/builder (#2), redhat/rhel-ai/rhai/pipeline (ambient-code#11) Unblocks PR ambient-code#347 (13 more GitLab repos) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
Warning
|
| Cohort / File(s) | Summary |
|---|---|
Workflow Infrastructure .github/workflows/leaderboard.yml |
Added "Detect repository host" step with URL pattern matching to identify GitHub vs. GitLab. Reworked verification steps to branch by HOST: GitHub uses GitHub API for public/private checks, non-GitHub uses git ls-remote for public accessibility. Updated host-specific access verification and adjusted clone operations to use derived HTTPS clone_url. |
Leaderboard Data Generation scripts/generate-leaderboard-data.py |
Added URL parsing helpers (git_url_to_https(), repo_display_name_from_url()) to convert git remote URLs to HTTPS browse URLs and extract host/org[/repo...] display paths. Updated leaderboard entry construction to derive repository metadata from latest["repository"]["url"] with fallback to GitHub-style assumptions when URL data is unavailable. |
CLI Submission Module src/agentready/cli/submit.py |
Extended host support beyond GitHub with SUPPORTED_HOSTS constant and repository URL parser (_parse_repo_url()). Added _repo_browse_url() helper and updated extract_repo_info() to return 6 values including host and full_path. Modified submission functions to branch verification logic: GitHub API checks for GitHub, git ls-remote for non-GitHub hosts. Updated PR body generation to include browse URL and host labeling. |
Unit Tests tests/unit/test_cli_submit.py, tests/unit/test_generate_leaderboard.py |
Updated existing extract_repo_info tests to unpack new 6-value return tuple and assert host/full_path values. Added GitLab SSH and HTTPS repository parsing test cases. Created new test module validating URL parsing helpers and end-to-end leaderboard generation for both GitHub and GitLab with fallback behavior. |
Sequence Diagram(s)
sequenceDiagram
actor User
participant CLI as CLI (submit.py)
participant GitHub as GitHub API
participant GitLab as GitLab (git ls-remote)
participant Repo as Repository
User->>CLI: submit(repo_url)
CLI->>CLI: extract_repo_info(assessment_data)
CLI->>CLI: _parse_repo_url(repo_url) → host, org/repo
CLI->>CLI: _repo_browse_url(host, path) → browse_url
alt host == "github.com"
CLI->>GitHub: verify repo (API)
GitHub->>GitHub: check public/private, collaborator access
GitHub-->>CLI: accessibility confirmed
else host == "gitlab.com"
CLI->>GitLab: git ls-remote --exit-code clone_url HEAD
GitLab->>Repo: check public access
Repo-->>GitLab: HEAD ref found
GitLab-->>CLI: public repo confirmed
end
CLI->>CLI: generate_pr_body(host, full_path, browse_url)
CLI->>Repo: clone/submit assessment
Repo-->>CLI: submission complete
CLI-->>User: display browse_url & results
sequenceDiagram
participant Workflow as GitHub Actions
participant DetectHost as "Host Detection"
participant VerifyRepo as "Verify Repository"
participant VerifyAccess as "Verify Access"
participant GitHub as GitHub API
participant Git as git ls-remote
participant Clone as Clone & Assess
Workflow->>DetectHost: infer host from REPO_URL
DetectHost->>DetectHost: pattern matching (github.com vs gitlab.com)
DetectHost-->>Workflow: HOST, CLONE_URL
Workflow->>VerifyRepo: check if repo exists & public
alt HOST == "github.com"
VerifyRepo->>GitHub: gh repo view --json isPrivate
GitHub-->>VerifyRepo: public repo confirmed
else HOST != "github.com"
VerifyRepo->>Git: git ls-remote --exit-code CLONE_URL HEAD
Git-->>VerifyRepo: public repo confirmed
end
VerifyRepo-->>Workflow: repo verified
Workflow->>VerifyAccess: check submitter access
alt HOST == "github.com"
VerifyAccess->>GitHub: gh api to check collaborator/owner
GitHub-->>VerifyAccess: access confirmed or denied
else HOST != "github.com"
VerifyAccess-->>Workflow: emit warning (skip verification)
end
Workflow->>Clone: Re-run assessment
Clone->>Clone: git clone CLONE_URL
Clone-->>Workflow: assessment complete
Estimated code review effort
🎯 4 (Complex) | ⏱️ ~45 minutes
🚥 Pre-merge checks | ✅ 2 | ❌ 1
❌ Failed checks (1 warning)
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage | Docstring coverage is 69.44% which is insufficient. The required threshold is 80.00%. | Write docstrings for the functions missing them to satisfy the coverage threshold. |
✅ Passed checks (2 passed)
| Check name | Status | Explanation |
|---|---|---|
| Title check | ✅ Passed | The title accurately summarizes the main change: adding GitLab repository support for URLs and display names in the leaderboard pipeline. |
| Description check | ✅ Passed | The description is directly related to the changeset, explaining the GitLab support issue, changes made, and testing performed. |
✏️ Tip: You can configure your own custom pre-merge checks in the settings.
✨ Finishing Touches
🧪 Generate unit tests (beta)
- Create PR with unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands and usage tips.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
scripts/generate-leaderboard-data.py (1)
189-199:⚠️ Potential issue | 🟡 MinorInconsistent use of
repo_nameinmost_improvedentries.The
most_improvedlist still uses the directory-derivedrepo_name(Line 191) rather than the URL-deriveddisplay_name. This creates inconsistency: main leaderboard entries show full GitLab paths likeredhat/rhel-ai/wheels/builder, but themost_improvedsection would show truncatedredhat/builder.Proposed fix
most_improved.append( { - "repo": repo_name, + "repo": display_name, "improvement": round(improvement, 1), "from_score": float(oldest["overall_score"]),🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/generate-leaderboard-data.py` around lines 189 - 199, The most_improved entries are using the directory-derived repo_name instead of the URL-derived display_name; update the dictionary added to most_improved (the block that sets "repo": repo_name, "improvement": ..., etc.) to use "repo": display_name so the most_improved section matches the main leaderboard naming; ensure display_name is the variable in scope where the append occurs (in the same loop that computes improvement and uses submissions, oldest, latest).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/leaderboard.yml:
- Around line 107-114: The current grep checks (grep -q "github\.com" /
"gitlab\.com") can match those strings anywhere in the URL; update the checks to
match the actual host portion of REPO_URL (so malicious paths won't match).
Replace the two grep lines with anchored/URL-aware regex checks such as using
grep -E (or grep -Eq) against a pattern that ensures the hostname is github.com
or gitlab.com (for example matching ://...github\.com(/|$) or allowing optional
userinfo/www), i.e. change the grep commands that reference REPO_URL to stricter
regexes and leave the echo "host=github"/"host=gitlab" writes to GITHUB_OUTPUT
unchanged.
In `@src/agentready/cli/submit.py`:
- Around line 99-105: The click.echo call inside the host check uses an
unnecessary f-string (no placeholders) causing ruff F541; change the first
click.echo argument to a regular string literal (remove the leading f) in the
conditional that checks SUPPORTED_HOSTS (the block that prints "Error:
Unsupported repository host. Only GitHub and GitLab are supported." and echoes
repo_url), leaving the repo_url echo unchanged.
---
Outside diff comments:
In `@scripts/generate-leaderboard-data.py`:
- Around line 189-199: The most_improved entries are using the directory-derived
repo_name instead of the URL-derived display_name; update the dictionary added
to most_improved (the block that sets "repo": repo_name, "improvement": ...,
etc.) to use "repo": display_name so the most_improved section matches the main
leaderboard naming; ensure display_name is the variable in scope where the
append occurs (in the same loop that computes improvement and uses submissions,
oldest, latest).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: ed6d5cf5-8432-41be-a048-cccb70365a72
📒 Files selected for processing (5)
.github/workflows/leaderboard.ymlscripts/generate-leaderboard-data.pysrc/agentready/cli/submit.pytests/unit/test_cli_submit.pytests/unit/test_generate_leaderboard.py
Comment on lines
+107
to
114
| if echo "$REPO_URL" | grep -q "github\.com"; then | ||
| echo "host=github" >> "$GITHUB_OUTPUT" | ||
| elif echo "$REPO_URL" | grep -q "gitlab\.com"; then | ||
| echo "host=gitlab" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "::error::Unsupported repository host in URL: $REPO_URL" | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
Grep patterns may match unintended hosts.
The patterns grep -q "github\.com" and grep -q "gitlab\.com" match these strings anywhere in the URL. A malicious URL like https://evil.com/fake-github.com/org/repo would incorrectly be classified as GitHub. Consider using stricter patterns.
Proposed fix using anchored patterns
- if echo "$REPO_URL" | grep -q "github\.com"; then
+ if echo "$REPO_URL" | grep -qE '^(git@|https?://)github\.com[:/]'; then
echo "host=github" >> "$GITHUB_OUTPUT"
- elif echo "$REPO_URL" | grep -q "gitlab\.com"; then
+ elif echo "$REPO_URL" | grep -qE '^(git@|https?://)gitlab\.com[:/]'; then
echo "host=gitlab" >> "$GITHUB_OUTPUT"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if echo "$REPO_URL" | grep -q "github\.com"; then | |
| echo "host=github" >> "$GITHUB_OUTPUT" | |
| elif echo "$REPO_URL" | grep -q "gitlab\.com"; then | |
| echo "host=gitlab" >> "$GITHUB_OUTPUT" | |
| else | |
| echo "::error::Unsupported repository host in URL: $REPO_URL" | |
| exit 1 | |
| fi | |
| if echo "$REPO_URL" | grep -qE '^(git@|https?://)github\.com[:/]'; then | |
| echo "host=github" >> "$GITHUB_OUTPUT" | |
| elif echo "$REPO_URL" | grep -qE '^(git@|https?://)gitlab\.com[:/]'; then | |
| echo "host=gitlab" >> "$GITHUB_OUTPUT" | |
| else | |
| echo "::error::Unsupported repository host in URL: $REPO_URL" | |
| exit 1 | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/leaderboard.yml around lines 107 - 114, The current grep
checks (grep -q "github\.com" / "gitlab\.com") can match those strings anywhere
in the URL; update the checks to match the actual host portion of REPO_URL (so
malicious paths won't match). Replace the two grep lines with anchored/URL-aware
regex checks such as using grep -E (or grep -Eq) against a pattern that ensures
the hostname is github.com or gitlab.com (for example matching
://...github\.com(/|$) or allowing optional userinfo/www), i.e. change the grep
commands that reference REPO_URL to stricter regexes and leave the echo
"host=github"/"host=gitlab" writes to GITHUB_OUTPUT unchanged.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
📉 Test Coverage Report
Coverage calculated from unit tests only |
github-actions bot
pushed a commit
that referenced
this pull request
Mar 26, 2026
# [2.31.0](v2.30.1...v2.31.0) (2026-03-26) ### Bug Fixes * **assessors:** support all YAML file naming conventions in dbt assessors ([3ff475a](3ff475a)) * **leaderboard:** add GitLab repository support for URLs and display names ([#350](#350)) ([47d8e71](47d8e71)), closes [#2](#2) [#11](#11) [#347](#347) ### Features * add python-wheel-build/fromager to leaderboard ([#346](#346)) ([6a9fab1](6a9fab1)) * add redhat/builder to leaderboard ([#348](#348)) ([480a4a4](480a4a4)) * add redhat/rhai-pipeline to leaderboard ([#349](#349)) ([e305a0f](e305a0f)) * add redhat/rhel-ai AIPCC productization repos to leaderboard ([#347](#347)) ([9b07e37](9b07e37)) * **assessors:** add first-class dbt SQL repository support ([8660e6b](8660e6b))
Contributor
|
🎉 This PR is included in version 2.31.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The leaderboard pipeline assumed all repositories were on GitHub, causing GitLab-hosted repos to display broken links (https://github.com/redhat/builder instead of https://gitlab.com/redhat/rhel-ai/wheels/builder) and truncated names.
Type of Change
Related Issues
Affected entries: redhat/rhel-ai/wheels/builder (#2), redhat/rhel-ai/rhai/pipeline (#11) Unblocks PR #347 (13 more GitLab repos)
Changes Made
Changes:
Testing
pytest)Checklist
Screenshots (if applicable)
Additional Notes