[BUG] Submit command allows fork submissions without upstream detection or user warning

[kami619]

Bug Description

The agentready submit command accepts leaderboard submissions for repository forks without detecting that the repository is a fork or warning the user. This results in incorrect leaderboard entries when users intend to submit the upstream repository but run the assessment on their local fork clone.

To Reproduce

Steps to reproduce the behavior:

1. Fork a repository on GitHub (e.g., fork opendatahub-io/opendatahub-tests to username/opendatahub-tests)
2. Clone your fork locally: git clone git@github.com:username/opendatahub-tests.git
3. Run agentready assess . in the clone directory
4. Run agentready submit to submit to the leaderboard
5. Observe that a PR is created for username/opendatahub-tests instead of opendatahub-io/opendatahub-tests

Expected Behavior

When submitting an assessment for a repository that is a fork, the tool should:

1. Detect that the repository is a fork using the GitHub API
2. Display the fork and its upstream repository to the user
3. Prompt the user to choose which repository to submit:
   • The fork (if user wants to showcase their fork)
   • The upstream (recommended for most cases)
4. If upstream is chosen, either re-run assessment or allow manual override with warning

Actual Behavior

The tool blindly submits whatever repository URL is in the assessment JSON file, which is extracted from git config remote.origin.url. No fork detection or validation occurs.

Result: Incorrect leaderboard entries for forks when user intended to submit upstream.

Example: PR #301 submitted dbasunag/opendatahub-tests when the user intended opendatahub-io/opendatahub-tests (which was correctly submitted in PR #314).

Environment

• Affected versions: All versions with leaderboard feature (v2.9.0+)
• Component: src/agentready/cli/submit.py, src/agentready/services/scanner.py
• Platform: All (macOS, Linux, Windows)

Root Cause

The bug has two contributing factors:

1. Assessment phase (src/agentready/services/scanner.py:188-192):

   • Extracts repository URL from git config remote.origin.url
   • Does not validate or detect if this is a fork vs upstream
   • Trusts local git configuration without questioning user intent

2. Submission phase (src/agentready/cli/submit.py:50-93, 177-213):

   • Extracts org/repo from assessment JSON
   • Validates submitter has access to that repo
   • Does not detect fork relationship or prompt user to confirm intent
   • Does not check if assessed repo matches user's intended submission target

This is a user error scenario that lacks proper tooling validation.

Impact

• Severity: Medium
• User Impact: Users working on forks will submit incorrect leaderboard entries
• Leaderboard Impact: Duplicate/confusing entries (fork + upstream)
• Manual Cleanup: Maintainers must manually remove incorrect submissions
• Recurrence Risk: High — common workflow for contributors

Verified Occurrences:

• PR Leaderboard: dbasunag/opendatahub-tests (69.9/100 - Silver) #301 (incorrect submission for dbasunag/opendatahub-tests)
• PR Leaderboard: opendatahub-io/opendatahub-tests (73.5/100 - Silver) #314 (correct submission for opendatahub-io/opendatahub-tests)

Immediate Fix (Completed)

Status: ✅ Fixed in commit bdd1d47

What was done:

• Removed incorrect submission file: submissions/dbasunag/opendatahub-tests/2026-02-18T19-35-35-assessment.json
• Leaderboard data will be automatically regenerated by GitHub Actions when merged to main
• Commit references both PR Leaderboard: dbasunag/opendatahub-tests (69.9/100 - Silver) #301 (incorrect) and PR Leaderboard: opendatahub-io/opendatahub-tests (73.5/100 - Silver) #314 (correct)

Files Changed:

• submissions/dbasunag/opendatahub-tests/2026-02-18T19-35-35-assessment.json - Deleted

Verification:

• Incorrect submission file removed
• GitHub Actions workflow verified to trigger on deletions
• Leaderboard data regenerated (will happen automatically on merge)
• Manual verification of leaderboard.json after workflow runs

Permanent Fix (Recommended)

To prevent recurrence, implement fork detection in the submit command:

Implementation Approach

File: src/agentready/cli/submit.py

Changes:

1. After extracting repo info from assessment (line ~547), query GitHub API to check if repo is a fork
2. If fork detected, retrieve parent.full_name (upstream repository)
3. Display interactive prompt:

   Warning: This assessment is for a fork.
   
   Repository assessed: username/repo-name
   Upstream repository: org/repo-name
   
   Which repository do you want to submit to the leaderboard?
     [1] username/repo-name (your fork)
     [2] org/repo-name (upstream) [RECOMMENDED]
     [3] Cancel
   
   Choice:
   

4. If user selects upstream:
   • Option A: Re-run assessment on upstream repository
   • Option B: Allow manual override with warning about score differences
5. Update assessment data or abort as needed

Alternative Approaches:

• Add --upstream flag for explicit control
• Add validation in GitHub Actions (rejects fork submissions)
• Always prompt user to confirm "Submit {org}/{repo}?" even for non-forks

Trade-offs: Fork detection requires GitHub API call (minor), adds UX friction (acceptable for correctness)

Testing Requirements

Unit Tests Needed

• Test fork detection logic (mock GitHub API response)
• Test user prompt and choice handling
• Test that non-fork submissions work unchanged
• Test error handling when API call fails

Integration Tests Needed

• End-to-end test: assess fork → submit → verify warning shown
• Test leaderboard generation script handles deletions correctly
• Test GitHub Actions workflow triggers on submission deletions

Manual Testing

• Fork a repository, run assessment, attempt submission
• Verify fork detection warning appears
• Test both "submit fork" and "submit upstream" choices
• Verify leaderboard data updates correctly

Additional Context

Related Issues: None (first occurrence documented)

Similar Patterns:

• Renamed repositories (local clone has old URL)
• Organization transfers (repo moved between orgs)
• Multiple remotes (origin vs upstream)

Analysis Document: See artifacts/bugfix/analysis/root-cause.md for detailed analysis

Proposed Solution Summary

Short-term (✅ Complete): Remove incorrect submission via PR
Long-term (📋 TODO): Implement fork detection and user confirmation in submit command

Estimated Effort: Medium (4-6 hours for implementation + tests)
Priority: Medium (prevents user confusion, improves leaderboard quality)
Risk: Low (additive feature, doesn't break existing functionality)

[kami619]

fixed in #321