Skip to content

feat: Add Git Commit Forensics for granular template and AI-clone detection - #341

Open
Shreesh-Sree wants to merge 1 commit into
interviewstreet:mainfrom
Shreesh-Sree:feat/git-commit-forensics
Open

feat: Add Git Commit Forensics for granular template and AI-clone detection#341
Shreesh-Sree wants to merge 1 commit into
interviewstreet:mainfrom
Shreesh-Sree:feat/git-commit-forensics

Conversation

@Shreesh-Sree

Copy link
Copy Markdown
Contributor

Problem Statement

The current evaluation pipeline relies purely on repository descriptions, languages, and star counts to evaluate a candidate's projects. This creates a significant blind spot where the agent cannot accurately differentiate between:

  1. A candidate who cloned a large tutorial or template repository and made a single, trivial commit.
  2. A candidate who authentically contributed to an existing open-source project over an extended period.
  3. A candidate who iteratively built a complex system from scratch.

Without commit forensics, the LLM frequently misinterprets cloned repositories as high-impact original work.

Proposed Solution

This PR introduces Git Commit Forensics into the retrieval pipeline. By querying the GitHub contributors endpoint, the system extracts the candidate's exact commit volume and compares it against the repository's total commit volume. This data is injected into the prompt context, allowing the LLM to accurately deduce authentic development effort.

Key Changes

  1. GitHub API Enhancements (github.py):
    • Implemented fetch_repo_contributors to retrieve contribution statistics.
    • Implemented fetch_contributions_count to calculate author_commit_count versus total_commit_count.
    • Enhanced fetch_all_github_repos to dynamically classify projects as open_source (multiple contributors) or self_project (single contributor).
  2. Schema & Model Updates (models.py):
    • Updated the Project model schema to ingest the new forensic metrics.
  3. LLM Context Injection:
    • The evaluation context now receives exact commit ratios, explicitly instructing the LLM to penalize projects that have massive codebases but minimal author commits.

Proof of Enhancement & Validation

Scenario A: The Cloned Tutorial Repository

Before this PR:
The agent evaluates a "React E-Commerce Dashboard" repository with 50 stars and assigns a 25/30 Self-Project score, praising the complex architecture.

After this PR:
The LLM context receives the following structural data:

{
  "name": "react-ecommerce-dashboard",
  "project_type": "self_project",
  "author_commit_count": 1,
  "total_commit_count": 142
}

Result: The LLM correctly identifies this as a cloned template. The score is reduced to 5/30 with the corresponding evidence: "Candidate claims ownership but only authored 1 out of 142 commits; highly likely to be a cloned tutorial or boilerplate."

Scenario B: Authentic Open Source Contribution

Before this PR:
The agent identifies a fork of facebook/react on the candidate's profile and incorrectly assumes the candidate authored the entire framework, heavily skewing the score.

After this PR:
The LLM context receives the following structural data:

{
  "name": "react",
  "project_type": "open_source",
  "author_commit_count": 12,
  "total_commit_count": 15830
}

Result: The LLM accurately identifies this as a legitimate open-source contribution, scoring it under the open_source criteria rather than self_projects, and rewards the candidate proportionally for their 12 commits to a major production codebase.

Testing Performed

  • Verified github.py gracefully handles rate limits when fetching contributors for large profiles.
  • Verified score.py accurately passes the author_commit_count to the Jinja templates.
  • Evaluated locally against sample resumes; successfully detected and penalized low-effort template clones while accurately scoring legitimate open-source contributions.

Copilot AI review requested due to automatic review settings July 11, 2026 18:21

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants