feat: Add Git Commit Forensics for granular template and AI-clone detection - #341
Open
Shreesh-Sree wants to merge 1 commit into
Open
feat: Add Git Commit Forensics for granular template and AI-clone detection#341Shreesh-Sree wants to merge 1 commit into
Shreesh-Sree wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem Statement
The current evaluation pipeline relies purely on repository descriptions, languages, and star counts to evaluate a candidate's projects. This creates a significant blind spot where the agent cannot accurately differentiate between:
Without commit forensics, the LLM frequently misinterprets cloned repositories as high-impact original work.
Proposed Solution
This PR introduces Git Commit Forensics into the retrieval pipeline. By querying the GitHub contributors endpoint, the system extracts the candidate's exact commit volume and compares it against the repository's total commit volume. This data is injected into the prompt context, allowing the LLM to accurately deduce authentic development effort.
Key Changes
github.py):fetch_repo_contributorsto retrieve contribution statistics.fetch_contributions_countto calculateauthor_commit_countversustotal_commit_count.fetch_all_github_reposto dynamically classify projects asopen_source(multiple contributors) orself_project(single contributor).models.py):Projectmodel schema to ingest the new forensic metrics.Proof of Enhancement & Validation
Scenario A: The Cloned Tutorial Repository
Before this PR:
The agent evaluates a "React E-Commerce Dashboard" repository with 50 stars and assigns a
25/30Self-Project score, praising the complex architecture.After this PR:
The LLM context receives the following structural data:
{ "name": "react-ecommerce-dashboard", "project_type": "self_project", "author_commit_count": 1, "total_commit_count": 142 }Result: The LLM correctly identifies this as a cloned template. The score is reduced to
5/30with the corresponding evidence: "Candidate claims ownership but only authored 1 out of 142 commits; highly likely to be a cloned tutorial or boilerplate."Scenario B: Authentic Open Source Contribution
Before this PR:
The agent identifies a fork of
facebook/reacton the candidate's profile and incorrectly assumes the candidate authored the entire framework, heavily skewing the score.After this PR:
The LLM context receives the following structural data:
{ "name": "react", "project_type": "open_source", "author_commit_count": 12, "total_commit_count": 15830 }Result: The LLM accurately identifies this as a legitimate open-source contribution, scoring it under the
open_sourcecriteria rather thanself_projects, and rewards the candidate proportionally for their 12 commits to a major production codebase.Testing Performed
github.pygracefully handles rate limits when fetching contributors for large profiles.score.pyaccurately passes theauthor_commit_countto the Jinja templates.