Code Search Dynamic Relevancy Criteria Agent

[pranath-reddy]

Summary 📝

This PR introduces a Dynamic Relevance Criteria Agent that automatically generates query-specific relevance criteria for evaluating code repositories. Rather than applying a fixed assessment rubric, the agent analyzes each search query to extract tailored required and nice-to-have criteria that capture the query's intent and domain-specific requirements. This dynamic approach enables adaptive relevance assessment that responds to the nuances of each unique query.

Details

Dynamic Criteria Generation

Core Functionality

• Uses an LLM to analyze search queries and extract query-specific relevance criteria
• Distinguishes between required criteria (core requirements) and nice-to-have criteria (bonus features)
• Focuses on repository characteristics: algorithms, scientific domains, functionality, technical approaches, data types, programming languages
• Preserves concept relationships by keeping related concepts together (e.g., "machine learning for climate modeling" as single criterion)
• Generates minimal set of criteria necessary to comprehensively cover query requirements

Criteria Organization

• Required Criteria: Core requirements repositories must address to be considered relevant
• Nice-to-Have Criteria: Optional features providing bonus value beyond essentials
• Each criterion includes snake_case identifier and clear description for assessment guidance
• Query intent summary captures overall search objective
• Reasoning steps provide transparency into criteria generation logic

Code Changes

Added system prompt at akd/configs/code_prompts.py:

• CRITERIA_GEN_PROMPT for dynamic criteria generation logic

New Agent Implementation: DynamicRelevanceCriteriaAgent

• Extends LiteLLMInstructorBaseAgent with specialized input/output schemas
• Input: Query string with optional context
• Output: Required criteria, nice-to-have criteria, intent summary, reasoning steps

Usage

from akd.agents.relevancy import DynamicRelevanceCriteriaAgent, DynamicRelevanceCriteriaAgentInputSchema
from akd.agents._base import BaseAgentConfig

_criteria_gen_config = BaseAgentConfig(
    model_name="gpt-4o-mini",
    system_prompt=CRITERIA_GEN_PROMPT,
)
criteria_gen_agent = DynamicRelevanceCriteriaAgent(config=_criteria_gen_config)

result = await criteria_gen_agent.arun(DynamicRelevanceCriteriaAgentInputSchema(query=query))

Checks

• Tested Changes
• Stakeholder Approval

[github-actions]

❌ Tests failed (exit code: 2)

📊 Test Results

• Passed: 0
• Failed: 0
• Skipped: 0
• Warnings: 7
• Coverage: 0%

⚠️ Note: Test counts are 0, which may indicate parsing issues or early test failure. Check the workflow logs for details.
Parsing strategy used: summary_line

Branch: feature/code-search-relevancy
PR: #282
Commit: 640f194

📋 Full coverage report and logs are available in the workflow run.

[NISH1001]

@iamsims lookout for this aswell.. this might be useful for reranker as well. for scoring criteria....we want to unify scoring/relevancy check mechniams across several usecases....and adapt to usecases when needed

[github-actions]

❌ Tests failed (exit code: 2)

📊 Test Results

• Passed: 0
• Failed: 0
• Skipped: 0
• Warnings: 7
• Coverage: 0%

⚠️ Note: Test counts are 0, which may indicate parsing issues or early test failure. Check the workflow logs for details.
Parsing strategy used: summary_line

Branch: feature/code-search-relevancy
PR: #282
Commit: 047e5df

📋 Full coverage report and logs are available in the workflow run.

[NISH1001]

cc: @iamsims maybe discuss with @pranath-reddy and likewisecome to common ground. The implications are: we will use this ultimately for llm-based reranker. Feels like there are multiple things disconnected which should be under similar usecase.

[github-actions]

❌ Tests failed (exit code: 2)

📊 Test Results

• Passed: 0
• Failed: 0
• Skipped: 0
• Warnings: 8
• Coverage: 0%

⚠️ Note: Test counts are 0, which may indicate parsing issues or early test failure. Check the workflow logs for details.
Parsing strategy used: summary_line

Branch: feature/code-search-relevancy
PR: #282
Commit: 633d3b3

📋 Full coverage report and logs are available in the workflow run.