Skip to content

Conversation

himanshusinghs
Copy link
Collaborator

Proposed changes

A small change to bring back accuracy to previous known levels.

Checklist

@himanshusinghs himanshusinghs requested a review from a team as a code owner September 22, 2025 11:25
@Copilot Copilot AI review requested due to automatic review settings September 22, 2025 11:25
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates accuracy tests to bring them back to previously known performance levels by adding missing expectations and improving test prompts.

  • Adds missing verbosity parameter expectations in explain tool accuracy tests
  • Improves the prompt clarity for collection index query testing

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
tests/accuracy/explain.test.ts Adds missing verbosity: Matcher.string() expectations to three test cases
tests/accuracy/collectionIndexes.test.ts Improves prompt wording from "Is the following query indexed?" to "Is there an index covering the following query?"

@himanshusinghs himanshusinghs added the no-title-validation Add this label to disable the title check for this PR. label Sep 22, 2025
@coveralls
Copy link
Collaborator

coveralls commented Sep 22, 2025

Pull Request Test Coverage Report for Build 17914396130

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 82.33%

Totals Coverage Status
Change from base Build 17911130906: 0.0%
Covered Lines: 5232
Relevant Lines: 6247

💛 - Coveralls

Copy link
Contributor

📊 Accuracy Test Results

📈 Summary

Metric Value
Commit SHA 200425db350c40a67e92ba9d4555ab65ad11a9ae
Run ID f16ec78a-166b-463b-91a3-926fa8057666
Status done
Total Prompts Evaluated 61
Models Tested 1
Average Accuracy 97.1%
Responses with 0% Accuracy 1
Responses with 75% Accuracy 3
Responses with 100% Accuracy 57

📊 Baseline Comparison

Metric Value
Baseline Commit 9f4c48b786d16093ae2936c2b8ddc270221eaaed
Baseline Run ID ca24c181-d9a9-4669-9982-4cdc1df5939f
Baseline Run Status done
Responses Improved 4
Responses Regressed 0

📎 Download Full HTML Report - Look for the accuracy-test-summary artifact for detailed results.

Report generated on: 9/22/2025, 11:58:43 AM

@himanshusinghs himanshusinghs merged commit 0501bdb into main Sep 22, 2025
21 checks passed
@himanshusinghs himanshusinghs deleted the chore/fix-accuracy-tests branch September 22, 2025 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accuracy-tests no-title-validation Add this label to disable the title check for this PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants