Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat (reranking) : Support for reranking models #3764

Open
wants to merge 27 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
da3ba16
feat (reranking) : Added reranking model in provider
patelvivekdev Nov 19, 2024
32d5bc6
prettier-fix
patelvivekdev Nov 19, 2024
33c2a61
Merge branch 'main' into vivek/reranking
patelvivekdev Nov 19, 2024
d4dbae7
Added rerank function, added test, updated rerank model in provider.
patelvivekdev Nov 20, 2024
cbc8a8b
Added reranking in cohere.
patelvivekdev Nov 20, 2024
2aa8181
Merge branch 'main' into vivek/reranking
patelvivekdev Nov 20, 2024
8ff22c7
Merge branch 'vercel:main' into vivek/reranking
patelvivekdev Nov 21, 2024
98c5ee8
Merge branch 'vercel:main' into vivek/reranking
patelvivekdev Nov 21, 2024
c1103c7
Fix: typo and fix tests.
patelvivekdev Nov 21, 2024
eb37050
Merge branch 'vercel:main' into vivek/reranking
patelvivekdev Nov 23, 2024
bc50079
Merge branch 'vercel:main' into vivek/reranking
patelvivekdev Nov 24, 2024
9868eb6
Merge branch 'vercel:main' into vivek/reranking
patelvivekdev Nov 25, 2024
1579e06
Fix: add rerankingModel to various providers with NoSuchModelError.
patelvivekdev Nov 25, 2024
62005a9
Merge branch 'vercel:main' into vivek/reranking
patelvivekdev Nov 26, 2024
96fedff
Merge branch 'main' into vivek/reranking
patelvivekdev Nov 27, 2024
d0faa9c
Merge branch 'main' into vivek/reranking
patelvivekdev Dec 2, 2024
8444585
Merge branch 'vercel:main' into vivek/reranking
patelvivekdev Dec 5, 2024
1217d24
fix: update cohere reranking API endpoint to v1
patelvivekdev Dec 5, 2024
ebafbf5
fix typo
patelvivekdev Dec 5, 2024
8ba391e
Merge branch 'vercel:main' into vivek/reranking
patelvivekdev Dec 5, 2024
f5a260b
Merge branch 'vercel:main' into vivek/reranking
patelvivekdev Dec 7, 2024
bab43b3
feat (reranking): Add reranking model support with error handling on …
patelvivekdev Dec 7, 2024
2b0741d
Merge branch 'main' into vivek/reranking
patelvivekdev Dec 13, 2024
1a06e44
Merge branch 'main' into vivek/reranking
patelvivekdev Jan 2, 2025
0599406
Merge branch 'vercel:main' into vivek/reranking
patelvivekdev Jan 5, 2025
528b737
Added NoSuchModelError on DeepSeekProvider
patelvivekdev Jan 6, 2025
7190f77
Merge branch 'vercel:main' into vivek/reranking
patelvivekdev Jan 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
134 changes: 134 additions & 0 deletions packages/ai/core/rerank/__snapshots__/rerank.test.ts.snap
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
// Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html

exports[`telemetry > should not record telemetry inputs / outputs when disabled 1`] = `
[
{
"attributes": {
"ai.model.id": "mock-model-id",
"ai.model.provider": "mock-provider",
"ai.operationId": "ai.rerank",
"ai.usage.tokens": 10,
"operation.name": "ai.rerank",
},
"events": [],
"name": "ai.rerank",
},
{
"attributes": {
"ai.model.id": "mock-model-id",
"ai.model.provider": "mock-provider",
"ai.operationId": "ai.rerank.doRerank",
"ai.usage.tokens": 10,
"operation.name": "ai.rerank.doRerank",
},
"events": [],
"name": "ai.rerank.doRerank",
},
]
`;

exports[`telemetry > should record telemetry data when enabled (multiple calls path) 1`] = `
[
{
"attributes": {
"ai.model.id": "mock-model-id",
"ai.model.provider": "mock-provider",
"ai.operationId": "ai.rerank",
"ai.rerankedDocuments": "[]",
"ai.rerankedIndices": "[1,0]",
"ai.telemetry.functionId": "test-function-id",
"ai.telemetry.metadata.test1": "value1",
"ai.telemetry.metadata.test2": false,
"ai.usage.tokens": 20,
"ai.values": "["sunny day at the beach","rainy day in the city"]",
"operation.name": "ai.rerank test-function-id",
"resource.name": "test-function-id",
},
"events": [],
"name": "ai.rerank",
},
{
"attributes": {
"ai.model.id": "mock-model-id",
"ai.model.provider": "mock-provider",
"ai.operationId": "ai.rerank.doRerank",
"ai.rerankedDocuments": "[]",
"ai.rerankedIndices": "[1]",
"ai.telemetry.functionId": "test-function-id",
"ai.telemetry.metadata.test1": "value1",
"ai.telemetry.metadata.test2": false,
"ai.usage.tokens": 10,
"ai.values": [
"["sunny day at the beach"]",
],
"operation.name": "ai.rerank.doRerank test-function-id",
"resource.name": "test-function-id",
},
"events": [],
"name": "ai.rerank.doRerank",
},
{
"attributes": {
"ai.model.id": "mock-model-id",
"ai.model.provider": "mock-provider",
"ai.operationId": "ai.rerank.doRerank",
"ai.rerankedDocuments": "[]",
"ai.rerankedIndices": "[0]",
"ai.telemetry.functionId": "test-function-id",
"ai.telemetry.metadata.test1": "value1",
"ai.telemetry.metadata.test2": false,
"ai.usage.tokens": 10,
"ai.values": [
"["rainy day in the city"]",
],
"operation.name": "ai.rerank.doRerank test-function-id",
"resource.name": "test-function-id",
},
"events": [],
"name": "ai.rerank.doRerank",
},
]
`;

exports[`telemetry > should record telemetry data when enabled (single call path) 1`] = `
[
{
"attributes": {
"ai.model.id": "mock-model-id",
"ai.model.provider": "mock-provider",
"ai.operationId": "ai.rerank",
"ai.rerankedDocuments": "["rainy day in the city","sunny day at the beach"]",
"ai.rerankedIndices": "[1,0]",
"ai.telemetry.functionId": "test-function-id",
"ai.telemetry.metadata.test1": "value1",
"ai.telemetry.metadata.test2": false,
"ai.usage.tokens": 10,
"ai.values": "["sunny day at the beach","rainy day in the city"]",
"operation.name": "ai.rerank test-function-id",
"resource.name": "test-function-id",
},
"events": [],
"name": "ai.rerank",
},
{
"attributes": {
"ai.model.id": "mock-model-id",
"ai.model.provider": "mock-provider",
"ai.operationId": "ai.rerank.doRerank",
"ai.rerankedDocuments": "["rainy day in the city","sunny day at the beach"]",
"ai.rerankedIndices": "[1,0]",
"ai.telemetry.functionId": "test-function-id",
"ai.telemetry.metadata.test1": "value1",
"ai.telemetry.metadata.test2": false,
"ai.usage.tokens": 10,
"ai.values": [
"["sunny day at the beach","rainy day in the city"]",
],
"operation.name": "ai.rerank.doRerank test-function-id",
"resource.name": "test-function-id",
},
"events": [],
"name": "ai.rerank.doRerank",
},
]
`;
2 changes: 2 additions & 0 deletions packages/ai/core/rerank/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
export * from './rerank';
export * from './rerank-result';
40 changes: 40 additions & 0 deletions packages/ai/core/rerank/rerank-result.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import { RerankedDocumentIndex } from '../types';
import { RerankingModelUsage } from '../types/usage';

/**
The result of an `rerank` call.
It contains the documents, the reranked indices, and additional information.
*/
export interface RerankResult<VALUE> {
/**
The documents that were reranked.
*/
readonly documents: Array<VALUE>;

/**
The reranked indices.
*/
readonly rerankedIndices: Array<RerankedDocumentIndex>;

/**
* The reranked documents.
* Only available if `returnDocuments` was set to `true`.
* The order of the documents is the same as the order of the indices.
*/
readonly rerankedDocuments?: Array<VALUE>;

/**
The reranking token usage.
*/
readonly usage: RerankingModelUsage;

/**
Optional raw response data.
*/
readonly rawResponse?: {
/**
Response headers.
*/
headers?: Record<string, string>;
};
}
Loading
Loading