Skip to content

Add Option for Final Chunk Embedding Model to be different #14

@jparkerweb

Description

@jparkerweb

this can be important when using prefixes. it doesn't make since to use prefix model to create semantic similarity since they need separate document/query prefixes when creating embedding before calculating cosine similarity. but, it could be useful to specify a model that uses prefixes for the final embedding output.

  • calculate chunks and combine with a model like L6
  • on final chunk return, optionally use a different embedding model that works with "prefixes" (modern-bert, etc).

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

Status

Ready

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions