Skip to content

fix: Log warning when input text is truncated to context limit#38

Open
echobt wants to merge 1 commit intomainfrom
fix/issue-162
Open

fix: Log warning when input text is truncated to context limit#38
echobt wants to merge 1 commit intomainfrom
fix/issue-162

Conversation

@echobt
Copy link
Copy Markdown
Contributor

@echobt echobt commented Jan 20, 2026

Description

This PR addresses the issue of silent token truncation when input text exceeds the context size limit. Previously, tokens were dropped without any indication to the user, potentially leading to incomplete embeddings and misleading search results.

Changes

  • Added a warning log to src/core/embeddings.rs that prints to stderr when truncation occurs.
  • The warning includes the original token count and the truncated (context limit) count.

Verification

  • Verified by code inspection that the truncation logic now includes a warning side-effect.
  • Existing tests pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant