feat(langchain): sanitize Summarization Content to remove extra metadata #34570
Conversation
github-actions
bot
added
langchain
|
Thanks for the PR! The sanitization approach does help reduce token usage, but after investigation we found a simpler solution that fully resolves the issue. The root cause: When The sanitization approach reduces the ratio from 2.4x to ~1.8x, but doesn't fully invert the inequality. Using This is more token-efficient and aligns with how Superseding with a simpler fix in #34607 that uses formatted_messages = get_buffer_string(trimmed_messages)
response = self.model.invoke(self.summary_prompt.format(messages=formatted_messages))Thanks for working on this - your PR helped identify that metadata was the issue! |
#34607) Fixes #34517 Supersedes #34557, #34570 Fixes token inflation in `SummarizationMiddleware` that caused context window overflow during summarization. **Root cause:** When formatting messages for the summary prompt, `str(messages)` was implicitly called, which includes all Pydantic metadata fields (`usage_metadata`, `response_metadata`, `additional_kwargs`, etc.). This caused the stringified representation to use ~2.5x more tokens than `count_tokens_approximately` estimates. **Problem:** - Summarization triggers at 85% of context window based on `count_tokens_approximately` - But `str(messages)` in the prompt uses 2.5x more tokens - Results in `ContextLengthExceeded` **Fix:** Use `get_buffer_string()` to format messages, which produces compact output: ``` Human: What's the weather? AI: Let me check...[tool_calls] Tool: 72°F and sunny ``` Instead of verbose Pydantic repr: ```python [HumanMessage(content='What's the weather?', additional_kwargs={}, response_metadata={}), ...] ```
2 participants
SummarizationMiddlewarehistory to prevent token overflow #34557 for this, but that completely get's rid of Tool Call information which might be important for Summarization.SummarizationMiddlewareincludes metadata in prompt causing context length overflow #34517