RERANK_CONTEXT_SIZE (2048) too small — qmd query crashes on CJK content

[SDNAnyFlow]

Summary

qmd query crashes during reranking when the combined input (query + document chunk + Qwen3 template overhead) exceeds RERANK_CONTEXT_SIZE = 2048. The error is deterministic and reproducible.

Environment

• QMD version: 1.1.0 (also reproduced on 1.0.7)
• OS: Rocky Linux 9 (x86_64)
• Node.js: v22.22.0
• GPU: NVIDIA RTX 3090 (24GB VRAM)
• Content: ~345 markdown files, primarily CJK (Chinese) text
• Index: 1386 chunks from 338 documents

Error

$ qmd query "test" --json
Searching 6 queries...
Reranking 40 chunks...

Error: The input lengths of some of the given documents exceed the context size.
Try to increase the context size to at least 2099
    at LlamaRankingContext.rankAll (...LlamaRankingContext.js:50:19)
    at LlamaCpp.rerank (...llm.js:751:82)

Root Cause

In src/llm.ts:

static RERANK_CONTEXT_SIZE = 2048;

The reranker input = query tokens + chunk tokens + Qwen3 template overhead (~200). The comment says chunks are capped at ~800 tokens so ~1100 should fit, but:

1. CJK tokenization produces different token counts — a chunk ~900 tokens in the embedding tokenizer may be longer in the Qwen3 reranker tokenizer.
2. Query expansion generates HyDE documents 100+ tokens, pushing total past 2048.
3. The error requests at least 2099 — only 51 tokens over.

Workaround

Changing RERANK_CONTEXT_SIZE to 4096 in dist/llm.js resolves the issue.

Suggested Fix

1. Increase default to 4096 (safest, modest VRAM cost)
2. Dynamic sizing: compute required context from actual longest (query + chunk) pair
3. Graceful fallback: skip oversized chunks during reranking instead of crashing (use retrieval score)

Option 3 is most robust.

Related

• Changelog v1.0.0: right-sized reranker context (40960 to 2048, 17x less memory)
• The reduction was too aggressive for CJK content with long query expansions

Thank you for building QMD!

[joshmedeski]

I also ran into this issue, perhaps we could make the RERANK_CONTEXT_SIZE configurable for those that want to increase it?

[yogesh-aggarwal]

Making RERANK_CONTEXT_SIZE configurable treats the symptom. The actual reranker input is:

template_tokens + query_tokens (incl. HyDE) + chunk_tokens

Two things make any fixed limit unreliable. First, chunk sizes are estimated using the embedding tokenizer, but reranking goes through Qwen and with CJK text, that mismatch can inflate counts well beyond the estimate. Second, HyDE/query expansion adds 100–200 tokens unpredictably, so inputs that look fine at index time can blow up at query time.

The fix is probably a combination of things. Token-aware truncation as the primary guard compute available space and trim the chunk before it hits the reranker:

available = RERANK_CONTEXT_SIZE - template_tokens - query_tokens
chunk = truncate(chunk, available)

And if something still slips through, fall back to the retrieval score instead of failing the whole query. Making the context size configurable on top of that is still useful, just not sufficient on its own.

[joshmedeski]

My only concern about the fallback method is I would want to know when it happens, so some sort of messaging would be helpful.

[SDNAnyFlow]

Tested the proposed fix locally — token-aware truncation + fallback works well.

Patch applied to rerank() in dist/llm.js:

1. Tokenize query with the reranker model's tokenizer (not embedding tokenizer) to get accurate token counts
2. Compute available = RERANK_CONTEXT_SIZE - template_overhead(200) - query_tokens
3. Truncate chunks exceeding available via rerankModel.tokenize() → slice → rerankModel.detokenize()
4. Wrap rankAll in try/catch as safety net — returns score 0 on failure instead of crashing

// Key truncation logic added to rerank():
const queryTokens = rerankModel.tokenize(query);
const availableForChunk = RERANK_CONTEXT_SIZE - 200 - queryTokens.length;

const texts = documents.map((doc) => {
    let text = doc.text;
    const docTokens = rerankModel.tokenize(text);
    if (docTokens.length > availableForChunk) {
        text = rerankModel.detokenize(docTokens.slice(0, availableForChunk));
    }
    return text;
});

With RERANK_CONTEXT_SIZE kept at 2048, CJK content that previously crashed now completes successfully. The truncation has no noticeable impact on ranking quality since the reranker only needs enough context to judge relevance.

Happy to submit a PR if helpful.

[yogesh-aggarwal]

@SDNAnyFlow did you PR or should I?

[bonk-moltbot]

The truncation patch from @SDNAnyFlow fixes the crash but introduces a secondary issue: after truncation, the text no longer matches the key in textToDoc, so the downstream lookup fails with TypeError: Cannot read properties of undefined (reading 'file').

Fix: register the truncated text in the map after truncation:

if (docTokens.length > availableForChunk) {
    text = rerankModel.detokenize(docTokens.slice(0, availableForChunk));
    textToDoc.set(text, textToDoc.get(doc.text));
}

[yongsubaek]

Same issue with Korean content — error requested at least 2064 (16 tokens over).

The 4096 workaround resolves it. +1 for option 3 (graceful fallback with truncation). @bonk-moltbot's textToDoc fix looks necessary too.

[mvanhorn]

I've submitted a fix for this in PR #369. Adds QMD_RERANK_CONTEXT_SIZE env var (defaults to 2048, follows the QMD_EXPAND_CONTEXT_SIZE pattern).