`chunk_markdown` emits chunks far beyond `max_chunk_size`; padded ONNX batches then explode (seq² attention)

[ParkHwan]

Summary. In _split_large_section, the paragraph-boundary branch emits the accumulated text without any size enforcement. If the accumulated lines include a single very long line (e.g. a Markdown table cell containing a dumped JSON response — 21,004 chars in our corpus), the whole accumulation is emitted as one chunk. _split_long_text exists but is never applied on this path, so max_chunk_size=1500 is not honored.

Observed. A 23 KB Confluence-exported markdown file produced a 21,061-char chunk (8,518 tokens). Since OnnxEmbedding pads the whole batch (default 32) to the longest sequence (truncation cap 8192), one such chunk inflates an entire batch to batch × 8192 tokens, and attention scales with batch × seq². On an Apple M3 Pro the file stalled for ~88 minutes and the process was then killed by memory pressure. Re-running the same file with --batch-size 2 indexed it in 41 s — which is a workaround, not a fix, since the oversized chunk still gets truncated and embedded as a single degraded vector.

Repro sketch.

from memsearch.chunker import chunk_markdown

text = "# T\n\n| h |\n| :-- |\n| " + ("x" * 21000) + " |\n\nnext paragraph\n"
chunks = chunk_markdown(text, source="t.md", max_chunk_size=1500)
print(max(len(c.content) for c in chunks))  # far beyond 1500

Suggested fix.

1. Enforce max_chunk_size on every emit path in _split_large_section (apply _split_long_text to any oversized emit), and/or
2. Sort chunks by token length before batching in the embedding providers so padding waste is bounded by similar-length neighbors.

Related operational context in #613 (we found both while diagnosing a multi-hour indexing loop).

[zc277584121]

Thanks for the detailed report. I tried the repro sketch against current main and got bounded chunks: 47 chunks with max content length 1500 when max_chunk_size=1500.

Current _split_large_section routes the paragraph-boundary, forced line-boundary, and final flush paths through _emit_bounded(), which applies _split_long_text() when content exceeds max_size. That behavior appears to have been added in 7a2f37b / #528.

Could you confirm the exact version and code path your environment is loading? A few useful checks would be:

python -m memsearch --version
python - <<'PY'
import inspect
import memsearch
import memsearch.chunker as c
print("memsearch module:", getattr(memsearch, "__file__", "unknown"))
print("chunker module:", getattr(c, "__file__", "unknown"))
print("package version:", getattr(memsearch, "__version__", "no __version__"))
print(inspect.getsource(c._split_large_section))
PY

If the installed CLI is what you are testing, this is also useful:

which memsearch
memsearch --version

A small sanitized markdown sample that still produces a chunk larger than max_chunk_size would also help. If your corpus is hitting a different path than the repro sketch, we definitely want to fix it.