Que: chunk size check in SemanticChunker

[nathnet]

Question
Hi Chonkers! I'm new to LLM and RAG in general and I'm currently taking a course to learn how to implement RAG. I came across the concepts of how each embedding model has token window limit that will silent fail any text that has more than tokens they allow. And while I was working with SemanticChunker, I received Token indices sequence length is longer than the specified maximum sequence length for this model (595 > 512). Running this sequence through the model will result in indexing errors from HuggingFace tokenizer lib, which sounds like my document whose size is larger than the embedding model I use has already gone through the embedding stage of some sorts, even though I have already set chunk_size on SemanticChunker to limit it below the model's limit.

So I went to check the codebase and found that the chunk_size is only used to break down the chunks after it has already gone through vector similarity computation. My question/confusion is, shouldn't the chunk_size be somewhat in the prepare_sentences stage to make sure no sentences would go into the embedding model if the size exceeds the limit?

Thank you!

Additional Context
https://github.com/chonkie-inc/chonkie/blob/ac3a74fa4fba7551ad3f1fffc2a23e5b6815273f/src/chonkie/chunker/semantic.py#L463

[chonk-lain]

Hi @nathnet
chunk_size controls the output chunk size, not what gets passed to the embedding model during the similarity computation
what happens under the hood is the following :

1. _prepare_sentences : splits your text into individual sentences
2. _get_similarity : embeds each sentence and sliding window sentences to compute the semantic similarity between them
3. _split_groups : only after similarities are computed, chunk_size is used to enforce the final chunk size limit

The warning you're seeing likely comes from _get_window_embeddings, you might be able to work around it if you reduce similarity_window to maybe 2.
Would love to get more context or suggestions about your current topic, but in general the warning the error is safe to silence since most models truncate to max seq_len by default, and we only wanted to calculate the semanticity between sentences to see if they are related or not.