❓ Remote Embedding Generation

[selbyk]

It would be nice if we could configure embedding generation by passing in an async function to do the generation ourselves.

[jparkerweb]

@selbyk can you please provide an hypothetical example of what this would look like and why you would like it.
I'm open to the idea (feel free to fork and create a PR as well) :)

[geleto]

Add a simple API with a callback to do the embeddings (array of strings in, array of embeddings out).
Just the chunking algorithm with no scripts, containers, downloads, services, etc...
While it is useful in production environments to run your embedding model locally, most people that are in the development stage would just prefer to use whatever provider/API they use for the rest of their project.
And for production you may have different requirements - e.g. use the embedding models from your cloud provider.

[geleto]

I extracted the core chunking functionality from semantic-chunking - no scripts, containers, downloads, services, etc., modified it to be model-agnostic - and wrapped it in a simple API with an embedding callback:

Check it out: semachunk

const chunks = await chunkText(text, async (texts) => {
    // Your embedding logic here - OpenAI, HuggingFace, whatever
    return await openai.embeddings.create({ input: texts, model: "text-embedding-3-small" });
}, { maxChunkSize: 500 });

I also ran into a rate-limiting issue with the original merging approach - after the initial sentence embeddings, chunks get merged and re-embedded one at a time, which burns through API quotas quickly and is slow even with a local model.

So I rewrote the merging algorithm: instead of merging linearly, it scores all adjacent chunk pairs, selects the best candidates, merges them in one pass, then batch re-embeds. The new algorithm should give better results and drastically reduces API calls, playing nicely with rate limits.

Haven't battle-tested it yet; I just needed it for a simple RAG example, but it's working well for that. Feedback welcome!