Add Embedding Generation Support

[adnan-cto]

Implement embedding generation for extracted text using models like OpenAI's embeddings or other open-source alternatives.

[adnan-cto]

/bounty $50

[algora-pbc]

💎 $50 bounty • Unsiloed AI

Steps to solve:

1. Start working: Comment /attempt #25 with your implementation plan
2. Submit work: Create a pull request including /claim #25 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

❗ Important guidelines:

• To claim a bounty, you need to provide a short demo video of your changes in your pull request
• If anything is unclear, ask for clarification before starting as this will help avoid potential rework
• Low quality AI PRs will not receive review and will be closed
• Do not ask to be assigned unless you've contributed before

Thank you for contributing to Unsiloed-AI/Unsiloed-chunker!

Attempt	Started (UTC)	Solution	Actions
🟢 @Nitish-Naik	Jul 04, 2025, 04:54:59 PM	WIP

[Nitish-Naik]

@adnan-cto
Could you please assign this to me? I’ve already contributed to two issues, which are currently awaiting review and merging.
Issues : #28 , #29

[Nitish-Naik]

/attempt #25
Implementation Plan for Issue #25: Add Embedding Generation Support

1. Create a New Utility Function for Embeddings:

   • I will add a new function, generate_embeddings, to the Unsiloed/utils/openai.py module.
   • This function will accept a list of text chunks and an optional embedding_model parameter (defaulting to a cost-effective model like
     text-embedding-3-small(paid)).
   • It will use the existing OpenAI client to call the embeddings API endpoint (client.embeddings.create) and will return a list of embedding vectors.

2. Integrate Embedding Generation into the Main Workflow:

   • I will modify the main process function in Unsiloed/init.py.
   • It will now accept a new boolean option in the options dictionary: generate_embeddings.
   • If generate_embeddings is True, after the document chunking is complete, I will call the new generate_embeddings function with the text from the resulting chunks.

3. Add Embeddings to the Final Output:

   • The generated embedding for each chunk will be added directly to that chunk's dictionary under a new key, embedding.
   • The final result returned by the process and process_sync functions will therefore include the chunks with their corresponding embeddings, if requested.

4. Update Documentation:

   • I will update the README.md file to document the new generate_embeddings option and provide a clear usage example.

This approach integrates the new functionality cleanly into the existing architecture and provides a straightforward way for users to enable embedding generation.