Vector Store Issue - Track Which Documents Were Loaded?

[dkindlund]

It seems like the general DocumentLoader logic is quite "basic" -- meaning, regardless of vector store -- when you run an upsert operation, for all matching documents, the automation will load every document. This presents some challenges:

• If the source documents change, there's no built-in mechanism to update that content in the vector database. Instead, you're forced to figure out which records in the vector database need to be removed, manually.
• Alternatively, you can drop the entire table in the vector database and rebuild all records from scratch. However, this is very expensive if the number of documents you want in your vector database is quite large.

It would be great if Flowise had some sort of general mechanism to keep track of:

• Which documents were successfully loaded in the vector database.
• Report back to the source datastore where the documents originated from to mark/report which document was successfully loaded -- this could be setting a flag per record in the source datastore or even updating the source record with the UUID of the vector database record -- a sort of "proof" that the source document now has a copy in the vector database
• If we had this mechanism, then the source database could present a filtered view of just the subset of records that have yet to be vectorized.
• Then, upon every subsequent upsert, the DocumentLoader would only need to either create or update corresponding new or updated records in the vector database.

@HenryHengZJ , I'm not sure if any of these ideas have already been discussed or are already on a future roadmap. If so, any pointers to existing conversations would be helpful, as I couldn't find any after searching existing issues or discussions.

While I could try to implement a mechanism specific to the Airtable DocumentLoader node, I think figuring out how to implement this sort of tracking mechanism at a more generic level would benefit every combination of DocumentLoader and Vector Store combination instead.

After thinking about this further, it feels like each DocumentLoader needs some sort of datastore specific "callback" mechanism. The gist would be something like:

• In the general vector store interface, implement an optional "callback registration" mechanism
• After successfully inserting a document into the vector database, if registered, then invoke the callback and pass back the UUID of the vector store record with some sort of metadata about the original document.
• Then, within the DocumentLoader node, the callback function then processes the callback with the provided UUID and originating document metadata in order to update a reference in the source document database that the document was successfully "vectorized"

Note:

• Performing this sort of callback processing per individual document might be very costly and slow.
• Therefore, it might be a good idea to somehow collect "batches" of these updates upon successfully inserting batches of these documents in the vector database
• Then, provide each "batch" to the callback so that the callback can process those together

Unfortunately, I do not know enough about how the general Flowise DocumentLoader to Vector Store data path to implement something generically. I also wonder if there are fundamental limitations within LangChain itself that prevent us from implementing this capability within Flowise. In other words, I wonder if this mechanism at least partially would need some sort of code improvement at the LangChain level in order for this to be implemented correctly.

[HenryHengZJ]

@dkindlund we are working on the new Indexing API from langchain (https://js.langchain.com/docs/modules/data_connection/indexing/). This will let you keep track what embeddings have been upserted, avoide duplication. Exactly as what you described, the splitted docs with content and metadata will be hashed and compared against the new ones everytime upsert happens.

[dkindlund]

Thanks for the pointers, @HenryHengZJ ! I'm glad LangChain is offering a mechanism that Flowise can leverage to support this functionality. I'll be looking forward to when you have an initial implementation ready for testing, and I'd be happy to test it on my end when ready.

[DanielsLuz]

@HenryHengZJ do you have any updates on this one? My use case currently is managing Document Loaders from the API.

For more context, I see the /upsert endpoint does one part of it, but it does not reflect the Canva when new files are added, and we currently have no way of removing an outdated file via API.

[HenryHengZJ]

We're still reviewing the PR, currently waiting for some prerequisite PR to merge in first. Will update again once it's there

[pgaurav007]

Hi @HenryHengZJ , is there any update on thins feature? Thanks!

[Giusti10]

Up