Consultation - Metadata query and atomic upserts

[AntonPeniaziev]

Hi guys, I haven't found any community forum so asking here a few questions:

I'm using this documentation for my RAG project .

1. Im doing kb.upsert(documents)
   and immediately after that, I query (chat) with those docs. I found that upsert returns before the documents are indexed, I haven't found any smarter solution than this:

while index.describe_index_stats()['total_vector_count'] < len(docs):
            time.sleep(0.05)

The same goes for delete()
please advise a proper way to validate that the upserted documents are indexed and available.

2. How can I limit the chat_engine.chat() to use only documents with specific metadata fields? Is it possible at least in context_engine.query() if not in ChatEngine?
3. When using chat_engine, how can I see the fetched documents and the final prompt being sent to OpenAI?
   Alternatively how can I use tiktoken to estimate my actual OpenAI costs?
4. Which chunking/embedding method is used and could be alternated? (I still see 1 vector per document even if its very long, is it average of all the chunks?)
5. Is ChatEngine using stuff / map-reduce / map-rerank or any other approach?

Thanks!

[igiloh-pinecone]

Hi @AntonPeniaziev, glad to hear you're finding our documentation useful!

Let me try to answer all of your questions:

1. Indeed, the underlying index.upsert() call to the Pinecone database returns as soon as the database have received the data, so that does not always guarantee that the same vector was already processed and indexed. Usually, the vector will be available for querying within a few seconds. In most use cases, the documents will be residing in the Knowledge base for days \ weeks \ months, so those few seconds are usually negligible.
   If you want to start querying immediately after upsert, you might wanna use a retry mechanism:

   from tenacity import retry
   @retry(top=stop_after_attempt(3), wait=wait_random(3, 10))
   def my_query(...):
      res = knowledge_base.query(...)
      assert len(res) > 0
      return res

   But that depends on the use case. It might be that waiting until all docs were indexed is more suitable for your need.

2. The ContextEngine indeed offers the ability to filter vectors by metadata. In fact - there are two levels of query filtering:

   • Global filtering, which will be automatically applied for every query, and can be set by the ContextEngine(global_metadata_filter=) init argument.
   • Each Query object that is passed to ContextEngine.query() has a .metadata_filter. This is an additional filter that will be applied on top of the global filter, not instead of it.

   If you want all queries in your ChatEngine to use the same filter - simply set global_metadata_filter. However, if you want the LLM to dynamically formulate a filter per user interaction - you will need to change the prompt of the QueryGenerator, or derive your own custom QueryGenerator class.

3. First - the ChatEngine allows you not only to "see" the amount of tokens sent to the LLM, but also control it. Please see the max_prompt_tokens and max_context_tokens arguments in the documentation.
   In addition, if you'd like to print the full retrieved documents that were sent to the llm, you can print the .debug_info field of the returned answer:

    answer = chat_engine.chat(message)
    print(answer.debug_info)

4. Canopy supports multiple Chunker classes, as well as various RecordEncoder classes for embedding. Each of the existing Chunker classes has several configuration parameters that you can control. Users can also derive their own custom Chunker and RecordEncoder classes.

5. The behaviors you mention depend on the ContextBuilder class you choose when initializing the ContextEngine. Currently Canopy only provides a StuffingContextBuilder, but we will add more options in the future. You are welcome to implement your own custom ContextBuilder, and even contribute back and PR it to Canopy.

[AntonPeniaziev]

Thank you for the detailed answers!

can you please point to the relevant documentation on what are the default QueryGenerator, Chunker and RecordEncoder used?

[davdvp]

Hello,
I am also interested in knowing the method of passing a metadata variable value into the query generator.
Thanks