IndexError: index 4 is out of bounds for axis 0 with size 4

[RafaelJohn9]

Error

 self.last_retrieved = self.vector_store.retrieve(query=input_data, top_k=top_k)
  File "/swarmauri-sdk/pkgs/swarmauri/swarmauri/vector_stores/concrete/Doc2VecVectorStore.py", line 63, in retrieve
    transform_matrix = self._embedder.fit_transform(documents)
  File "/swarmauri-sdk/pkgs/swarmauri/swarmauri/embeddings/concrete/Doc2VecEmbedding.py", line 62, in fit_transform
    self.fit(documents, **kwargs)
  File "/swarmauri-sdk/pkgs/swarmauri/swarmauri/embeddings/concrete/Doc2VecEmbedding.py", line 34, in fit
    self._model.build_vocab(tagged_data)
  File "/.local/lib/python3.10/site-packages/gensim/models/doc2vec.py", line 888, in build_vocab
    report_values = self.prepare_vocab(update=update, keep_raw_vocab=keep_raw_vocab, trim_rule=trim_rule, **kwargs)
  File "/.local/lib/python3.10/site-packages/gensim/models/word2vec.py", line 769, in prepare_vocab
    self.wv.sort_by_descending_frequency()
  File "/.local/lib/python3.10/site-packages/gensim/models/keyedvectors.py", line 757, in sort_by_descending_frequency
    self.vectors = self.vectors[count_sorted_indexes]
IndexError: index 4 is out of bounds for axis 0 with size 4

This Error is becoming to common. A permanent solution is needed 👍

[MichaelDecent]

yes. experienced it too. i will be on a lookout for a permanent solution

[RafaelJohn9]

Thanks mate! 👍

[cobycloud]

please share a snippet of the code executed where this appeared, so we all have something reproducible to discuss

[RafaelJohn9]

Traceback (most recent call last):
  File "/home/john/miniconda3/envs/py310/lib/python3.10/site-packages/gradio/queueing.py", line 622, in process_events
    response = await route_utils.call_process_api(
  File "/home/john/miniconda3/envs/py310/lib/python3.10/site-packages/gradio/route_utils.py", line 323, in call_process_api
    output = await app.get_blocks().process_api(
  File "/home/john/miniconda3/envs/py310/lib/python3.10/site-packages/gradio/blocks.py", line 2016, in process_api
    result = await self.call_function(
  File "/home/john/miniconda3/envs/py310/lib/python3.10/site-packages/gradio/blocks.py", line 1569, in call_function
    prediction = await anyio.to_thread.run_sync(  # type: ignore
  File "/home/john/.local/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/home/john/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
  File "/home/john/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
  File "/home/john/miniconda3/envs/py310/lib/python3.10/site-packages/gradio/utils.py", line 846, in wrapper
    response = f(*args, **kwargs)
  File "/mnt/sdb1/repositories/rag_assistant/rag_assistant/gradio_ui.py", line 244, in _reply_to_chat
    self.assistant.agent.exec(
  File "/mnt/sdb1/repositories/swarmauri-sdk/pkgs/swarmauri/swarmauri/agents/concrete/RagAgent.py", line 105, in exec
    raise e
  File "/mnt/sdb1/repositories/swarmauri-sdk/pkgs/swarmauri/swarmauri/agents/concrete/RagAgent.py", line 72, in exec
    self.last_retrieved = self.vector_store.retrieve(
  File "/mnt/sdb1/repositories/swarmauri-sdk/pkgs/swarmauri/swarmauri/vector_stores/concrete/Doc2VecVectorStore.py", line 63, in retrieve
    transform_matrix = self._embedder.fit_transform(documents)
  File "/mnt/sdb1/repositories/swarmauri-sdk/pkgs/swarmauri/swarmauri/embeddings/concrete/Doc2VecEmbedding.py", line 49, in fit_transform
    self.fit(documents, **kwargs)
  File "/mnt/sdb1/repositories/swarmauri-sdk/pkgs/swarmauri/swarmauri/embeddings/concrete/Doc2VecEmbedding.py", line 34, in fit
    self._model.build_vocab(tagged_data)
  File "/home/john/.local/lib/python3.10/site-packages/gensim/models/doc2vec.py", line 888, in build_vocab
    report_values = self.prepare_vocab(update=update, keep_raw_vocab=keep_raw_vocab, trim_rule=trim_rule, **kwargs)
  File "/home/john/.local/lib/python3.10/site-packages/gensim/models/word2vec.py", line 769, in prepare_vocab
    self.wv.sort_by_descending_frequency()
  File "/home/john/.local/lib/python3.10/site-packages/gensim/models/keyedvectors.py", line 757, in sort_by_descending_frequency
    self.vectors = self.vectors[count_sorted_indexes]
IndexError: index 12 is out of bounds for axis 0 with size 11

[RafaelJohn9]

There are two reasons why this issue showed up:

Adding more doc inside a vectorstore with new vocab

• In the Doc2VecVectorStore , when the vector store was initialized and added documents, it does it with no errors, however if you add more documents after that have new vocabs in the vector store again it causes the IndexError

Potential solution

• genism's build_vocab method has the parameter update= which is usually False as the default though if changed to True after build it fixes the problem. Below is an example

# Check if the model already has a vocabulary built
        if len(self._model.wv) == 0:
            self._model.build_vocab(
                tagged_data
            )  # Build vocabulary if not already built
        else:
            self._model.build_vocab(
                tagged_data, update=True
            )  # Update the vocabulary if it exists

Using OOV during retrieval

Using Out Of Vocabluary (OOV) words during retrieval also causes this errors.

Potential solution

Check if the word is already a known vocab before retrieval.

def infer_vector(self, data: str) -> Vector:
        words = data.split()
        # Check if words are known to the model's vocabulary
        known_words = [word for word in words if word in self._model.wv]

        if not known_words:
            # Return a zero-vector if all words are OOV
            vector = [0.0] * self._model.vector_size
        else:
            # Infer vector from known words
            vector = self._model.infer_vector(known_words)

        return Vector(value=vector)

Solved all this in PR: #676
Added tests in PR: #677