[Question]: Maximum recursion depth exceeded while calling a Python object

[benhgm]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Following the tutorial for Single Query Transformation, I got the error RecursionError: maximum recursion depth exceeded while calling a Python object

I am following the notebook closely. My dataset is different. I am using excel data that is converted to text format. The text length is between 50000 to 100000 characters. Could this be an issue? How can I solve this?

[dosubot[bot]]

🤖

Hello,

The RecursionError: maximum recursion depth exceeded while calling a Python object error you're encountering could be due to the use of the RecursiveRetriever module in the Single Query Transformation tutorial. This module is designed to recursively explore links from nodes to other retrievers or query engines, which can lead to recursive function calls, especially when dealing with hierarchical data or large datasets. This might be the case with your dataset of 50000 to 100000 characters.

To resolve this issue, you might need to adjust the recursion limit in Python using sys.setrecursionlimit(). However, this should be done with caution as it can lead to a crash if the limit is too high. Alternatively, you could modify the use of the RecursiveRetriever module in your code to prevent excessive recursion.

Here is a similar issue that was resolved: [Bug]: RecursionError: maximum recursion depth exceeded while calling a Python object for HierarchicalNodeParser. This issue was closed by a pull request: Improve error logs for text splitters.

For a more accurate solution, could you please provide more details about the transformation rules and the structure of your dataset? This will help in understanding the exact cause of the issue.

Best,
Dosu

Sources

• [Bug]: RecursionError: maximum recursion depth exceeded while calling a Python object for HierarchicalNodeParser
• llama_index/query_engine/pandas_query_engine.py
• docs/examples/query_engine/pdf_tables/recursive_retriever.ipynb
• llama_index/retrievers/recursive_retriever.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[benhgm]

@dosu-bot each document is 50000 - 100000 characters long in the knowledge graph. Is this causing recursion error?

[dosubot[bot]]

🤖

Hello,

Yes, the length of your documents could indeed be causing the RecursionError: maximum recursion depth exceeded while calling a Python object. This error typically occurs when the depth of the call stack exceeds Python's maximum recursion limit. In the context of the RecursiveRetriever class in LlamaIndex, this could happen if the documents you're working with have a deeply nested structure that causes the _retrieve_rec method to be called recursively many times.

Even though you've tried adjusting the recursion limit in Python using sys.setrecursionlimit(), this is generally not a recommended solution because it can lead to a crash if it exceeds the stack depth limit of your Python interpreter.

One solution could be to refactor the _retrieve_rec method to use an iterative approach instead of a recursive one. This could involve using a stack or queue data structure to keep track of the nodes that need to be processed, and then processing each node in a loop until there are no more nodes left to process.

Another solution could be to simplify the structure of your documents. For example, you could try flattening the hierarchical relationships in your documents, or breaking your documents up into smaller, less complex pieces.

Finally, you could try reducing the similarity_top_k parameter in your VectorStoreIndex retriever. This parameter controls the number of most similar nodes that the retriever returns, and reducing it could help to limit the depth of the recursion.

You can refer to the following sources for more information:

• llama_index/retrievers/recursive_retriever.py
• docs/examples/query_engine/pdf_tables/recursive_retriever.ipynb

As for the default recursion limit set in Python, it's typically 1000, but it can vary depending on the platform and Python implementation. However, I wasn't able to find specific information on how this affects the functioning of the LlamaIndex framework in the repository.

I hope this helps! Let me know if you have any other questions.

Sources

• [Bug]: Knowledge Graph is not working
• llama_index/retrievers/recursive_retriever.py
• docs/examples/query_engine/pdf_tables/recursive_retriever.ipynb

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Disiok]

Hey @bengm, it'd be super helpful to post a stack trace to help us debug.