Replies: 1 comment 1 reply
-
Hey there, @OsirisMP! I'm here to help you out with any bugs, questions, or contributions you have while waiting for a human maintainer. Let's figure this out together! The issue you're encountering with
These steps should help you manage and query larger CSV datasets more effectively. If the problem persists, consider checking the data consistency and ensuring that the CSV files are correctly formatted and loaded. |
Beta Was this translation helpful? Give feedback.
-
Hello. So I would be happy if someone could help.
Trying to add some csv data to VectoreStoreIndex to query on like "What is the CodeName for Code".
Using SimpleDirectoryReader I gave it csv with 100 rows with 2 columns Code and CodeName. Then created index like:
index = VectorStoreIndex.from_documents
It gave 50% wrong answers for given Codes.
So I gave it only 50 rows. It knew everything perfectly. What is the limitation ?
As I dont know why, I tried to split the csvs into 2 with 50, 50 rows using the following code:
data = SimpleDirectoryReader(input_dir="./diagnozy_semicol_noclear_0-50_50-100/").load_data(show_progress=True)
index = VectorStoreIndex.from_documents(data)
It completely forgot first 50 rows but knew perfectly rows 51-100. What is happening? How to teach it more then a few rows?
Thank you so much, I am completele lost.
Beta Was this translation helpful? Give feedback.
All reactions