[BUG]: RAG feature returning incorrect count of records from uploaded input file

[punk1234567]

How are you running AnythingLLM?

Running Anything LLM executable version on my laptop locally.

What happened?

Describe the bug
When exploring the RAG feature of Anything LLM with my local documents, the response is incorrect.
Example: I upload a .md document containing 102 records and ask: How many records or projects are listed in the input document?" The RAG system returns "4", which is inaccurate.

Expected behavior
The correct number of records or projects from the source document should be returned (102 in this case).

Environment

OS: Windows 11

Embedding Model: Default
Vector database: Default
LLM Model Provider: Mistral Large
Other settings: All default
I experimented using other LLM models and file format as input, but still the same result.

Are there known steps to reproduce?

To Reproduce

Add .md documents to the Knowledge Base (document contains 102 records).
Go to Chat.
Select the uploaded file in the Knowledge Collection.
Ask: "How many records in the input?"
Observed result: Returns "4" records.
Expected result: Should return "102" records.

[shatfield4]

This is most likely due to how RAG works. This link will help you understand more about how this works. When you are chatting with a large document and embed a file, you are not actually giving the entire document as context to your LLM but just chunks of text that relate to the question you are asking.

What you can try here is document pinning.

An update is coming to the desktop version of AnythingLLM soon allowing you to utilize the entire context window of your LLM without having to use document pinning :)

[punk1234567]

Thank you very much for your prompt and detailed response.

I will certainly explore document pinning as a workaround. At the same time, I’m eagerly anticipating the upcoming update that will enable utilization of the entire context window of the LLM.

Could you kindly share an approximate timeline for when this feature is expected to be released? Thanks again for your support.

[timothycarambat]

@punk1234567 For Desktop, we were going to do a 1.8.5 release this week. The timing is tight right now, and we really wanted to get #4279 in for this release to make it solid 1.9.0, but because this UX improvement is so massive and very useful, we might just do two quick releases back to back.

I'm a little unsure at this time, but you will know as we announce it here, on the the docs page, and in Discord + youll see the version number highlight orange in the desktop app when it detects a new version exists.

[punk1234567]

Thank you so much for the update. I understand that 1.8.5 is on priority. Look forward to the 1.9.0 release whenever it is ready. Truly appreciate the heads up.

[punk1234567]

Just a gentle follow up to check when the updated RAG support feature we talked about in this email thread would be implemented. Thanks in advance.

[timothycarambat]

@punk1234567 It is live, we released 1.8.5-r2 last friday which has this functionality now when you drag and drop a document in chat.