feat: add parallel processing to SimpleDirectoryReader #830

ghost · 2024-05-10T18:54:58Z

Second part of #808

Reworked everything, a lot cleaner and simpler now and implemented straight into SimpleDirectoryReader.

Did not overload LlamaParse when testing this time.
The data in the example is now just the brk-2022.pdf split in 6 parts.

changeset-bot · 2024-05-10T18:55:02Z

🦋 Changeset detected

Latest commit: c8ddbc2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 9 packages

Name	Type
llamaindex	Patch
docs	Patch
@llamaindex/experimental	Patch
@llamaindex/cloudflare-worker-agent-test	Patch
@llamaindex/next-agent-test	Patch
@llamaindex/nextjs-edge-runtime-test	Patch
@llamaindex/waku-query-engine-test	Patch
@llamaindex/autotool-01-node-example	Patch
@llamaindex/autotool-02-next-example	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vercel · 2024-05-10T18:55:04Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
llama-index-ts-docs	❌ Failed (Inspect)			May 22, 2024 6:14pm

himself65 · 2024-05-10T22:29:48Z

packages/core/src/readers/SimpleDirectoryReader.edge.ts

+    );
+
+    const results = await Promise.all(workerPromises);
+    docs.push(...results.flat());


Suggested change

docs.push(...results.flat());

docs.push(...results.flat());

use Array.prototype.concat, push will throw error when results larger than 1000 items

marcusschiesser · 2024-05-14T07:55:39Z

packages/core/src/readers/SimpleDirectoryReader.edge.ts

@@ -46,19 +47,53 @@ export class SimpleDirectoryReader implements BaseReader {
    const {
      directoryPath,
      fs = defaultFS,
-      defaultReader = new TextFileReader(),
+      defaultReader,


don't change default reader

marcusschiesser · 2024-05-14T07:57:31Z

packages/core/src/readers/SimpleDirectoryReader.edge.ts

-        if (fileExtToReader && fileExt in fileExtToReader) {
-          reader = fileExtToReader[fileExt];
-        } else if (defaultReader != null) {
+        if (defaultReader) {


revert code - defaultReader is supposed to be the fallback to fileExtToReader

Oh, misinterpreted the code then.
In that case, suggestion: add a new param to overwrite all fileExtToReader if defined.

Could become more important with the addition of more external readers that are capable of reading multiple formats.

marcusschiesser · 2024-05-14T07:58:35Z

packages/core/src/readers/SimpleDirectoryReader.edge.ts

    } = params;

+    // Check if LlamaParseReader is used as the defaultReader and if so checks if numWorkers is in the valid range
+    if (defaultReader instanceof LlamaParseReader && numWorkers > 9) {


doesn't work with setting LlamaParseReader in fileExtToReader -
suggestion: generally don't allow more than 9 workers

marcusschiesser · 2024-05-14T08:01:38Z

packages/core/src/readers/SimpleDirectoryReader.edge.ts

+      filePathQueue.push(filePath);
+    }
+
+    const workerPromises = Array.from({ length: numWorkers }, () =>


is this handling queue.length > numWorkers? maybe using something like p-limit helps?

Yes it is, I thought about using p-limit, I just always try to avoid adding packages if possible. But will check it out and test around. Thanks for the review!

marcusschiesser

@MightBeAScam looks great! just a few notes and it needs to be rebased

marcusschiesser · 2024-05-21T14:56:57Z

packages/core/src/readers/SimpleDirectoryReader.edge.ts

-        let reader: BaseReader;
+    const limit = pLimit(numWorkers);
+    const workerPromises = filePathQueue.map((filePath) =>
+      limit(() => this.processFiles(filePath, processFilesParams)),


isn't it processing just one file?

Suggested change

limit(() => this.processFiles(filePath, processFilesParams)),

limit(() => this.processFile(filePath, processFilesParams)),

marcusschiesser · 2024-05-21T14:57:46Z

examples/readers/package.json

@@ -11,7 +11,7 @@
    "start:pdf": "node --import tsx ./src/pdf.ts",
    "start:llamaparse": "node --import tsx ./src/llamaparse.ts",
    "start:notion": "node --import tsx ./src/notion.ts",
-    "start:llamaparse2": "node --import tsx ./src/llamaparse_2.ts"
+    "start:llamaparse2": "node --import tsx ./src/simple-directory-reader-with-llamaparse.ts"


Suggested change

"start:llamaparse2": "node --import tsx ./src/simple-directory-reader-with-llamaparse.ts"

"start:llamaparse-dir": "node --import tsx ./src/simple-directory-reader-with-llamaparse.ts"

marcusschiesser · 2024-05-21T14:58:23Z

.changeset/stale-dots-search.md

+"llamaindex": patch
+---
+
+add p-limit for enhanced concurrency managment


Suggested change

add p-limit for enhanced concurrency managment

add concurrency managment for SimpleDirectoryReader

…cessing

…ader

KindOfAScam · 2024-05-22T20:23:47Z

@marcusschiesser fixed the suggested changes and rebased the branch. Seems like I did something wrong, is there a way to check the vercel for git logs if the deploy is not in my account? Not really sure what could be the issue and where to start

himself65

This account has been deleted, closing

himself65 · 2024-05-23T10:12:13Z

KindOfAScam · 2024-05-24T14:38:05Z

@himself65 sorry my bad, realized 2 accounts are stupid, so I deleted the newer one..

But I messed up rebasing anyway, so let me try again and I'll open a new PR

vercel bot deployed to Preview May 10, 2024 19:23 View deployment

vercel bot deployed to Preview May 10, 2024 22:27 View deployment

himself65 requested changes May 10, 2024

View reviewed changes

vercel bot deployed to Preview May 10, 2024 22:57 View deployment

ghost requested a review from himself65 May 10, 2024 22:58

vercel bot deployed to Preview May 11, 2024 06:16 View deployment

vercel bot deployed to Preview May 13, 2024 17:01 View deployment

marcusschiesser reviewed May 14, 2024

View reviewed changes

ghost marked this pull request as draft May 14, 2024 16:51

ghost marked this pull request as ready for review May 15, 2024 18:26

vercel bot deployed to Preview May 15, 2024 18:42 View deployment

ghost requested a review from marcusschiesser May 15, 2024 21:07

marcusschiesser reviewed May 21, 2024

View reviewed changes

KindOfAScam and others added 14 commits May 22, 2024 17:19

feat: add parallel processing to SimpleDirectoryReader

299076b

feat: more structured parallel processing

e96aad0

feat: SimpleDirectoryReader respect LlamaParse worker limit

9e7efb5

example: LlamaParseReader in SimpleDirectoryReader using parallel pro…

392f3df

…cessing

covering up the last traces of my fruitless tries of adding showProgress

41508a3

fix: defaultReader in SimpleDirectpryReader

ff1f539

fix: review changes

9ee8ec9

fix: use flat

333e6cc

revert changes to defaultReader

a452d0e

feat: add a new param to be reader for all filetypes if defined

e7c640c

fix exampled + increase LlamaParse verbosity

abe80bf

feat: add p-limit to manage concurrency requests in SimpleDirectoryRe…

443e4fe

…ader

fix: review changes

598eda4

rebase cleanup + some docs

128bd26

KindOfAScam force-pushed the Batch branch from 239ec5c to 128bd26 Compare May 22, 2024 18:11

Merge branch 'main' into Batch

c8ddbc2

vercel bot had a problem deploying to Preview May 22, 2024 18:14 Failure

himself65 requested changes May 23, 2024

View reviewed changes

himself65 closed this May 23, 2024

KindOfAScam mentioned this pull request May 24, 2024

feat: add parallel processing to SimpleDirectoryReader #883

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add parallel processing to SimpleDirectoryReader #830

feat: add parallel processing to SimpleDirectoryReader #830

ghost commented May 10, 2024

changeset-bot bot commented May 10, 2024 •

edited

Loading

vercel bot commented May 10, 2024 •

edited

Loading

himself65 May 10, 2024

marcusschiesser May 14, 2024

marcusschiesser May 14, 2024

ghost May 14, 2024

marcusschiesser May 14, 2024

marcusschiesser May 14, 2024

ghost May 14, 2024

marcusschiesser left a comment

marcusschiesser May 21, 2024

marcusschiesser May 21, 2024

marcusschiesser May 21, 2024

KindOfAScam commented May 22, 2024

himself65 left a comment

himself65 commented May 23, 2024

KindOfAScam commented May 24, 2024

	limit(() => this.processFiles(filePath, processFilesParams)),
	limit(() => this.processFile(filePath, processFilesParams)),

	"start:llamaparse2": "node --import tsx ./src/simple-directory-reader-with-llamaparse.ts"
	"start:llamaparse-dir": "node --import tsx ./src/simple-directory-reader-with-llamaparse.ts"

	add p-limit for enhanced concurrency managment
	add concurrency managment for SimpleDirectoryReader

feat: add parallel processing to SimpleDirectoryReader #830

feat: add parallel processing to SimpleDirectoryReader #830

Conversation

ghost commented May 10, 2024

changeset-bot bot commented May 10, 2024 • edited Loading

🦋 Changeset detected

vercel bot commented May 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcusschiesser left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KindOfAScam commented May 22, 2024

himself65 left a comment

Choose a reason for hiding this comment

himself65 commented May 23, 2024

KindOfAScam commented May 24, 2024

changeset-bot bot commented May 10, 2024 •

edited

Loading

vercel bot commented May 10, 2024 •

edited

Loading