Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possibility to tap into already executed data #207

Open
stychu opened this issue Jun 26, 2024 · 0 comments
Open

Possibility to tap into already executed data #207

stychu opened this issue Jun 26, 2024 · 0 comments

Comments

@stychu
Copy link

stychu commented Jun 26, 2024

Hi I want to hit my external API with bunch of requests. API allows for 30 TPS. I need to execute 500 requests which each request returns 100 objects that I need to save to the database. Is it possible to tap into the queue while the queue is still running and access already resolved data? It will be a bit heavy process on memory where I need to wait till the whole queue resolve and only then I can access my in memory result of 50 000 objects to be saved into database.

If I have pqueue as

const queue = new PQueue({
    intervalCap: 30,
    interval: 1000,
    carryoverConcurrencyCount: true,
  });

const result = await queue.addAll([500requestPromises])

I would like to be able to tap into the data and every batch save to database so approx. 17 writes to database. Is it possible ??

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant