General feedback / thoughts

[mckernanin]

Thanks for the quick fix on #2! Just wanted to share some feedback on how this is currently working for me.

Figuring out a solution for optimistically grabbing new jobs when the queue is idle would be fantastic. I have one queue that currently runs ~700 jobs once a day, which will also handle intermittent jobs throughout the day (I run a job after a user completes a few steps, then schedule and run daily going forward). Currently running at 2 concurrency, I could up the polling rate, but for the majority of the day, it'll be idle. I could up the concurrency, but an API that I rely on can get cranky if it's hit too hard (and other parts of my application hit the same API). Jobs take 2-5s to complete, so the queue ends up being idle for a substantial amount of the 10s default polling time.

I also pushed some changes today that I'll monitor over the next few days, I was using a userId as my key in separate queues. I think but am not positive that queues were deleting each other's jobs. I have since updated my keys to be userId + more information, I'll let you know how that goes.

Overall really like the library, I already use Prisma in my project and was about to look at implementing a pg-based worker queue myself when I came across your project.

[mgcrea]

Hey thanks for the feedback, I did notice the slow uptake of new jobs on one of my deployments and I agree that it would be nice to fix that. I'd like to have some nice reproducible test cases to assess the queue's behaviour and properly test against several queue workloads, will try to dig a bit on this.

Regarding the queues deleting each others jobs, I just released a 1.9.1 that fixes it.

[mgcrea]

So I ended up refactoring a bit the code and this should allow for smooth job chaining until the estimated queue size (available size of the queue at the time of the poll) is down to zero while processing them concurrently up to maxConcurrency.

Regarding CPU usage, there is now a jobInterval * 2 wait (50*2 = 100ms) while loop that is waiting for available concurrency while waiting for current jobs to finish when there is a long queue. I think a 100ms tight loop is relax enough to not tax the CPU at all but still interesting if you have numbers on it.

I have released a 1.10.3 with the improvements, let me know if it works for you.

You can glance at the changes here: 1.9.1...1.10.3#diff-9a9411d8c98c21f5ba0cbd900476870eea9c239077f02689846946b88077c423

[mckernanin]

Awesome, I'll update and let you know how it goes.