Feat/workers per api instance

[sanggusti]

What does this PR do?

Added feature requested on enabling ability to control workers_per_device per API instance #572 .

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

Tests are passing locally. I added coverage in tests/unit/test_lit_server.py and verified

On my device (4 x T4) the tests results in:

⚡ feat/workers-per-api-instance ~/LitServe pytest -q tests/unit/test_lit_server.py -k workers_per_device_can_be_configured_per_route
..                                                                                                                    [100%]
2 passed, 49 deselected, 7 warnings in 2.10s

Repro (4× T4):

pytest -q tests/unit/test_lit_server.py -k workers_per_device_can_be_configured_per_route

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Definitely!

[sanggusti]

The CI and e2e tests stuck at test_worker_restart_and_server_shutdown and went timeout. Any suggestion?

[sanggusti]

Precommit and tests successful. I hope this PR could get reviewed

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 85%. Comparing base (d244f2c) to head (fe0b2e6).

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #646   +/-   ##
===================================
- Coverage    85%    85%   -0%     
===================================
  Files        39     39           
  Lines      3212   3261   +49     
===================================
+ Hits       2721   2762   +41     
- Misses      491    499    +8     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sanggusti]

Hmmm, failed on macos. Any suggestion?

[sanggusti]

request for review. Need approval for pending checks, cc @bhimrazy @aniketmaurya

[sanggusti]

I've see that all tests (automatic and dispatch) and codecov is passed. Looking to see review about this PR

[sanggusti]

Weird, it passed before in 3.12. Can elaborate @bhimrazy?

[bhimrazy]

Weird, it passed before in 3.12. Can elaborate @bhimrazy?

yeah, it looks like some flaky tests are reappearing after the merge of #644. I’ll investigate this shortly. It’s failing on the main branch as well.

[sanggusti]

@bhimrazy looks like in main it is passed now, but somehow only fail in ubuntu 3.10

[bhimrazy]

@bhimrazy looks like in main it is passed now, but somehow only fail in ubuntu 3.10

yeah, it could be another flaky test 😅

[sanggusti]

I actually have no clue here, should I make some change or else? @bhimrazy

[bhimrazy]

I actually have no clue here, should I make some change or else? @bhimrazy

Maybe try triggering the CI again with an empty commit.

btw, we’re still waiting on a review/decision from @andyland @aniketmaurya.

[sanggusti]

Alright, I've added some minimal comment commit to trigger the test. Can you retrigger the pending checks? @bhimrazy

[sanggusti]

Seems like it passed all the checks required, looking for review of the code owner. cc @bhimrazy @aniketmaurya