Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: use global thread pool to support prefetch #1187

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

storage: use global thread pool to support prefetch #1187

wants to merge 1 commit into from
+243 −225

Conversation

jiangliu
Copy link
Collaborator

Currently we use a prefetch manager per backend, this design has two drawbacks:

  • spawn too many worker threads for prefetch
  • the network ratelimit has been defunct

So change the design to use a global prefetch manager with thread pool to supporrt all prefetch requests.

@jiangliu jiangliu requested a review from a team as a code owner March 31, 2023 02:40
@jiangliu jiangliu requested review from bergwolf, liubogithub and imeoer and removed request for a team March 31, 2023 02:40
@anolis-bot
Copy link
Collaborator

@jiangliu , a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/63200

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd❌ FAIL

Sorry, your test job failed. Please get the details in the link.

@jiangliu
storage: use global thread pool to support prefetch
fbe54a2 
Currently we use a prefetch manager per backend, this design has two
drawbacks:
- spawn too many worker threads for prefetch
- the network ratelimit has been defunct

So change the design to use a global prefetch manager with thread pool
to supporrt all prefetch requests.

Signed-off-by: Jiang Liu <[email protected]>
@anolis-bot
Copy link
Collaborator

@jiangliu , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/63211

@anolis-bot
Copy link
Collaborator

@jiangliu , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@jiangliu jiangliu requested a review from changweige March 31, 2023 06:47
ccx1024cc
ccx1024cc suggested changes Oct 30, 2023
View reviewed changes
Copy link
Contributor

@ccx1024cc ccx1024cc left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A test for improvement is required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ccx1024cc ccx1024cc ccx1024cc requested changes

@bergwolf bergwolf Awaiting requested review from bergwolf bergwolf is a code owner automatically assigned from dragonflyoss/nydus-reviewers

@liubogithub liubogithub Awaiting requested review from liubogithub liubogithub is a code owner automatically assigned from dragonflyoss/nydus-reviewers

@imeoer imeoer Awaiting requested review from imeoer imeoer is a code owner automatically assigned from dragonflyoss/nydus-reviewers

@changweige changweige Awaiting requested review from changweige

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
anolis_test_pass
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jiangliu @anolis-bot @ccx1024cc