support generating new image with separate blob for prefetched files #1649
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Maxing1997
requested review from
imeoer,
jiangliu and
gaius-qi
and removed request for
a team
Maxing1997
changed the title
Maxing1997
force-pushed
the
maxing_mr/nydusify_optimize
branch
from
08bff89 to
8437d8d
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1649 +/- ##
==========================================
- Coverage 60.10% 60.01% -0.10%
==========================================
Files 147 147
Lines 49513 49583 +70
Branches 46994 47037 +43
==========================================
- Hits 29762 29756 -6
- Misses 17970 18045 +75
- Partials 1781 1782 +1
|
imeoer
reviewed
Dec 4, 2024
|
Thanks! |
Maxing1997
force-pushed
the
maxing_mr/nydusify_optimize
branch
5 times, most recently
from
78717fb to
e43282e
Compare
imeoer
reviewed
Dec 9, 2024
| Blobs []string `json:"blobs"` | ||
| } | ||
|
|
||
| func Build(option BuildOption) (string, error) { |
There was a problem hiding this comment.
this conflicts Optimize declaration in optimizer.go
added 2 commits
December 9, 2024 12:02
Major changes: 1. Added compatibility for rafs v5/v6 formats; 2. Set IS_SEPARATED_WITH_PREFETCH_FILES flag in BlobInfo for prefetchblob; 3. Add option output-json to store build output. Signed-off-by: Xing Ma <[email protected]>
We can statically analyze the image entrypoint dependency, or use runtime dynamic analysis technologies such as ebpf, fanotify, metric, etc. to obtain the container file access pattern, and then build this part of data into an independent image layer: * preferentially fetch blob during the image startup phase to reduce network and disk IO. * avoid frequent image builds, allows for better local cache utilization. Implement optimize subcommand of nydusify to generate a new image, which references a new blob included prefetch file chunks. ``` nydusify optimize --policy separated-prefetch-blob \ --source $existed-nydus-image \ --target $new-nydus-image \ --prefetch-files /path/to/prefetch-files ``` More detailed process is as follows: 1. nydusify first downloads the source image and bootstrap, utilize nydus-image to output a new bootstrap along with an independent prefetchblob; 2. nydusify generate&push new meta layer including new bootstrap and the prefetch-files , also generates&push new manifest/config/prefetchblob, completing the incremental image build. Signed-off-by: Xing Ma <[email protected]>
Maxing1997
force-pushed
the
maxing_mr/nydusify_optimize
branch
from
e43282e to
6b13369
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relevant Issue #1589
The objective is to generate a new image, which references a new blob included prefetch file chunks. This will accelerate an startup performance due to high-priority pulling of prefetch blob.
Details
Implement optimize subcommand of nydusify to generate a new image, which references a new blob included prefetch file chunks.
Types of changes
What types of changes does your PullRequest introduce? Put an
xin all the boxes that apply:Checklist
Go over all the following points, and put an
xin all the boxes that apply.