Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support generating separate blob for prefetched files #1629

Merged
merged 1 commit into from
Oct 29, 2024

Conversation

daiyongxuan
Copy link
Contributor

Relevant Issue #1589

The objective is to create an independent image layer to prioritize the fetching of necessary blobs during startup, enhancing performance.

Details

This introduces the "optimize" subcommand, which generates a new blob containing prefetch files and creates a new RAFS format bootstrap. This ensures compatibility with the nydus-image check and optimizes the startup process.

@daiyongxuan daiyongxuan requested a review from a team as a code owner September 25, 2024 03:09
@daiyongxuan daiyongxuan requested review from jiangliu, gaius-qi and Desiki-high and removed request for a team September 25, 2024 03:09
Copy link

codecov bot commented Oct 22, 2024

Codecov Report

Attention: Patch coverage is 1.40845% with 350 lines in your changes missing coverage. Please review.

Project coverage is 60.10%. Comparing base (15ec192) to head (7127c1e).
Report is 9 commits behind head on master.

Files with missing lines Patch % Lines
builder/src/optimize_prefetch.rs 0.00% 197 Missing ⚠️
src/bin/nydus-image/main.rs 0.00% 90 Missing ⚠️
storage/src/device.rs 0.00% 21 Missing ⚠️
builder/src/core/tree.rs 0.00% 16 Missing ⚠️
storage/src/cache/cachedfile.rs 12.50% 6 Missing and 1 partial ⚠️
builder/src/core/overlay.rs 0.00% 6 Missing ⚠️
src/bin/nydus-image/deduplicate.rs 0.00% 5 Missing ⚠️
builder/src/core/context.rs 0.00% 4 Missing ⚠️
src/bin/nydus-image/inspect.rs 0.00% 1 Missing ⚠️
storage/src/cache/dedup/db.rs 66.66% 0 Missing and 1 partial ⚠️
... and 2 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1629      +/-   ##
==========================================
- Coverage   60.43%   60.10%   -0.33%     
==========================================
  Files         146      147       +1     
  Lines       48841    49513     +672     
  Branches    46322    46994     +672     
==========================================
+ Hits        29517    29760     +243     
- Misses      17558    17971     +413     
- Partials     1766     1782      +16     
Files with missing lines Coverage Δ
builder/src/chunkdict_generator.rs 0.00% <ø> (ø)
builder/src/core/blob.rs 42.63% <100.00%> (ø)
builder/src/lib.rs 63.15% <ø> (ø)
rafs/src/metadata/inode.rs 90.51% <ø> (ø)
rafs/src/metadata/layout/v6.rs 85.03% <ø> (ø)
storage/src/utils.rs 95.98% <100.00%> (+0.10%) ⬆️
src/bin/nydus-image/inspect.rs 0.00% <0.00%> (ø)
storage/src/cache/dedup/db.rs 79.18% <66.66%> (+0.17%) ⬆️
storage/src/meta/mod.rs 72.17% <0.00%> (-0.05%) ⬇️
storage/src/meta/toc.rs 70.96% <0.00%> (-0.10%) ⬇️
... and 8 more

... and 2 files with indirect coverage changes

@imeoer imeoer changed the title Support Chunk Prefetch By Generating a New Blob And Bootstrap support generating separate blob for prefetched files Oct 23, 2024
storage/src/backend/localfs.rs Outdated Show resolved Hide resolved
src/bin/nydus-image/main.rs Outdated Show resolved Hide resolved
src/bin/nydus-image/main.rs Outdated Show resolved Hide resolved
src/bin/nydus-image/main.rs Show resolved Hide resolved
output/chunkdict_bootstrap Outdated Show resolved Hide resolved
@daiyongxuan daiyongxuan requested a review from a team as a code owner October 25, 2024 03:58
@daiyongxuan daiyongxuan requested review from liubin and removed request for a team October 25, 2024 03:58
@@ -81,6 +81,7 @@ func (n *NativeLayerTestSuite) TestMakeLayers() test.Generator {
ctx.Runtime.RafsMode = scenario.GetString(paramRafsMode)
ctx.Runtime.EnablePrefetch = scenario.GetBool(paramEnablePrefetch)
ctx.Runtime.AmplifyIO = scenario.GetUInt64(paramAmplifyIO)
ctx.Runtime.ChunkDedupDb = scenario.GetString(paramChunkDedupDb)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like an unnecessary merge commit.

builder/src/chunkdict_generator.rs Outdated Show resolved Hide resolved
builder/src/chunkdict_generator.rs Outdated Show resolved Hide resolved
builder/src/chunkdict_generator.rs Outdated Show resolved Hide resolved
builder/src/chunkdict_generator.rs Outdated Show resolved Hide resolved
builder/src/chunkdict_generator.rs Outdated Show resolved Hide resolved
builder/src/chunkdict_generator.rs Outdated Show resolved Hide resolved
src/bin/nydus-image/main.rs Outdated Show resolved Hide resolved
tree: &mut Tree,
ctx: &mut BuildContext,
bootstrap_mgr: &mut BootstrapManager,
blobtable: &mut RafsV6BlobTable,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blob_table, and need also to be compatible with RAFS v5.

builder/src/chunkdict_generator.rs Outdated Show resolved Hide resolved
builder/src/chunkdict_generator.rs Outdated Show resolved Hide resolved
@daiyongxuan daiyongxuan force-pushed the master branch 4 times, most recently from 7127c1e to a2a2478 Compare October 29, 2024 03:38
We can statically analyze the image entrypoint dependency, or use runtime dynamic
analysis technologies such as ebpf, fanotify, metric, etc. to obtain the container
file access pattern, and then build this part of data into an independent image layer:

* preferentially fetch blob during the image startup phase to reduce network and disk IO.
* avoid frequent image builds, allows for better local cache utilization.

Implement optimize subcommand to optimize image bootstrap
from a prefetch file list, generate a new blob.

```
nydus-image optimize --prefetch-files /path/to/prefetch-files.txt \
  --bootstrap /path/to/bootstrap \
  --blob-dir /path/to/blobs
```
This will generate a new bootstrap and new blob in `blob-dir`.

Signed-off-by: daiyongxuan <[email protected]>
Copy link
Collaborator

@imeoer imeoer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM!

@imeoer imeoer merged commit 375f55f into dragonflyoss:master Oct 29, 2024
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants