Error converting image using chunk dict

[Jaycean]

Version of nydus being used (nydusd --version)

[root@master ~]# nydusd --version
Version:        v2.3.0
Git Commit:     a4683baa1ea8cafecc9013bc4d6a5259b5a82678
Build Time:     2024-10-11T02:51:53.888907475Z
Profile:        release
Rustc:          rustc 1.72.1 (d5c2e9c34 2023-09-13)

Version of nydus-snapshotter being used (containerd-nydus-grpc --version)

[root@master ~]# containerd-nydus-grpc --version
Version:     v0.13.13
Revision:    e9d1bb738f778a2e30b8284ca4f479fa2517456c
Go version:  go1.19.6
Build time:  2024-05-15T03:57:13

Kernel information (uname -r)

[root@master ~]# uname -r
4.18.0-348.el8.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

[root@master ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"

containerd-nydus-grpc command line used, if applicable (ps aux | grep containerd-nydus-grpc)

[root@master ~]# ps aux | grep containerd-nydus-grpc
root     51767  0.0  0.3 1351116 46656 ?       Ssl  20:16   0:03 /usr/bin/containerd-nydus-grpc --config /etc/nydus/config.toml
root     52243  0.0  0.0  12136  1128 pts/0    S+   22:35   0:00 grep --color=auto containerd-nydus-grpc

Screenshots (if applicable)

Details about issue

I use three of the following oci images.

docker pull jaycean/kylinos:java1-8-openjdk
docker pull jaycean/kylinos:python27
docker pull jaycean/kylinos:python28

I will convert three images into nydus images.

nydusify convert --source 192.168.237.131:80/library/kylinos:python27 --target 192.168.237.131:80/library/kylinos:python27-nydus  --source-insecure --target-insecure
nydusify convert --source 192.168.237.131:80/library/kylinos:python38 --target 192.168.237.131:80/library/kylinos:python38-nydus  --source-insecure --target-insecure
nydusify convert --source 192.168.237.131:80/library/kylinos:java1-8-openjdk --target 192.168.237.131:80/library/kylinos:java1-8-openjdk-nydus  --source-insecure --target-insecure

Calculate chunk dict using these three images.

nydusify chunkdict generate --sources 192.168.237.131:80/library/kylinos:python38-nydus,192.168.237.131:80/library/kylinos:python27-nydus,192.168.237.131:80/library/kylinos:java1-8-openjdk-nydus  --target 192.168.237.131:80/library/kylinos:chunk-dict --source-insecure --target-insecure

Using chunk dict to convert image, error occurred.

nydusify convert --source 192.168.237.131:80/library/kylinos:python27 --target 192.168.237.131:80/library/kylinos:python27-nydus-new --chunk-dict bootstrap:registry:192.168.237.131:80/library/kylinos:chunk-dict  --source-insecure --target-insecure

[root@master nydus-static]# nydusify convert --source 192.168.237.131:80/library/kylinos:python27 --target 192.168.237.131:80/library/kylinos:python27-nydus-new --chunk-dict bootstrap:registry:192.168.237.131:80/library/kylinos:chunk-dict  --source-insecure --target-insecure
INFO[2024-12-16T19:35:01-08:00] pulling image 192.168.237.131:80/library/kylinos:python27  module=converter
INFO[2024-12-16T19:35:01-08:00] trying next host                              error="failed to do request: Head \"https://192.168.237.131:80/v2/library/kylinos/manifests/python27\": http: server gave HTTP response to HTTPS client" host="192.168.237.131:80"
INFO[2024-12-16T19:35:01-08:00] try to pull with plain HTTP for 192.168.237.131:80/library/kylinos:python27  module=converter
INFO[2024-12-16T19:35:03-08:00] pulled image 192.168.237.131:80/library/kylinos:python27 , elapse 1.252592954s  module=converter
INFO[2024-12-16T19:35:03-08:00] converting image 192.168.237.131:80/library/kylinos:python27  module=converter
ERRO[2024-12-16T19:35:04-08:00] fail to run nydus-image [create --log-level warn --prefetch-policy fs --blob tmp/nydus-converter-905387813/blob.rafs --whiteout-spec none --fs-version 6 --type tar-rafs --blob-inline-meta --features blob-toc --chunk-dict bootstrap=./tmp/nydus-chunk-dict-2321129161 --compressor zstd --chunk-size 0x100000 tmp/nydus-converter-905387813/blob.targz]  error="exit status 1"
INFO[2024-12-16T19:35:04-08:00] [2024-12-16 19:35:04.268169 -08:00] ERROR [/src/error.rs:23] Error:  module=builder
INFO[2024-12-16T19:35:04-08:00]         "failed to get chunk info for chunk 0/0/60824"  module=builder
INFO[2024-12-16T19:35:04-08:00]         at rafs/src/metadata/direct_v6.rs:1366       module=builder
INFO[2024-12-16T19:35:04-08:00]         note: enable `RUST_BACKTRACE=1` env to display a backtrace  module=builder
INFO[2024-12-16T19:35:04-08:00] Error: failed to build tree from bootstrap    module=builder
INFO[2024-12-16T19:35:04-08:00]                                               module=builder
INFO[2024-12-16T19:35:04-08:00] Caused by:                                    module=builder
INFO[2024-12-16T19:35:04-08:00]     No such file or directory (os error 2)    module=builder
INFO[2024-12-16T19:35:04-08:00] [2024-12-16 19:35:04.273915 -08:00] ERROR [/src/error.rs:23] Error:  module=builder
INFO[2024-12-16T19:35:04-08:00]         "failed to get chunk info for chunk 0/0/60824"  module=builder
INFO[2024-12-16T19:35:04-08:00]         at rafs/src/metadata/direct_v6.rs:1366       module=builder
INFO[2024-12-16T19:35:04-08:00]         note: enable `RUST_BACKTRACE=1` env to display a backtrace  module=builder
INFO[2024-12-16T19:35:04-08:00] Error: failed to build tree from bootstrap    module=builder
INFO[2024-12-16T19:35:04-08:00]                                               module=builder
INFO[2024-12-16T19:35:04-08:00] Caused by:                                    module=builder
INFO[2024-12-16T19:35:04-08:00]     No such file or directory (os error 2)    module=builder
ERRO[2024-12-16T19:35:04-08:00] fail to run nydus-image [create --log-level warn --prefetch-policy fs --blob tmp/nydus-converter-3843820213/blob.rafs --whiteout-spec none --fs-version 6 --type tar-rafs --blob-inline-meta --features blob-toc --chunk-dict bootstrap=./tmp/nydus-chunk-dict-2321129161 --compressor zstd --chunk-size 0x100000 tmp/nydus-converter-3843820213/blob.targz]  error="exit status 1"
INFO[2024-12-16T19:35:04-08:00] [2024-12-16 19:35:04.280547 -08:00] ERROR [/src/error.rs:23] Error:  module=builder
INFO[2024-12-16T19:35:04-08:00]         "failed to get chunk info for chunk 0/0/60824"  module=builder
INFO[2024-12-16T19:35:04-08:00]         at rafs/src/metadata/direct_v6.rs:1366       module=builder
INFO[2024-12-16T19:35:04-08:00]         note: enable `RUST_BACKTRACE=1` env to display a backtrace  module=builder
INFO[2024-12-16T19:35:04-08:00] Error: failed to build tree from bootstrap    module=builder
INFO[2024-12-16T19:35:04-08:00]                                               module=builder
INFO[2024-12-16T19:35:04-08:00] Caused by:                                    module=builder
INFO[2024-12-16T19:35:04-08:00]     No such file or directory (os error 2)    module=builder
ERRO[2024-12-16T19:35:04-08:00] fail to run nydus-image [create --log-level warn --prefetch-policy fs --blob tmp/nydus-converter-2017203011/blob.rafs --whiteout-spec none --fs-version 6 --type tar-rafs --blob-inline-meta --features blob-toc --chunk-dict bootstrap=./tmp/nydus-chunk-dict-2321129161 --compressor zstd --chunk-size 0x100000 tmp/nydus-converter-2017203011/blob.targz]  error="exit status 1"
INFO[2024-12-16T19:35:04-08:00] [2024-12-16 19:35:04.294410 -08:00] ERROR [/src/error.rs:23] Error:  module=builder
INFO[2024-12-16T19:35:04-08:00]         "failed to get chunk info for chunk 0/0/60824"  module=builder
INFO[2024-12-16T19:35:04-08:00]         at rafs/src/metadata/direct_v6.rs:1366       module=builder
INFO[2024-12-16T19:35:04-08:00]         note: enable `RUST_BACKTRACE=1` env to display a backtrace  module=builder
INFO[2024-12-16T19:35:04-08:00] Error: failed to build tree from bootstrap    module=builder
INFO[2024-12-16T19:35:04-08:00]                                               module=builder
INFO[2024-12-16T19:35:04-08:00] Caused by:                                    module=builder
INFO[2024-12-16T19:35:04-08:00]     No such file or directory (os error 2)    module=builder
ERRO[2024-12-16T19:35:04-08:00] fail to run nydus-image [create --log-level warn --prefetch-policy fs --blob tmp/nydus-converter-3492053700/blob.rafs --whiteout-spec none --fs-version 6 --type tar-rafs --blob-inline-meta --features blob-toc --chunk-dict bootstrap=./tmp/nydus-chunk-dict-2321129161 --compressor zstd --chunk-size 0x100000 tmp/nydus-converter-3492053700/blob.targz]  error="exit status 1"
INFO[2024-12-16T19:35:04-08:00] [2024-12-16 19:35:04.289991 -08:00] ERROR [/src/error.rs:23] Error:  module=builder
INFO[2024-12-16T19:35:04-08:00]         "failed to get chunk info for chunk 0/0/60824"  module=builder
INFO[2024-12-16T19:35:04-08:00]         at rafs/src/metadata/direct_v6.rs:1366       module=builder
INFO[2024-12-16T19:35:04-08:00]         note: enable `RUST_BACKTRACE=1` env to display a backtrace  module=builder
INFO[2024-12-16T19:35:04-08:00] Error: failed to build tree from bootstrap    module=builder
INFO[2024-12-16T19:35:04-08:00]                                               module=builder
INFO[2024-12-16T19:35:04-08:00] Caused by:                                    module=builder
INFO[2024-12-16T19:35:04-08:00]     No such file or directory (os error 2)    module=builder
ERRO[2024-12-16T19:35:04-08:00] fail to run nydus-image [create --log-level warn --prefetch-policy fs --blob tmp/nydus-converter-2522871383/blob.rafs --whiteout-spec none --fs-version 6 --type tar-rafs --blob-inline-meta --features blob-toc --chunk-dict bootstrap=./tmp/nydus-chunk-dict-2321129161 --compressor zstd --chunk-size 0x100000 tmp/nydus-converter-2522871383/blob.targz]  error="exit status 1"
INFO[2024-12-16T19:35:04-08:00] [2024-12-16 19:35:04.304504 -08:00] ERROR [/src/error.rs:23] Error:  module=builder
INFO[2024-12-16T19:35:04-08:00]         "failed to get chunk info for chunk 0/0/60824"  module=builder
INFO[2024-12-16T19:35:04-08:00]         at rafs/src/metadata/direct_v6.rs:1366       module=builder
INFO[2024-12-16T19:35:04-08:00]         note: enable `RUST_BACKTRACE=1` env to display a backtrace  module=builder
INFO[2024-12-16T19:35:04-08:00] Error: failed to build tree from bootstrap    module=builder
INFO[2024-12-16T19:35:04-08:00]                                               module=builder
INFO[2024-12-16T19:35:04-08:00] Caused by:                                    module=builder
INFO[2024-12-16T19:35:04-08:00]     No such file or directory (os error 2)    module=builder
ERRO[2024-12-16T19:35:04-08:00] fail to run nydus-image [create --log-level warn --prefetch-policy fs --blob tmp/nydus-converter-2646883128/blob.rafs --whiteout-spec none --fs-version 6 --type tar-rafs --blob-inline-meta --features blob-toc --chunk-dict bootstrap=./tmp/nydus-chunk-dict-2321129161 --compressor zstd --chunk-size 0x100000 tmp/nydus-converter-2646883128/blob.targz]  error="exit status 1"
INFO[2024-12-16T19:35:04-08:00] [2024-12-16 19:35:04.300661 -08:00] ERROR [/src/error.rs:23] Error:  module=builder
INFO[2024-12-16T19:35:04-08:00]         "failed to get chunk info for chunk 0/0/60824"  module=builder
INFO[2024-12-16T19:35:04-08:00]         at rafs/src/metadata/direct_v6.rs:1366       module=builder
INFO[2024-12-16T19:35:04-08:00]         note: enable `RUST_BACKTRACE=1` env to display a backtrace  module=builder
INFO[2024-12-16T19:35:04-08:00] Error: failed to build tree from bootstrap    module=builder
INFO[2024-12-16T19:35:04-08:00]                                               module=builder
INFO[2024-12-16T19:35:04-08:00] Caused by:                                    module=builder
INFO[2024-12-16T19:35:04-08:00]     No such file or directory (os error 2)    module=builder
ERRO[2024-12-16T19:35:04-08:00] fail to run nydus-image [create --log-level warn --prefetch-policy fs --blob tmp/nydus-converter-3320800464/blob.rafs --whiteout-spec none --fs-version 6 --type tar-rafs --blob-inline-meta --features blob-toc --chunk-dict bootstrap=./tmp/nydus-chunk-dict-2321129161 --compressor zstd --chunk-size 0x100000 tmp/nydus-converter-3320800464/blob.targz]  error="exit status 1"
INFO[2024-12-16T19:35:04-08:00] [2024-12-16 19:35:04.316758 -08:00] ERROR [/src/error.rs:23] Error:  module=builder
INFO[2024-12-16T19:35:04-08:00]         "failed to get chunk info for chunk 0/0/60824"  module=builder
INFO[2024-12-16T19:35:04-08:00]         at rafs/src/metadata/direct_v6.rs:1366       module=builder
INFO[2024-12-16T19:35:04-08:00]         note: enable `RUST_BACKTRACE=1` env to display a backtrace  module=builder
INFO[2024-12-16T19:35:04-08:00] Error: failed to build tree from bootstrap    module=builder
INFO[2024-12-16T19:35:04-08:00]                                               module=builder
INFO[2024-12-16T19:35:04-08:00] Caused by:                                    module=builder
INFO[2024-12-16T19:35:04-08:00]     No such file or directory (os error 2)    module=builder
ERRO[2024-12-16T19:35:04-08:00] fail to run nydus-image [create --log-level warn --prefetch-policy fs --blob tmp/nydus-converter-4259805247/blob.rafs --whiteout-spec none --fs-version 6 --type tar-rafs --blob-inline-meta --features blob-toc --chunk-dict bootstrap=./tmp/nydus-chunk-dict-2321129161 --compressor zstd --chunk-size 0x100000 tmp/nydus-converter-4259805247/blob.targz]  error="exit status 1"
INFO[2024-12-16T19:35:04-08:00] [2024-12-16 19:35:04.322875 -08:00] ERROR [/src/error.rs:23] Error:  module=builder
INFO[2024-12-16T19:35:04-08:00]         "failed to get chunk info for chunk 0/0/60824"  module=builder
INFO[2024-12-16T19:35:04-08:00]         at rafs/src/metadata/direct_v6.rs:1366       module=builder
INFO[2024-12-16T19:35:04-08:00]         note: enable `RUST_BACKTRACE=1` env to display a backtrace  module=builder
INFO[2024-12-16T19:35:04-08:00] Error: failed to build tree from bootstrap    module=builder
INFO[2024-12-16T19:35:04-08:00]                                               module=builder
INFO[2024-12-16T19:35:04-08:00] Caused by:                                    module=builder
INFO[2024-12-16T19:35:04-08:00]     No such file or directory (os error 2)    module=builder
ERRO[2024-12-16T19:35:04-08:00] fail to run nydus-image [create --log-level warn --prefetch-policy fs --blob tmp/nydus-converter-2473474945/blob.rafs --whiteout-spec none --fs-version 6 --type tar-rafs --blob-inline-meta --features blob-toc --chunk-dict bootstrap=./tmp/nydus-chunk-dict-2321129161 --compressor zstd --chunk-size 0x100000 tmp/nydus-converter-2473474945/blob.targz]  error="exit status 1"
ERRO[2024-12-16T19:35:04-08:00] fail to run nydus-image [create --log-level warn --prefetch-policy fs --blob tmp/nydus-converter-4022444059/blob.rafs --whiteout-spec none --fs-version 6 --type tar-rafs --blob-inline-meta --features blob-toc --chunk-dict bootstrap=./tmp/nydus-chunk-dict-2321129161 --compressor zstd --chunk-size 0x100000 tmp/nydus-converter-4022444059/blob.targz]  error="exit status 1"
INFO[2024-12-16T19:35:04-08:00] [2024-12-16 19:35:04.321340 -08:00] ERROR [/src/error.rs:23] Error:  module=builder
INFO[2024-12-16T19:35:04-08:00]         "failed to get chunk info for chunk 0/0/60824"  module=builder
INFO[2024-12-16T19:35:04-08:00]         at rafs/src/metadata/direct_v6.rs:1366       module=builder
INFO[2024-12-16T19:35:04-08:00]         note: enable `RUST_BACKTRACE=1` env to display a backtrace  module=builder
INFO[2024-12-16T19:35:04-08:00] Error: failed to build tree from bootstrap    module=builder
INFO[2024-12-16T19:35:04-08:00]                                               module=builder
INFO[2024-12-16T19:35:04-08:00] Caused by:                                    module=builder
INFO[2024-12-16T19:35:04-08:00]     No such file or directory (os error 2)    module=builder
INFO[2024-12-16T19:35:04-08:00] [2024-12-16 19:35:04.328760 -08:00] ERROR [/src/error.rs:23] Error:  module=builder
INFO[2024-12-16T19:35:04-08:00]         "failed to get chunk info for chunk 0/0/60824"  module=builder
INFO[2024-12-16T19:35:04-08:00]         at rafs/src/metadata/direct_v6.rs:1366       module=builder
INFO[2024-12-16T19:35:04-08:00]         note: enable `RUST_BACKTRACE=1` env to display a backtrace  module=builder
INFO[2024-12-16T19:35:04-08:00] Error: failed to build tree from bootstrap    module=builder
INFO[2024-12-16T19:35:04-08:00]                                               module=builder
INFO[2024-12-16T19:35:04-08:00] Caused by:                                    module=builder
INFO[2024-12-16T19:35:04-08:00]     No such file or directory (os error 2)    module=builder
INFO[2024-12-16T19:35:04-08:00] [2024-12-16 19:35:04.341094 -08:00] ERROR [/src/error.rs:23] Error:  module=builder
INFO[2024-12-16T19:35:04-08:00]         "failed to get chunk info for chunk 0/0/60824"  module=builder
INFO[2024-12-16T19:35:04-08:00]         at rafs/src/metadata/direct_v6.rs:1366       module=builder
INFO[2024-12-16T19:35:04-08:00]         note: enable `RUST_BACKTRACE=1` env to display a backtrace  module=builder
INFO[2024-12-16T19:35:04-08:00] Error: failed to build tree from bootstrap    module=builder
INFO[2024-12-16T19:35:04-08:00]                                               module=builder
INFO[2024-12-16T19:35:04-08:00] Caused by:                                    module=builder
INFO[2024-12-16T19:35:04-08:00]     No such file or directory (os error 2)    module=builder
ERRO[2024-12-16T19:35:04-08:00] fail to run nydus-image [create --log-level warn --prefetch-policy fs --blob tmp/nydus-converter-3189638095/blob.rafs --whiteout-spec none --fs-version 6 --type tar-rafs --blob-inline-meta --features blob-toc --chunk-dict bootstrap=./tmp/nydus-chunk-dict-2321129161 --compressor zstd --chunk-size 0x100000 tmp/nydus-converter-3189638095/blob.targz]  error="exit status 1"
ERRO[2024-12-16T19:35:04-08:00] fail to run nydus-image [create --log-level warn --prefetch-policy fs --blob tmp/nydus-converter-4060495969/blob.rafs --whiteout-spec none --fs-version 6 --type tar-rafs --blob-inline-meta --features blob-toc --chunk-dict bootstrap=./tmp/nydus-chunk-dict-2321129161 --compressor zstd --chunk-size 0x100000 tmp/nydus-converter-4060495969/blob.targz]  error="exit status 1"
FATA[2024-12-16T19:35:04-08:00] convert image: copy nydus blob to content store: failed to copy: io: read/write on closed pipe 

@cslinwang PTAL. Thks.