rerun after image removed cause error

[VincentLeeMax]

Additional Information

Version of nydus being used (nydusd --version)

Version: v2.3.0
Git Commit: a4683ba
Build Time: 2024-10-11T02:51:53.888907475Z
Profile: release
Rustc: rustc 1.72.1 (d5c2e9c34 2023-09-13)

Version of nydus-snapshotter being used (containerd-nydus-grpc --version)

Version: v0.14.0
Revision: 8e72f94dcad2d4898b6437fc84d6c9c36f78ea50
Go version: go1.21.12
Build time: 2024-08-02T06:25:54

Kernel information (uname -r)

5.4.0-48-generic

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

containerd-nydus-grpc command line used, if applicable (ps aux | grep containerd-nydus-grpc)

root     53507  0.3  0.1 1272200 40048 ?       Ssl  16:34   0:01 /usr/local/bin/containerd-nydus-grpc --config /etc/nydus/config.toml
root     65231  0.0  0.0   6436  2416 pts/0    S+   16:43   0:00 grep --color=auto containerd-nydus-grpc

client command line used, if applicable (such as: nerdctl, docker, kubectl, ctr)

nerdctl --snapshotter nydus run --network=host -it --rm harbor.info/machine-learn/mpioperator_tensorflow-benchmarks:0.3.0-nydus bash

nerdctl image rm harbor.info/machine-learn/mpioperator_tensorflow-benchmarks:0.3.0-nydus

mount -l | grep rafs

Screenshots (if applicable)

Details about issue

After I remove the image and re-run the container, there are some problems with the new mount point. I could not get the content inside the file .

I try v2.2.5, it works well in this case.

[imeoer]

@VincentLeeMax Hi, any error logs can be found in nydus snapshotter (containerd-nydus-grpc) process?

[VincentLeeMax]

@VincentLeeMax Hi, any error logs can be found in nydus snapshotter (containerd-nydus-grpc) process?

No error logs found in nydus-snapshotter.log. The old mount point is /data/containerd/io.containerd.snapshotter.v1.nydus/snapshots/24/mnt, and the new mount point is /data/containerd/io.containerd.snapshotter.v1.nydus/snapshots/50/mnt.

Everytime this happen, I need to remove the whole $containerd_home to resume. It's horrible.

[Desiki-high]

@VincentLeeMax Hi, I test in nyudsd v2.3.0 and containerd-nydus-grpc v0.14.0, but i can't reproduce your issue. Can you provide more infomations, such as your nerdctl containerd version and your config for nydusd.

It's even better if you can provide the image you used or test with ghcr.io/dragonflyoss/nydus/busybox:latest.

[imeoer]

@Desiki-high Did you check some files in the mount point?

[Desiki-high]

@Desiki-high Did you check some files in the mount point?

I checked the dir bin and the file in etc.

[imeoer]

maybe it's related to the cleanup of chunk_map file in cache directory, please try: nydus-snapshotter v0.15.0, the fixup PR：containerd/nydus-snapshotter#619 cc @VincentLeeMax @Desiki-high