CI failure: nerdctl - TestRemove/Remove_image_with_running_container_-_with_-f : images appears to be racy / fail in certain circumstances

[apostasie]

Description

Seen on the CI:

=== Failed
=== FAIL: cmd/nerdctl/image TestRemove/Remove_image_with_running_container_-_with_-f (0.93s)
    image_remove_test.go:306: ======================== Pre-test cleanup ========================
    command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 rm -f testremove-remove-image-with-running-container-with-f-e9c3b26f
    command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 rmi -f 
    image_remove_test.go:306: ======================== Test setup ========================
    command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 run --pull always -d --name testremove-remove-image-with-running-container-with-f-e9c3b26f ghcr.io/stargz-containers/alpine:3.13-org sleep 3600
    command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 image inspect ghcr.io/stargz-containers/alpine:3.13-org
    image_remove_test.go:306: ======================== Test Run ========================
    command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 rmi -f ghcr.io/stargz-containers/alpine:3.13-org
    command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 images
    command.go:112: assertion failed: expression is false: strings.Contains(stdout, compare): Output does not contain: "<none>"
        Command:  /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 images
        ExitCode: 0
        Stdout:   REPOSITORY    TAG    IMAGE ID    CREATED    PLATFORM    SIZE    BLOB SIZE
        
        Stderr:   time="2024-12-02T22:04:46Z" level=warning msg="content digest sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a: not found"
        
        Env:
        HOSTNAME=cd6946756de5
        SYSTEMD_EXEC_PID=47
        container=docker
        HOME=/root
        LANG=C.UTF-8
        INVOCATION_ID=c7ba162bd0dd405697721433a973325e
        TERM=xterm
        SHLVL=3
        CGO_ENABLED=0
        _=/usr/local/bin/gotestsum
        PATH=/usr/local/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        ***
        DOCKER_CONFIG=/tmp/TestRemoveRemove_image_with_running_container_-_with_-f268936920/001
        NERDCTL_TOML=/tmp/TestRemoveRemove_image_with_running_container_-_with_-f268936920/001/nerdctl.toml
    image_remove_test.go:306: ======================== Processing subtests ========================
    case.go:164: ======================== Post-test cleanup ========================
    command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 rm -f testremove-remove-image-with-running-container-with-f-e9c3b26f
    command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 rmi -f ec14c799

So, nerdctl images does output a content digest sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a: not found, which may suggest that it first retrieved the full image list, then tried to do some lookup on one of the images, and failed.

It is unclear if this prevents other images from being listed, or if this is a different problem entirely.

Must be investigated.

Steps to reproduce the issue

Describe the results you received and expected

na

What version of nerdctl are you using?

main

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

[apostasie]

This appears more and more.
Could it be a recent regression?

https://github.com/containerd/nerdctl/actions/runs/12131315521/job/33823232239

[apostasie]

Confirming this is hitting a good number of builds:

https://github.com/containerd/nerdctl/actions/runs/12322888864/job/34397435860?pr=3739