[Bug]: Ryuk container created, then it is removed before it starts

[RiverPhillips]

Testcontainers version

0.31.0

Using the latest Testcontainers version?

Yes

Host OS

Mac OS

Host arch

ARM

Go version

1.22

Docker version

Client: Docker Engine - Community
 Version:           26.1.4
 API version:       1.45
 Go version:        go1.22.4
 Git commit:        5650f9b102
 Built:             Wed Jun  5 10:47:13 2024
 OS/Arch:           darwin/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.1
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       ac2de55
  Built:            Tue Apr 30 11:48:47 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client: Docker Engine - Community
 Version:    26.1.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.0
    Path:     /Users/river/.docker/cli-plugins/docker-buildx
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 26.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-31-generic
 Operating System: Ubuntu 24.04 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 2
 Total Memory: 1.91GiB
 Name: colima
 ID: 5a6217f8-7cca-498b-91f6-d4070390a0e4
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

What happened?

Any attempt to run a test using test containers appears to create the Ryuk Container, return an ID. Then it is removed while waiting for the container to start.

Relevant log output

024/06/18 16:26:45 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/06/18 16:26:45 ✅ Container created: 4879564c4f90
2024/06/18 16:26:45 🐳 Starting container: 4879564c4f90
2024/06/18 16:26:45 ✅ Container started: 4879564c4f90
2024/06/18 16:26:45 ⏳ Waiting for container id 4879564c4f90 image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/06/18 16:26:45 failed accessing container logs: Error response from daemon: No such container: 4879564c4f9093fffa6b43a4a75d48e63a71672bc391fd3ce69951bb13152643

Additional information

I attempted to run test tests with main and observed the same failures so I don't believe this fixes it. #2477

[ash2k]

Experiencing the same issue. macOS 14.5 Sonoma, Rancher Desktop 1.14.2. Testcontainers 36573d0 (a few commits after 0.31.0).

Client: Docker Engine - Community
 Version:    26.1.4
 Context:    rancher-desktop
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.1
    Path:     /Users/mike/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.1
    Path:     /Users/mike/.docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 25.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 4e1fe7492b9df85914c389d1f15a3ceedbb280ac
 runc version: 51d5e94601ceffbbd85688df1c928ecccbfa4685
 init version:
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.30-0-virt
 Operating System: Alpine Linux v3.19
 OSType: linux
 Architecture: aarch64
 CPUs: 9
 Total Memory: 15.6GiB
 Name: lima-rancher-desktop
 ID: d04c0365-7614-447f-960d-353ba1523b4f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

It worked yesterday, not sure what I did to break it.

[ash2k]

Set TESTCONTAINERS_RYUK_DISABLED to true and things work again.

[RiverPhillips]

Disabling Ryuk isn't really a solution, we'd like to continue to use it.

[mdelapenya]

Hi 👋 are you by the way using the compose module? If so please take a look at #2563 (comment)

If not, I'd like to see the ryuk logs if possible, so we can determine why it's being removed. Is it a timeout issue? You can check the ryuk configuration here: https://golang.testcontainers.org/features/configuration/#customizing-ryuk-the-resource-reaper

[RiverPhillips]

Hi @mdelapenya we're not using compose.

Ryuk doesn't appear to log anything in this case. Upon further inspection when I run go test from the command line it does work, but running it from VS code's test runner, or Goland's test runner it fails.

[mdelapenya]

Mmm that's even weirder. Can you verify that the IDE sets the proper go settings for testing, such as "go.testTimeout" (I configured it to 600s)? I wonder why the ryuk container dies in VSCode but not in your terminal.

Can you provide a repro repository?

[RiverPhillips]

Mmm that's even weirder. Can you verify that the IDE sets the proper go settings for testing, such as "go.testTimeout" (I configured it to 600s)? I wonder why the ryuk container dies in VSCode but not in your terminal.

The test timeouts look good, it fails almost instantly, it's not taking a long time.

Can you provide a repro repository?

I have this issue for any project using testcontainers

[stevenh]

This should be fixed by testcontainers/moby-ryuk#121 but we need a release and then testcontainers-go updated to use the new release.

Try cloning the moby-ryuk repo and running the following in it to replace the image that testcontainers-go uses to see if it does fix:

docker build -f linux/Dockerfile -t testcontainers/ryuk:0.7.0 .

[RiverPhillips]

Thanks for looking into this, I've updated to use main and I sometimes get the same error, and sometimes another error, I think it's still related. @mdelapenya are you able to re-open this please, or should I file another issue?

        	Error:      	Received unexpected error:
        	            	unexpected container status "removing": could not start container: creating reaper failed: failed to create container

[stevenh]

Thanks for testing.

I have a bug bash branch that I'm working on improving the stability of tests which includes a reworked reaper spawner and while this may not fix the issue, it also includes improvements to the errors so might help point to where the underlying issue is in your case.

If you could test on the branch from #2664 and let me know what you get that would be appreciated.

If you could also post a test which reproduces the issue that would help too.

[joseroblesv]

I fixed this issue modifying the ~/.zshrc file with this configuration

export DOCKER_HOST="unix:///Users/${HOME}/.colima/default/docker.sock"
export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE="/var/run/docker.sock"