[Regression]: no basic auth credentials when attempting to pull ryuk image

[greg0ire]

Testcontainers version

0.34.0

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host arch

arm

Go version

1.23

Docker version

Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.44 (downgraded from 1.47)
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:41:19 2024
 OS/Arch:           linux/arm64
 Context:           default
Server: Docker Engine - Community
 Engine:
  Version:          25.0.0
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       615dfdf
  Built:            Thu Jan 18 17:10:09 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
Server:
 Containers: 3
  Running: 2
  Paused: 0
  Stopped: 1
 Images: 8
 Server Version: 25.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 4.19.0-25-arm64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: aarch64
 CPUs: 1
 Total Memory: 7.684GiB
 Name: runner-k8kvlfp3-gitlab-ci-builder-1732008478-36648aa4
 ID: c29b1ee8-408f-43dc-bce9-4398da2b90f6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Labels:
  provider=amazonec2
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
WARNING: No swap limit support

What happened?

The test fails with an authentication error. It happens in a pipeline for a commit that only upgrades the testcontainers-go version

Relevant log output

postgresql_query_repository_test.go:145: generic container: create container: reaper: new reaper: run container: Error response from daemon: Head "https://redacted.dkr.ecr.eu-west-3.amazonaws.com/v2/public/testcontainers/ryuk/manifests/0.10.2": no basic auth credentials

Additional information

This runs before and prints Login succeeded:

aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${DOCKER_REGISTRY}

[greg0ire]

@stevenh hi! Do you think this could be related to #2728 ?

[stevenh]

It shouldn't be, I'm curious why it's trying to use a custom repo for the ryuk image.

Could you share a reproduction test?

[greg0ire]

I'm curious why it's trying to use a custom repo for the ryuk image

Oh, sorry for not mentioning it, but that's expected, because of this CI variable:

TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX: redacted.dkr.ecr.eu-west-3.amazonaws.com/public

For the reproduction test… I will try to have a look at that at some point, but since this is not an open source project, I suspect I will have a hard time with that.

When running my test suite with TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX=redacted.dkr.ecr.eu-west-3.amazonaws.com/public locally, and ensuring I do not have the image beforehand, I do not reproduce the issue. I will try altering my CI job to pull the image with a docker pull before, to see if that works (if it doesn't, then I should close this issue).

EDIT: That worked. Let us keep this open I guess. That being said, I do not reproduce it locally, even when using the private repository. From all this I think it's fair to say:

• There really is a bug in the library.
• Using the custom repository is not sufficient to reproduce the issue, some other condition appears to be necessary.

[greg0ire]

Finished the bisection, it points to 6a947dc

That commit touches docker_auth.go, which seems a lot more relevant to the issue.

Handle file not exist error in getDockerAuthConfigs, treating it as if
no authentication was provided.

That sounds like it could be it.

[stevenh]

Thanks for doing that, a super basic test would be great. I suspect all it will need is a custom registry, no product specifics.

[stevenh]

Could you also see if you get any messages in the log file? Maybe Failed to get image auth for...

[greg0ire]

Sorry, what log file? The log I get is already in my original message.

[stevenh]

If you can run it under the debugger, setting a breakpoint at the top of attemptToPullImage.

[stevenh]

Sorry, what log file? The log I get is already in my original message.

The line in the original report looks like a require error from a test. If you run the test in verbose mode it will enable logging which I suspect might add some additional context.

[stevenh]

I'm wondering if the registry you're using requires auth, even for ryuk and for some reason its not getting it.

[greg0ire]

Yes, the registry I'm using requires auth.

[greg0ire]

The code has this comment:

// 1. the DOCKER_AUTH_CONFIG environment variable, unmarshalling it into a dockercfg.Config
// 2. the DOCKER_CONFIG environment variable, as the path to the config file
// 3. else it will load the default config file, which is ~/.docker/config.json

Your commit starts with

Handle file not exist error in getDockerAuthConfigs, treating it as if
no authentication was provided.

Is the "file" you are talking about ~/.docker/config.json? And if yes, doesn't that clash with the comment?

[stevenh]

If the call here fails https://github.com/testcontainers/testcontainers-go/blob/main/docker.go#L1390 it will log.

[stevenh]

The code has this comment:

// 1. the DOCKER_AUTH_CONFIG environment variable, unmarshalling it into a dockercfg.Config
// 2. the DOCKER_CONFIG environment variable, as the path to the config file
// 3. else it will load the default config file, which is ~/.docker/config.json

Your commit starts with

Handle file not exist error in getDockerAuthConfigs, treating it as if
no authentication was provided.

Is the "file" you are talking about ~/.docker/config.json? And if yes, doesn't that clash with the comment?

The intent is it will just allow us to try the action even if the config file doesn't exist, just no auth will be provided, instead of hard failing.

[greg0ire]

Looks like I'm already using verbose mode: gotestsum --format standard-verbose --jsonfile report.json --junitfile report.xml -- -covermode=count -coverprofile=cover.out ${GO_TEST_PACKAGES}

[greg0ire]

If the call here fails https://github.com/testcontainers/testcontainers-go/blob/main/docker.go#L1390 it will log.

If the call here fails, we won't get to the point where we attempt do perform a HEAD request though, right?

[stevenh]

yer it will, if you look it falls through no matter what.

[greg0ire]

I'm trying to troubleshoot this further, and I added

- env
- cat ~/.docker/config.json

after the line where I perform a docker login.

I see that I have DOCKER_AUTH_CONFIG=$DOCKER_AUTH_CONFIG. I don't know if it's Gitlab's way of displaying this by performing a search and replace, or if something is wrong and the value of this env var is literally $DOCKER_AUTH_CONFIG

I see that I also have what appears to be a valid .docker/config.json :

{
	"auths": {
		"redacted.dkr.ecr.eu-west-3.amazonaws.com": {
			"auth": "redacted"
		}
	}
}

I do not reproduce the issue locally with DOCKER_AUTH_LOGIN=bogus TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX=redacted.dkr.ecr.eu-west-3.amazonaws.com/public just test 😞

[greg0ire]

I am tired, it's the wrong variable name. With DOCKER_AUTH_LOGIN=bogus TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX=redacted.dkr.ecr.eu-west-3.amazonaws.com/public just test I finally reproduce the issue locally.

Also, looking at other repositories in my company, it looks like people using testcontainers often unset this variable, without explaining why 🤦

[greg0ire]

Unsetting the variable worked. Sorry for wasting your time.