[Bug]: Unable to connect to container #1782
Comments
|
Thanks for open this issue, I'm taking a look at the log and I find that the Thanks! |
|
@mdelapenya , sure: seems like it is related to other issues regarding reaper timeout it seems to work |
|
@itziklavon please remember that since v0.25.0 it's possible to configure those values at the properties level. Please see #1668 |
|
@itziklavon is the issue resolved? Please let me know so we can close this issue as fixed Thanks! |
|
@mdelapenya it is only resolved when updating the conf, without it no |
|
Hi tried to upgrade to 0.26.0 something broken after version 0.23.0 |
|
Hello @mdelapenya I tried using Also i can confirm - We get error for 2 different containers. imageName := "redis:5.0"
port := "6379/tcp"
req := testcontainers.ContainerRequest{
Name: "redis",
Image: imageName,
ExposedPorts: []string{port},
SkipReaper: true,
AutoRemove: true,
}
redis, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
ContainerRequest: req,
Started: true,
}) req := testcontainers.ContainerRequest{
Name: "registry",
Image: registryImage,
ExposedPorts: []string{registryPort},
Env: map[string]string{
"REGISTRY_HTTP_ADDR": "0.0.0.0:5443",
"REGISTRY_HTTP_TLS_CERTIFICATE": "/certs/cert.pem",
"REGISTRY_HTTP_TLS_KEY": "/certs/key.pem",
"REGISTRY_AUTH": "token",
"REGISTRY_AUTH_TOKEN_REALM": fmt.Sprintf("%s/auth", authURL),
"REGISTRY_AUTH_TOKEN_SERVICE": "registry.docker.io",
"REGISTRY_AUTH_TOKEN_ISSUER": "Trivy auth server",
"REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE": "/certs/cert.pem",
"REGISTRY_AUTH_TOKEN_AUTOREDIRECT": "false",
},
Mounts: testcontainers.Mounts(
testcontainers.BindMount(filepath.Join(baseDir, "data", "certs"), "/certs"),
),
SkipReaper: true,
AutoRemove: true,
WaitingFor: wait.ForLog("listening on [::]:5443"),
}
registryC, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
ContainerRequest: req,
Started: true,
}) |
|
some new information about
imageName := "redis:5.0"
port := "6379/tcp"
req := testcontainers.ContainerRequest{
Name: "redis",
Image: imageName,
ExposedPorts: []string{port},
AutoRemove: true,
SkipReaper: true,
}
redis, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
ContainerRequest: req,
Started: true,
}) Server Version: 24.0.2
API Version: 1.43
Operating System: Docker Desktop
Total Memory: 7851 MB
Resolved Docker Host: unix:///var/run/docker.sock
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: c64b2db77b69988577dbb2df105a962669cb5fd0632d0410d702826ec9f8e8ac
Test ProcessID: e5c2b280-f2b1-4896-9b0b-02ab172596a6
2023/11/02 12:13:42 🐳 Creating container for image docker.io/testcontainers/ryuk:0.5.1
2023/11/02 12:13:42 ✅ Container created: faf4192b3372
2023/11/02 12:13:42 🐳 Starting container: faf4192b3372
2023/11/02 12:13:42 ✅ Container started: faf4192b3372
2023/11/02 12:13:42 🚧 Waiting for container id faf4192b3372 image: docker.io/testcontainers/ryuk:0.5.1. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2023/11/02 12:13:43 🐳 Creating container for image redis:5.0
...
**********************************************************************************************
Ryuk has been disabled for the current execution. This can cause unexpected behavior in your environment.
More on this: https://golang.testcontainers.org/features/garbage_collector/
**********************************************************************************************
2023/11/02 12:15:22 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 24.0.2
API Version: 1.43
Operating System: Docker Desktop
Total Memory: 7851 MB
Resolved Docker Host: unix:///var/run/docker.sock
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: 3228b887303b32989bb5a8ba658bed959fa3096cb3602b07c11b2e17d26aa473
Test ProcessID: 71e73b64-128f-4658-a6cf-d63fe95855c3
2023/11/02 12:15:22 🐳 Creating container for image redis:5.0When i used |
|
@DmitriyLewen thanks for your report 🙏 Just to confirm with you, SkipReaper bool // Deprecated: The reaper is globally controlled by the .testcontainers.properties file or the TESTCONTAINERS_RYUK_DISABLED environment variableand marked as a breaking change in #941. Please see release notes for
I'm going to look carefully to the changes in between to detect where the root cause is, but let's state the error and the expected behaviour. Let me sum up what we saw in the issue:
|
Oh... i missed that. Thanks for information!
setting timeouts did not help (but when I checked it - we used
I said about error |
|
@DmitriyLewen it's weird that only a few test methods failed in that GH run: if the reaper failed to be created, then it should fail for all the tests. Let me take a look at the test code for those errors 👀 |
|
I thought it had something to do with the first test I ran: But i didn't investigation this better :( |
|
I noticed all the three failing tests are disabling Ryuk: https://github.com/search?q=repo%3Aaquasecurity%2Ftrivy%20TESTCONTAINERS_RYUK_DISABLED&type=code Just to let you know, Ryuk starts once for an entire "test session", and it's reused during that session. To understand what a test session means for us, please refer to #1513:
Therefore, the containers in those tests will be managed by Ryuk, which has been created by the first test to be run. I'd like to know if possible why certain containers should not be killed by Ryuk to better understand the issue here. |
|
I don't know why these Ryuk settings were created for tests earlier (before my fix). But for now we don't use Ryuk because we run |
Got it, thanks! In any case, Ryuk will be there even you don't want it, because it's for the entire test session. Just want to match expectations about whether Ryuk is there or not handling the lifecycle of the containers. In this case, it will. Probably it's not contributing to the root cause. We do have this open PR about to be merged: #1904, which could be related if you are running multiple tests in multiple packages in parallel. |
|
@DmitriyLewen given #1904 was merged, could you try with the main branch instead of the release one? I'm preparing a new release, and would like to include this big as fixed if possible. Thanks! |
|
Hello @mdelapenya |
|
I have had a similar issue, though not exactly the same, figured I could just comment here with my test findings. As far as I can see #1671 is related to this issue and my issue. I can open a separate issue if it helps. When running on Github Actions I some times get this error with version Fetching the latest version from |
|
hello @mdelapenya I can reproduce error when using But looks like with |
|
I have the same issue running multiple tests, and creating multiple instances of the containers, locally works but fails on GitHub Actions, setting |
|
@DmitriyLewen @acevedomiguel did you try with current main branch, as pointed out by @mikalsande in #1782 (comment)?
|
I tried with version from main branch. But now i rechecked this and understand that i used old ➜ trivy git:(b0141cfba) ✗ cat go.mod | grep github.com/testcontainers/testcontainers-go
github.com/testcontainers/testcontainers-go v0.26.1-0.20231206105358-e9839072cc9b
github.com/testcontainers/testcontainers-go/modules/localstack v0.21.0Error: === RUN TestRegistry
2023/12/11 08:59:31 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 24.0.2
API Version: 1.43
Operating System: Docker Desktop
Total Memory: 7851 MB
Resolved Docker Host: unix:///var/run/docker.sock
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: 1ca48377acc30e1f43ac3d8898bcbfe9c292f557e41aeeb040ed0d759fd035ea
Test ProcessID: 297ecef5-8cca-4448-9f0b-7a4a03afda97
2023/12/11 08:59:31 🐳 Creating container for image docker.io/testcontainers/ryuk:0.5.1
2023/12/11 08:59:31 ✅ Container created: e39c0294e6c3
2023/12/11 08:59:31 🐳 Starting container: e39c0294e6c3
2023/12/11 08:59:32 ✅ Container started: e39c0294e6c3
2023/12/11 08:59:32 🚧 Waiting for container id e39c0294e6c3 image: docker.io/testcontainers/ryuk:0.5.1. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2023/12/11 08:59:32 🐳 Creating container for image cesanta/docker_auth:1
2023/12/11 08:59:32 ✅ Container created: 603b128e9f49
2023/12/11 08:59:32 🐳 Starting container: 603b128e9f49
2023/12/11 08:59:32 ✅ Container started: 603b128e9f49
2023/12/11 08:59:32 🐳 Creating container for image registry:2.7.0
2023/12/11 08:59:32 ✅ Container created: 1d3b320d021f
2023/12/11 08:59:32 🐳 Starting container: 1d3b320d021f
2023/12/11 08:59:32 ✅ Container started: 1d3b320d021f
2023/12/11 08:59:32 🚧 Waiting for container id 1d3b320d021f image: registry:2.7.0. Waiting for: &{timeout:<nil> Log:listening on [::]:5443 IsRegexp:false Occurrence:1 PollInterval:100ms}
2023/12/11 09:00:32 failed accessing container logs: Error response from daemon: No such container: 1d3b320d021f86ef124f88fa96bbd5c8986e18448616ed8a8efe051a38811bcc
registry_test.go:132:
Error Trace: /Users/work/work/aqua/trivy/integration/registry_test.go:132
Error: Received unexpected error:
context deadline exceeded: failed to start container
Test: TestRegistry
2023/12/11 09:00:32 🐳 Terminating container: 603b128e9f49 |
|
For me github actions broke after updating docker in the go.mod from 24.0.6+incompatible to 24.0.7+incompatible. |
|
@destel are you using current I'm planning to cut a release probably next week, so the Ryuk fix will be available for everybody |
|
@mdelapenya I am using v0.26.0 |
|
Ok, could you please do a quick test with |
|
@mdelapenya Unfortunately it does not work on github actions. Still works locally well. |
|
Are the GH logs available, or are they private? I wonder if we could troubleshoot them in any manner |
|
Somehow I confused myself and you. I did two builds one from the master and one from the #1094 branch. |
|
@mdelapenya I seem to have the same issue with v0.27.0: Anything I can try? |
Could you do a quick try with main? Not sure if #2084 could be related to this |
|
@mdelapenya |
Sure! Upgraded to main: Same results: |
|
Ummm, this is weird:
I wonder if there is anything trying to remove the reaper |
It's a pretty new OS install with Docker Desktop for Windows. And I had Testcontainers Desktop running, but stopped it (had no impact on the result). Now I deleted the existing ryuk images from the local docker images and get this output: |
Is there anything I could try? The problem persists. |
|
Getting the same problem on newest version. I'm bypassing it by disabling Ryuk with the env var:
|
|
hi, any updates? This is still happens for me |
|
I appear to have the same issue with 0.32.0 : parallel tests in Go with t.Parallel(), each creating a separate container, work normally on a local mac, but some pseudo-randomly fail on CircleCI, with the same message: (indenting added to clarify) No problem even on CircleCI when removing the |
|
@fgmarand The error changed for me from 0.31.0 to 0.32.0, now looks more like yours. With 0.31.0 I was able to use
|
Testcontainers version
0.25.0
Using the latest Testcontainers version?
Yes
Host OS
Linux, MacOS
Host arch
x86, Arm
Go version
1.21
Docker version
Docker info
What happened?
After upgrading to version 0.25.0 this error occurs:
this happens when i have multiple tests starting multiple containers, when i run single test(2-3 containers), no issues
when reverting back to version 0.23.0, all seems to be working well as expected
Relevant log output
[2023-10-18T20:52:04.113Z] ok withdrawal-go/tests/component_test_expanded_player_info 28.668s coverage: [no statements] [2023-10-18T20:52:04.113Z] ok withdrawal-go/tests/component_test_high_risk 27.613s coverage: [no statements] [2023-10-18T20:52:10.894Z] 2023/10/18 20:52:02 github.com/testcontainers/testcontainers-go - Connected to docker: [2023-10-18T20:52:10.894Z] Server Version: 20.10.12 [2023-10-18T20:52:10.894Z] API Version: 1.41 [2023-10-18T20:52:10.894Z] Operating System: Ubuntu 20.04.4 LTS [2023-10-18T20:52:10.894Z] Total Memory: 7700 MB [2023-10-18T20:52:10.894Z] Resolved Docker Host: unix:///var/run/docker.sock [2023-10-18T20:52:10.894Z] Resolved Docker Socket Path: /var/run/docker.sock [2023-10-18T20:52:10.894Z] Test SessionID: 27f2a3d02835da8cca598bb5a0551abf0074cff82ea48d2670bc4c0829619808 [2023-10-18T20:52:10.894Z] Test ProcessID: 3e61313b-c7af-465e-b3e9-d6841747cf04 [2023-10-18T20:52:10.894Z] {"level":"error","ts":"2023-10-18 20:52:09","caller":"component/mysql_init_ms.go:33","F":"bitbucket.org/gamescale-rnd/go_tests_utils/component.StartMSSql","msg":"Error response from daemon: No such container: d11e9589adc269f7720f307f7506385110c0e9ab5580a577c3c1f2568fe23333: creating reaper failed: failed to create container"} [2023-10-18T20:52:10.894Z] panic: runtime error: invalid memory address or nil pointer dereference [recovered] [2023-10-18T20:52:10.894Z] panic: runtime error: invalid memory address or nil pointer dereference [2023-10-18T20:52:10.894Z] [signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0x1000581] [2023-10-18T20:52:10.894Z] [2023-10-18T20:52:10.894Z] goroutine 1 [running]: [2023-10-18T20:52:10.894Z] withdrawal-go/tests/component_test_irregular_play.TestMain.func1() [2023-10-18T20:52:10.894Z] /var/lib/jenkins/workspace/nightly_go_withdrawal/tests/component_test_irregular_play/irregular_play_component_test.go:112 +0x57 [2023-10-18T20:52:10.894Z] panic({0x11e7780?, 0x1f66e30?}) [2023-10-18T20:52:10.894Z] /usr/local/go/src/runtime/panic.go:914 +0x21f [2023-10-18T20:52:10.894Z] bitbucket.org/gamescale-rnd/go_tests_utils/component.StartMSSql({0x139d80d, 0xb}, {0x139d80d, 0xb}, {0x164e4c8, 0xc00006e6d8}, {0x164e4c8, 0xc00006e6e0}) [2023-10-18T20:52:10.894Z] /go/pkg/mod/bitbucket.org/gamescale-rnd/[email protected]/component/mysql_init_ms.go:35 +0x321 [2023-10-18T20:52:10.894Z] withdrawal-go/tests/component_test_irregular_play.TestMain(0x1394c0a?) [2023-10-18T20:52:10.894Z] /var/lib/jenkins/workspace/nightly_go_withdrawal/tests/component_test_irregular_play/irregular_play_component_test.go:118 +0x22e [2023-10-18T20:52:10.894Z] main.main() [2023-10-18T20:52:10.894Z] _testmain.go:183 +0x1e6 [2023-10-18T20:52:10.894Z] FAIL withdrawal-go/tests/component_test_irregular_play 7.585s [2023-10-18T20:52:10.894Z] 2023/10/18 20:52:02 github.com/testcontainers/testcontainers-go - Connected to docker: [2023-10-18T20:52:10.894Z] Server Version: 20.10.12 [2023-10-18T20:52:10.894Z] API Version: 1.41 [2023-10-18T20:52:10.894Z] Operating System: Ubuntu 20.04.4 LTS [2023-10-18T20:52:10.894Z] Total Memory: 7700 MB [2023-10-18T20:52:10.894Z] Resolved Docker Host: unix:///var/run/docker.sock [2023-10-18T20:52:10.894Z] Resolved Docker Socket Path: /var/run/docker.sock [2023-10-18T20:52:10.894Z] Test SessionID: 27f2a3d02835da8cca598bb5a0551abf0074cff82ea48d2670bc4c0829619808 [2023-10-18T20:52:10.895Z] Test ProcessID: 6b064694-4144-441c-8cdc-eae0a06507a8 [2023-10-18T20:52:10.895Z] {"level":"error","ts":"2023-10-18 20:52:09","caller":"component/mysql_init_ms.go:33","F":"bitbucket.org/gamescale-rnd/go_tests_utils/component.StartMSSql","msg":"Error response from daemon: No such container: d11e9589adc269f7720f307f7506385110c0e9ab5580a577c3c1f2568fe23333: creating reaper failed: failed to create container"} [2023-10-18T20:52:10.895Z] panic: runtime error: invalid memory address or nil pointer dereference [recovered] [2023-10-18T20:52:10.895Z] panic: runtime error: invalid memory address or nil pointer dereference [2023-10-18T20:52:10.895Z] [signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0x1050b01]The text was updated successfully, but these errors were encountered: