-
-
Notifications
You must be signed in to change notification settings - Fork 512
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: Sporadically fails to create reaper in GitHub Actions #2172
Comments
Hi @abemedia ! I hope you are doing well! can you share link to the job? I check actions in this repo - repear works fine - for example - https://github.com/testcontainers/testcontainers-go/actions/runs/7730045562/job/21075148438#step:10:41 |
Did you manage to solve this issue? I'm facing an issue with reaper failing with the following error.. it only happens on GHA:
|
I am getting the same error apparently. |
@kostyay I was able to bypass this error by cleaning up containers on the tests and then using |
I am experiencing this exact same issue in gitlab runners when running a few different jobs that are all using testcontainers. |
I am getting a similar error in Github Actions:
|
I've seen this quite frequently in Jenkins when running a large number of tests. Disabling the reaper with If you're in an ephemeral environment enabling |
Adding a data point here: we never hit this until upgrading testcontainers-go from 0.19 to 0.28.
or
|
Still having this issue. I notice this when multiple containers are spawned. But when done individually, no errors of reaper or port issues occurred. Having the
|
To narrow @tuxxi 's anecdote. I didn't see this until we upgraded from 0.26 to 0.27 |
Hi folks, I think in the most recent versions we have introduced some fixes to the synchronisation of the reaper over a single test session. Issues/PRs of interest could be:
I'd appreciate if you can check them out so we can close this one. Thanks in advance for your support! |
@mdelapenya Just checked on Windows 11 with WSL2 docker. |
TL;DR: You can terminate the containers directly and no longer need the reaper. Hey, so I was having this issue. I'm using an old version of the project but I think the solution should work for newer versions. On my func TestMain(m *testing.M) {
container, testDB, err := tools.CreateContainer("test-db")
if err != nil {
log.Fatal(err)
}
TestSqlDB = testDB
result := m.Run()
// Deferred functions aren't run when os.Exit is called..
testDB.Close()
container.Terminate(context.Background()) // nolint
os.Exit(result)
} My issue was originally that the |
For the record, SkipReaper is deprecated in newer versions and will eventually disappear from the Request struct (see v1 branch: https://github.com/testcontainers/testcontainers-go/blob/v1/request.go#L30) If you want to disable Ryuk, please use the properties file here: https://golang.testcontainers.org/features/configuration/#customizing-ryuk-the-resource-reaper |
I'm also experiencing this in a regular dev environment (it's not a GitHub Action). It would be great if it just worked with Ryuk without needing to turn it off and manually manage the container state. I don't think this is a good solution. Ryuk should just work out of the box. This is especially important because it's challenging to manage container state across many tests run in parallel if you want to reuse containers (instead of creating a new container for each test). I'd much rather have Ryuk handle this itself, so I can run my tests in parallel without having to manage logic for terminating the containers once all tests have finished. Hopefully this bug gets fixed soon because it's been a PITA. |
Due to the need to update testcontainers lib also we needed to disable testcontainers reaper at Github actions as well as it seems related to testcontainers/testcontainers-go#2172
This issue seems related to testcontainers/testcontainers-go#2172
This issue seems related to testcontainers/testcontainers-go#2172
Possibly related to this fix testcontainers/moby-ryuk#121 which needs a release and then testcontainers-go updating to use the new image version Try cloning the moby-ryuk repo and running the following in it to replace the image that testcontainers-go uses to see if it does fix: docker build -f linux/Dockerfile -t testcontainers/ryuk:0.7.0 . |
@tbrown1979 please remember that the reaper also removes built images, volumes and networks, so if you disable it please carefully remove all those Docker resources. In any case, we have released a new version of Ryuk. As @stevenh pointed out in #2172 (comment), you could give it a try like that. |
@mdelapenya
I have 2 tests running in parallel and one of them failing in 90% of the cases. |
The creation of reapers in the current version is racy, could you test from the branch in #2664 which should fix this issue. |
@stevenh , the PR is a draft and has "DO NOT MERGE!" on it. Does it mean, maybe, that this issue is not actually fixed and should be opened? |
@strowk thanks for checking. I asked Steven to split that massive PR into smaller chunks and it's allowing me to review it in a more efficient manner. The parent issue is #2685, it contains all the fixes, one by one as subtasks. Thanks to @stevenh work, now it's much easier to see what's been fixed/done |
To build on @mdelapenya response the reaper fix extraction is still pending as there's some dependencies in the stack. If your issue is fixed by using the branch in the big PR, then once the individual fixes are all merged we should be good, so testing with that for now is useful. |
not sure why this was closed, still experience it with the latest 0.33 release |
Yep seeing here too @mdelapenya can you reopen this, requires the reaper fixes. |
Thanks for checking, reopening. |
Ah it was autoclosed by merging #2648 |
thanks for re-opening this 🙏 |
Unfortunately not it requires a significant rewrite of the both the reaper itself and the reaper integration |
For us this happens about daily. It seems to be slightly less frequent after upgrading to 0.33 but not by much. |
We solved the problem by putting all the tests into the same package. In this case, a container spins up only once. It is far from ideal, but it works for small to medium projects. |
You could test the branch for this PR, which includes the reaper fixes, to see if that fixes the problem. |
@stevenh Just wanted to chime in and say that #2664 fixed the issue for me. Thanks! For anyone else who wants to use it as a patch until it’s merged, you can include it in go.mod like this:
And if you're using any modules you can also replace those...
|
I've tried going this route to test if the patch would work for my project but unfortunately it did not work. I still get many failures with the following details:
|
Same here! |
We're working our way through the full set of PR's the reaper one is next on the list, however some cases might need the new ryuk version too. @borod108 could you clarify the error you're seeing? |
How did you do this? I did
before running any containers, but I'm still getting this issue.
|
@axilis-marko the env variable must be set at the terminal/shell you use to run the tests, as documented here: https://golang.testcontainers.org/features/garbage_collector/#ryuk |
I don't have control over env variables in my CI pipeline, so programatically this did the trick func init() {
err := os.Setenv("TESTCONTAINERS_RYUK_DISABLED", "true")
if err != nil {
panic(err)
}
} Thanks @mdelapenya! |
Is this because the CI is owned by other team? If so, you could ask them about that need and request it. Of course, I don't know the internal procedures you have for that.
The drawback here is that it will apply to everybody cloning the repo, and because env vars override the properties setting, everybody will have Ryuk disabled by default, which we usually disregard. |
Yes. We'll probably ask them to expose a variable so we know we're in CI, so we can add this, and any future "hack" without pinging them constantly.
Currently, we prefer manually deleting our local containers than sporadic failures in the pipeline. As our CI/CD creates ephemeral pods for tests, they're gonna be destroyed either way. |
Sounds good to me :) In that case, make sure you call |
The message you list above isn't present in the current version of the code, could you make sure you're using the latest version or provide details of the test which is creating these messages please @shepherdjerred |
I was using v0.33.0. I've since updated to v0.34.0 for the |
Thanks @axilis-marko 0.34.0 has a significant refactor of the reaper in it so would be good to know if that addresses this issue. |
Testcontainers version
v0.27.0
Using the latest Testcontainers version?
Yes
Host OS
Ubuntu 22.04.3 LTS
Host arch
amd64
Go version
1.20
Docker version
Docker info
What happened?
This does not seem to happen when running locally but occasionally my tests fail in GitHub Actions with the error
creating reaper failed
.Relevant log output
Additional information
No response
The text was updated successfully, but these errors were encountered: