Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

e2e flake: fatal error: concurrent map read and map write #1571

Open
nirs opened this issue Sep 23, 2024 · 0 comments
Open

e2e flake: fatal error: concurrent map read and map write #1571

nirs opened this issue Sep 23, 2024 · 0 comments
Labels
bug Something isn't working test Testing related issue

Comments

@nirs
Copy link
Member

nirs commented Sep 23, 2024

Looks like a random failure - maybe we are missing locking on a map?

2024-09-23T18:24:15.804+0530	INFO	util/config.go:48	No configuration file specified, using default value config.yaml
2024-09-23T18:24:15.816+0530	INFO	e2e/main_test.go:41	Global setting	{"Timeout": 600}
2024-09-23T18:24:15.816+0530	INFO	e2e/main_test.go:42	Global setting	{"Retry Interval": 30}
=== RUN   TestSuites
2024-09-23T18:24:15.817+0530	INFO	e2e/main_test.go:57	TestSuites
=== RUN   TestSuites/Validate
=== RUN   TestSuites/Validate/CheckRamenHubOperatorStatus
2024-09-23T18:24:15.817+0530	INFO	e2e/validation_suite_test.go:25	enter CheckRamenHubOperatorStatus
2024-09-23T18:24:15.860+0530	INFO	e2e/validation_suite_test.go:33	Ramen Hub Operator is running	{"pod": "ramen-hub-operator-84d7dc89bd-dv7xh"}
=== RUN   TestSuites/Validate/CheckRamenSpokeOperatorStatus
2024-09-23T18:24:15.860+0530	INFO	e2e/validation_suite_test.go:40	enter CheckRamenSpokeOperatorStatus
2024-09-23T18:24:15.886+0530	INFO	e2e/validation_suite_test.go:48	Ramen Spoke Operator is running on cluster 1	{"pod": "ramen-dr-cluster-operator-896d8c9f6-bj742"}
2024-09-23T18:24:15.911+0530	INFO	e2e/validation_suite_test.go:59	Ramen Spoke Operator is running on cluster 2	{"pod": "ramen-dr-cluster-operator-896d8c9f6-6jp9r"}
=== RUN   TestSuites/Exhaustive
=== PAUSE TestSuites/Exhaustive
=== CONT  TestSuites/Exhaustive
2024-09-23T18:24:15.970+0530	INFO	util/crud.go:77	channel ramen-gitops is created
=== RUN   TestSuites/Exhaustive/Deploy-cephfs
=== PAUSE TestSuites/Exhaustive/Deploy-cephfs
=== RUN   TestSuites/Exhaustive/Deploy-cephfs#01
=== PAUSE TestSuites/Exhaustive/Deploy-cephfs#01
=== RUN   TestSuites/Exhaustive/Deploy-cephfs#02
=== PAUSE TestSuites/Exhaustive/Deploy-cephfs#02
=== RUN   TestSuites/Exhaustive/Deploy-rbd
=== PAUSE TestSuites/Exhaustive/Deploy-rbd
=== RUN   TestSuites/Exhaustive/Deploy-rbd#01
=== PAUSE TestSuites/Exhaustive/Deploy-rbd#01
=== RUN   TestSuites/Exhaustive/Deploy-rbd#02
=== PAUSE TestSuites/Exhaustive/Deploy-rbd#02
=== CONT  TestSuites/Exhaustive/Deploy-cephfs
=== RUN   TestSuites/Exhaustive/Deploy-cephfs/Subscr
=== CONT  TestSuites/Exhaustive/Deploy-rbd#01
=== RUN   TestSuites/Exhaustive/Deploy-rbd#01/Appset
=== PAUSE TestSuites/Exhaustive/Deploy-rbd#01/Appset
=== CONT  TestSuites/Exhaustive/Deploy-rbd#01/Appset
=== CONT  TestSuites/Exhaustive/Deploy-rbd
=== RUN   TestSuites/Exhaustive/Deploy-rbd#01/Appset/Deploy
=== RUN   TestSuites/Exhaustive/Deploy-rbd/Subscr
=== CONT  TestSuites/Exhaustive/Deploy-rbd#02
=== PAUSE TestSuites/Exhaustive/Deploy-rbd/Subscr
2024-09-23T18:24:15.971+0530	INFO	deployers/applicationset.go:17	enter Deploy appset-deploy-rbd-busybox
=== CONT  TestSuites/Exhaustive/Deploy-cephfs#01
=== CONT  TestSuites/Exhaustive/Deploy-cephfs#02
=== RUN   TestSuites/Exhaustive/Deploy-cephfs#02/Disapp
=== PAUSE TestSuites/Exhaustive/Deploy-cephfs#02/Disapp
=== RUN   TestSuites/Exhaustive/Deploy-rbd#02/Disapp
=== CONT  TestSuites/Exhaustive/Deploy-rbd/Subscr
=== CONT  TestSuites/Exhaustive/Deploy-cephfs#02/Disapp
fatal error: concurrent map read and map write
=== PAUSE TestSuites/Exhaustive/Deploy-rbd#02/Disapp

goroutine 25 [running]:
github.com/ramendr/ramen/e2e/testcontext.GetTestContext({0xc0004b4240, 0x27})
	/home/github/actions-runner/_work/ramen/ramen/e2e/testcontext/testcontext.go:36 +0x38
github.com/ramendr/ramen/e2e_test.runTestFlow(0xc00069e000)
	/home/github/actions-runner/_work/ramen/ramen/e2e/exhaustive_suite_test.go:104 +0x45
github.com/ramendr/ramen/e2e_test.Exhaustive.func2.1(0xc00069e000)
	/home/github/actions-runner/_work/ramen/ramen/e2e/exhaustive_suite_test.go:93 +0xcc
testing.tRunner(0xc00069e000, 0xc000234000)
	/usr/local/go/src/testing/testing.go:1690 +0xf4
created by testing.(*T).Run in goroutine 1[14](https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349#step:12:15)
	/usr/local/go/src/testing/testing.go:1743 +0x390

goroutine 1 [chan receive]:
testing.(*T).Run(0xc00040b380, {0x187514f?, 0x0?}, 0x1969ce0)
	/usr/local/go/src/testing/testing.go:1751 +0x3ab
testing.runTests.func1(0xc00040b380)
	/usr/local/go/src/testing/testing.go:2168 +0x37
testing.tRunner(0xc00040b380, 0xc00005db30)
	/usr/local/go/src/testing/testing.go:1690 +0xf4
testing.runTests(0xc00049a330, {0x26dd020, 0x1, 0x1}, {0xe?, 0x1?, 0x0?})
	/usr/local/go/src/testing/testing.go:2166 +0x43d
testing.(*M).Run(0xc0004bfea0)
	/usr/local/go/src/testing/testing.go:2034 +0x64a
github.com/ramendr/ramen/e2e_test.TestMain(0xc0004bfea0)
	/home/github/actions-runner/_work/ramen/ramen/e2e/main_test.go:44 +0x2c5
main.main()
	_testmain.go:47 +0xa8

goroutine 51 [chan receive]:
testing.tRunner.func1()
	/usr/local/go/src/testing/testing.go:1651 +0x49b
testing.tRunner(0xc00040b6c0, 0x1969ce0)
	/usr/local/go/src/testing/testing.go:1696 +0x12d
created by testing.(*T).Run in goroutine 1
	/usr/local/go/src/testing/testing.go:1743 +0x390

	/usr/local/go/src/testing/testing.go:562 +0xa5
testing.(*T).Parallel(0xc000626820)
	/usr/local/go/src/testing/testing.go:1479 +0x1a5
github.com/ramendr/ramen/e2e_test.Exhaustive.func2.1(0xc000626820)
	/home/github/actions-runner/_work/ramen/ramen/e2e/exhaustive_suite_test.go:91 +0x45
testing.tRunner(0xc000626820, 0xc000493cb0)
	/usr/local/go/src/testing/testing.go:1690 +0xf4
created by testing.(*T).Run in goroutine 79
	/usr/local/go/src/testing/testing.go:1743 +0x390

goroutine 26 [runnable]:
testing.(*common).frameSkip(0xc00069e340, 0x4)
	/usr/local/go/src/testing/testing.go:755 +0x593
testing.(*common).decorate(0x50?, {0xc0006a6050, 0x42}, 0xc00050[15](https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349#step:12:16)08?)
	/usr/local/go/src/testing/testing.go:771 +0x35
testing.(*common).logDepth(0xc00069e340, {0xc0006a6050, 0x42}, 0x3)
	/usr/local/go/src/testing/testing.go:1037 +0x14a
testing.(*common).log(...)
	/usr/local/go/src/testing/testing.go:1011
testing.(*common).Skipf(0xc00069e340, {0x18b3fdb?, 0xc000413200?}, {0xc000629ef0?, 0x[16](https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349#step:12:17)aeb00?, 0x473db0?})
	/usr/local/go/src/testing/testing.go:1103 +0x54
github.com/ramendr/ramen/e2e_test.runTestFlow(0xc00069e340)
	/home/github/actions-runner/_work/ramen/ramen/e2e/exhaustive_suite_test.go:110 +0x16e
github.com/ramendr/ramen/e2e_test.Exhaustive.func2.1(0xc00069e340)
	/home/github/actions-runner/_work/ramen/ramen/e2e/exhaustive_suite_test.go:93 +0xcc
testing.tRunner(0xc00069e340, 0xc000234b40)
	/usr/local/go/src/testing/testing.go:1690 +0xf4
created by testing.(*T).Run in goroutine 81
	/usr/local/go/src/testing/testing.go:[17](https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349#step:12:18)43 +0x390

goroutine 27 [runnable]:
sync.(*Map).LoadAndDelete(0x26ff9a0, {0x15d0ca0, 0xc00062fed8})
	/usr/local/go/src/sync/map.go:296 +0x[19](https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349#step:12:20)2
sync.(*Map).Delete(...)
	/usr/local/go/src/sync/map.go:321
testing.(*T).Parallel(0xc00070a000)
	/usr/local/go/src/testing/testing.go:1481 +0x1dd
github.com/ramendr/ramen/e2e_test.Exhaustive.func2.1(0xc00070a000)
	/home/github/actions-runner/_work/ramen/ramen/e2e/exhaustive_suite_test.go:91 +0x45
testing.tRunner(0xc00070a000, 0xc0004408d0)
	/usr/local/go/src/testing/testing.go:1690 +0xf4
created by testing.(*T).Run in goroutine 116
	/usr/local/go/src/testing/testing.go:1743 +0x390
make: *** [Makefile:198: e2e-rdr] Error 1
exit status 2
FAIL	github.com/ramendr/ramen/e2e	0.[22](https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349#step:12:23)3s

Build: https://github.com/RamenDR/ramen/actions/runs/10992671688/job/30522993349
@nirs nirs added bug Something isn't working test Testing related issue labels Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working test Testing related issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nirs