Flaky Test: Upgrade tests failing as no entries exist in the persistentCache after upgrade

[ryanemerson]

The upgrade_operands_test.go is sometimes failing because the number of entries in the "persistentCache" is 0 after an upgrade, when it should be 100 as the cache is configured to use a file-store.

On a GracefulShutdown upgrade, the StatefulSet of the original cluster is deleted but the associated PVCs are kept and a new StatefulSet is created for the new cluster containing the upgraded Infinispan server. The expectation is that the data associated with the cache is written to the PVC and so after cluster upgrade all persisted entries should remain available via the cache.

Next Operand 15.0.13
Upgrading Operand to 15.0.13
GET: http://172.172.0.2:30552/rest/v2/container/health/status
GET: http://172.172.0.2:30552/rest/v2/caches/persistentCache?action=size
Encountered error: previous terminated container "manager" in pod "infinispan-operator-controller-manager-5d85f8984b-9dtqp" not found
Encountered error: previous terminated container "infinispan" in pod "test-operand-upgrades-0" not found
Encountered error: previous terminated container "infinispan" in pod "test-operand-upgrades-1" not found
Encountered error: previous terminated container "infinispan" in pod "14-0-13" not found
Encountered error: previous terminated container "infinispan" in pod "14-0-17" not found
Encountered error: previous terminated container "infinispan" in pod "14-0-19" not found
Encountered error: previous terminated container "infinispan" in pod "14-0-20" not found
Encountered error: previous terminated container "infinispan" in pod "14-0-21" not found
Encountered error: previous terminated container "infinispan" in pod "14-0-24" not found
Encountered error: previous terminated container "infinispan" in pod "14-0-27" not found
Encountered error: previous terminated container "infinispan" in pod "14-0-32" not found
Encountered error: previous terminated container "infinispan" in pod "14-0-9" not found
Encountered error: previous terminated container "infinispan" in pod "15-0-0" not found
Encountered error: previous terminated container "infinispan" in pod "15-0-10" not found
Encountered error: previous terminated container "infinispan" in pod "15-0-11" not found
Encountered error: previous terminated container "infinispan" in pod "15-0-3" not found
Encountered error: previous terminated container "infinispan" in pod "15-0-4" not found
Encountered error: previous terminated container "infinispan" in pod "15-0-5" not found
Encountered error: previous terminated container "infinispan" in pod "15-0-8" not found
Encountered error: previous terminated container "infinispan" in pod "upgrade-backup" not found
GET: http://localhost:11223/metrics
GET: http://localhost:11223/metrics
Encountered error: unable to write metrics to file for pod '14-0-13:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '14-0-17:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '14-0-19:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '14-0-20:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '14-0-21:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '14-0-24:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '14-0-27:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '14-0-32:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '14-0-9:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '15-0-0:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '15-0-10:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '15-0-11:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '15-0-3:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '15-0-4:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '15-0-5:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod '15-0-8:'namespace-for-testing': pod not running. status.phase='Succeeded'
Encountered error: unable to write metrics to file for pod 'upgrade-backup:'namespace-for-testing': pod not running. status.phase='Succeeded'
goroutine 97 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.23.7/x64/src/runtime/debug/stack.go:26 +0x5e
github.com/infinispan/infinispan-operator/test/e2e/utils.TestKubernetes.CleanNamespaceAndLogWithPanic({0xc0002d2d80}, 0xc0000cfa00, {0x188dd25, 0x15}, {0x1665280, 0xc000322c70})
	/home/runner/work/infinispan/infinispan/test/e2e/utils/kubernetes.go:224 +0x17c5
github.com/infinispan/infinispan-operator/test/e2e/utils.TestKubernetes.CleanupOLMTest({0x43cb20?}, 0xc0000cfa00, {0xc0006389c0, 0x13}, {0x188b117, 0x13}, {0x188dd25, 0x15}, {0x187f777, 0xa})
	/home/runner/work/infinispan/infinispan/test/e2e/utils/olm.go:178 +0x178
panic({0x1665280?, 0xc000322c70?})
	/opt/hostedtoolcache/go/1.23.7/x64/src/runtime/panic.go:791 +0x132
github.com/infinispan/infinispan-operator/test/e2e/utils.(*CacheHelper).AssertSize(0x1885388?, 0x64)
	/home/runner/work/infinispan/infinispan/test/e2e/utils/cache.go:44 +0xa5
github.com/infinispan/infinispan-operator/test/e2e/upgrade.TestOperandUpgrades(0xc0000cfa00)
	/home/runner/work/infinispan/infinispan/test/e2e/upgrade/upgrade_operands_test.go:97 +0xcef
testing.tRunner(0xc0000cfa00, 0x1973a40)
	/opt/hostedtoolcache/go/1.23.7/x64/src/testing/testing.go:1690 +0xf4
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.23.7/x64/src/testing/testing.go:1743 +0x390

    kubernetes.go:225: expected 100 entries found 0 
        
Encountered error: previous terminated container "manager" in pod "infinispan-operator-controller-manager-5d85f8984b-9dtqp" not found
Encountered error: previous terminated container "olm-operator" in pod "olm-operator-64cc47587b-jnt2m" not found
Encountered error: previous terminated container "catalog-operator" in pod "catalog-operator-7c4ff7c79b-6qzgm" not found
--- FAIL: TestOperandUpgrades (1425.27s)

Original run: https://github.com/infinispan/infinispan/actions/runs/13967255322/job/39100677903?pr=14410
upgrade-test-logs(3).zip