[BUG] Test org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex is Flaky #8352

Rishikesh1159 · 2023-06-29T13:34:51Z

Describe the bug
A clear and concise description of what the bug is.
org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex is flaky.

java.lang.AssertionError: Tried to stop the only cluster-manager eligible shared node
	at __randomizedtesting.SeedInfo.seed([D34AC87DB7FB7699:2BBC977EF7931736]:0)
	at org.opensearch.test.InternalTestCluster.stopRandomNode(InternalTestCluster.java:1789)
	at org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex(RestoreSnapshotIT.java:438)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:833)

Full stack trace: here

To Reproduce
Steps to reproduce the behavior:

./gradlew ':server:internalClusterTest' --tests "org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex" -Dtests.seed=D34AC87DB7FB7699 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=de-AT -Dtests.timezone=America/Nuuk -Druntime.java=17

Expected behavior
A clear and concise description of what you expected to happen.

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

OS: [e.g. iOS]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

nknize · 2023-06-30T19:27:52Z

Found another on #8382 and it was reproducible locally.

./gradlew ':server:internalClusterTest' --tests "org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex" -Dtests.seed=8CA2A73B7F437193 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sv-SE -Dtests.timezone=MST7MDT -Druntime.java=17

[testindex1/_DLgbwL0SEaGw-HtK67jdw], shard=[testindex1][0]]
org.opensearch.snapshots.SnapshotMissingException: [test-restore-snapshot-repo:test-restore-snapshot1/MiJULTGxSiWcbKXSap9dVQ] is missing
	at org.opensearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3117) ~[main/:?]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.getShardSnapshotStatus(BlobStoreRepository.java:2924) ~[main/:?]
	at org.opensearch.snapshots.InternalSnapshotsInfoService$FetchingSnapshotShardSizeRunnable.doRun(InternalSnapshotsInfoService.java:240) [main/:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [main/:?]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [main/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.nio.file.NoSuchFileException: /var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.snapshots.RestoreSnapshotIT_8CA2A73B7F437193-004/tempDir-002/repos/CvLRFjlcDt/indices/_DLgbwL0SEaGw-HtK67jdw/0/snap-MiJULTGxSiWcbKXSap9dVQ.dat
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218) ~[?:?]
	at java.nio.file.Files.newByteChannel(Files.java:380) ~[?:?]
	at java.nio.file.Files.newByteChannel(Files.java:432) ~[?:?]
	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422) ~[?:?]
	at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newInputStream(FilterFileSystemProvider.java:193) ~[lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newInputStream(FilterFileSystemProvider.java:193) ~[lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newInputStream(FilterFileSystemProvider.java:193) ~[lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.tests.mockfile.HandleTrackingFS.newInputStream(HandleTrackingFS.java:94) ~[lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newInputStream(FilterFileSystemProvider.java:193) ~[lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at org.apache.lucene.tests.mockfile.HandleTrackingFS.newInputStream(HandleTrackingFS.java:94) ~[lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
	at java.nio.file.Files.newInputStream(Files.java:160) ~[?:?]
	at org.opensearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:170) ~[main/:?]
	at org.opensearch.repositories.blobstore.ChecksumBlobStoreFormat.read(ChecksumBlobStoreFormat.java:121) ~[main/:?]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3115) ~[main/:?]
	... 7 more

sachinpkale · 2023-06-30T19:34:25Z

@harishbhakuni21 Can you please take a look?

Rishikesh1159 · 2023-07-03T17:10:09Z

Had a quick look into this flaky test. It fails with multiple type of failures. Here are few failures:

java.lang.AssertionError: Tried to stop the only cluster-manager eligible shared node
	at __randomizedtesting.SeedInfo.seed([D34AC87DB7FB7699:2BBC977EF7931736]:0)


./gradlew ':server:internalClusterTest' --tests "org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex" -Dtests.seed=8CA2A73B7F437193 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sv-SE -Dtests.timezone=MST7MDT -Druntime.java=17


org.opensearch.snapshots.SnapshotMissingException: [test-restore-snapshot-repo:test-restore-snapshot1/MiJULTGxSiWcbKXSap9dVQ] is missing
	at org.opensearch.repositories.blobstore.BlobStoreRepository.loadShardSnapshot(BlobStoreRepository.java:3117) ~[main/:?]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.getShardSnapshotStatus(BlobStoreRepository.java:2924) ~[main/:?]
	at org.opensearch.snapshots.InternalSnapshotsInfoService$FetchingSnapshotShardSizeRunnable.doRun(InternalSnapshotsInfoService.java:240) [main/:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [main/:?]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [main/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.nio.file.NoSuchFileException: /var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.snapshots.RestoreSnapshotIT_8CA2A73B7F437193-004/tempDir-002/repos/CvLRFjlcDt/indices/_DLgbwL0SEaGw-HtK67jdw/0/snap-MiJULTGxSiWcbKXSap9dVQ.dat

REPRODUCE WITH: ./gradlew 'null' --tests "org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex" -Dtests.seed=943DB37ABEDFD4A8 -Dtests.locale=en-US -Dtests.timezone=Africa/Khartoum -Druntime.java=14

NodeClosedException[node closed {node_s7}{GlcOjvQiR3ereHzpRji7Cg}{xF1004YkQs6yUQeIl004_Q}{127.0.0.1}{127.0.0.1:42999}{d}{shard_indexing_pressure_enabled=true}
]
	at __randomizedtesting.SeedInfo.seed([943DB37ABEDFD4A8:6CCBEC79FEB7B507]:0)
	at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onClusterServiceClose(TransportReplicationAction.java:1125)
	at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:387)

Rishikesh1159 · 2023-07-03T17:18:59Z

By making following changes, few of the above failures are fixed:

-> Change L351 to final String firstNode = internalCluster().startDataOnlyNode();
-> Change L381 to final String secondNode = internalCluster().startDataOnlyNode();
-> Change L438 to internalCluster().stopRandomNode(InternalTestCluster.nameFilter(firstNode));

But after running for few iterations with above fix I still see

REPRODUCE WITH: ./gradlew 'null' --tests "org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex" -Dtests.seed=943DB37ABEDFD4A8 -Dtests.locale=en-US -Dtests.timezone=Africa/Khartoum -Druntime.java=14

NodeClosedException[node closed {node_s7}{GlcOjvQiR3ereHzpRji7Cg}{xF1004YkQs6yUQeIl004_Q}{127.0.0.1}{127.0.0.1:42999}{d}{shard_indexing_pressure_enabled=true}
]
	at __randomizedtesting.SeedInfo.seed([943DB37ABEDFD4A8:6CCBEC79FEB7B507]:0)
	at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onClusterServiceClose(TransportReplicationAction.java:1125)
	at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:387)

Needs more deep dive

sachinpkale · 2023-07-04T08:14:40Z

Fixed in #8422

Rishikesh1159 added bug Something isn't working untriaged distributed framework labels Jun 29, 2023

nknize mentioned this issue Jun 30, 2023

[Backport 2.x] [Refactor] Version and LegacyESVersion from server module to core lib (#7328) #8382

Merged

BhumikaSaini-Amazon mentioned this issue Jul 3, 2023

[Remote Segment Store] Add Lucene major version to UploadedSegmentMetadata #8088

Merged

6 tasks

sachinpkale mentioned this issue Jul 4, 2023

Fix flaky tests in restore flow using snapshot interop #8422

Merged

6 tasks

ashking94 mentioned this issue Jul 4, 2023

[AUTOCUT] Gradle Check Failure on push to 2.x #8423

Closed

sachinpkale closed this as completed Jul 4, 2023

anasalkouz removed the untriaged label Jul 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Test org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex is Flaky #8352

[BUG] Test org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex is Flaky #8352

Rishikesh1159 commented Jun 29, 2023

nknize commented Jun 30, 2023 •

edited

Loading

sachinpkale commented Jun 30, 2023

Rishikesh1159 commented Jul 3, 2023 •

edited

Loading

Rishikesh1159 commented Jul 3, 2023 •

edited

Loading

sachinpkale commented Jul 4, 2023

[BUG] Test org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex is Flaky #8352

[BUG] Test org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex is Flaky #8352

Comments

Rishikesh1159 commented Jun 29, 2023

nknize commented Jun 30, 2023 • edited Loading

sachinpkale commented Jun 30, 2023

Rishikesh1159 commented Jul 3, 2023 • edited Loading

Rishikesh1159 commented Jul 3, 2023 • edited Loading

sachinpkale commented Jul 4, 2023

nknize commented Jun 30, 2023 •

edited

Loading

Rishikesh1159 commented Jul 3, 2023 •

edited

Loading

Rishikesh1159 commented Jul 3, 2023 •

edited

Loading