Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testRequestStats is flaky #9859

Closed
shiv0408 opened this issue Sep 6, 2023 · 5 comments · Fixed by #12014
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run

Comments

@shiv0408
Copy link
Member

shiv0408 commented Sep 6, 2023

Describe the bug
The org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testRequestStats is flaky

java.lang.AssertionError: Only index blobs should remain in repository but found [/container/tests-1Gw9amxARjeifC0te8CG_A/master.dat]
    Expected: a collection with size <0>
         but: collection size was <1>
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.opensearch.repositories.blobstore.OpenSearchMockAPIBasedRepositoryIntegTestCase.tearDownHttpServer(OpenSearchMockAPIBasedRepositoryIntegTestCase.java:145)
        at java.****/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.****/java.lang.reflect.Method.invoke(Method.java:578)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)

To Reproduce

./gradlew ':plugins:repository-azure:internalClusterTest' --tests "org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testRequestStats" -Dtests.seed=D1E7629061A3A0D2 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-OM -Dtests.timezone=Europe/Guernsey -Druntime.java=20

Expected behavior
Test must always pass

Plugins
Standard

Host/Environment (please complete the following information):

  • OS: [e.g. iOS]
  • Version [e.g. 22]

Additional context
https://build.ci.opensearch.org/job/gradle-check/24788/testReport/org.opensearch.repositories.azure/AzureBlobStoreRepositoryTests/testRequestStats/

@saratvemulapalli
Copy link
Member

This was closed with an unrelated PR.
Anyway I spent sometime on this test, I tried running this over 6k times and it passes.

BUILD SUCCESSFUL in 35s
65 actionable tasks: 6 executed, 59 up-to-date
Run #No (7269)

testRequestStats is run by multiple repositories implementations: https://github.com/opensearch-project/OpenSearch/blob/main/test/framework/src/main/java/org/opensearch/repositories/blobstore/OpenSearchMockAPIBasedRepositoryIntegTestCase.java#L190

S3 Repo failure: https://build.ci.opensearch.org/job/gradle-check/28369/testReport/junit/org.opensearch.repositories.s3/S3BlobStoreRepositoryTests/testRequestStats/

Azure Repo Failure:

  1. https://build.ci.opensearch.org/job/gradle-check/27280/consoleFull
org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests > testRequestStats FAILED
    java.lang.AssertionError: SDK sent [{PutBlock=4, GetBlobProperties=3, PutBlockList=4, GetBlob=34, PutBlob=35, ListBlobs=6}] calls and handler measured [{PutBlob=35, ListBlobs=7, GetBlob=34, GetBlobProperties=3, PutBlockList=4, PutBlock=4}] calls expected:<{PutBlob=35, ListBlobs=7, GetBlob=34, GetBlobProperties=3, PutBlockList=4, PutBlock=4}> but was:<{PutBlock=4, GetBlobProperties=3, PutBlockList=4, GetBlob=34, PutBlob=35, ListBlobs=6}>
  1. https://build.ci.opensearch.org/job/gradle-check/24788/testReport/org.opensearch.repositories.azure/AzureBlobStoreRepositoryTests/testRequestStats/
RepositoryVerificationException[[y] path  is not accessible on cluster-manager node]; nested: FileAlreadyExistsException[tests-1Gw9amxARjeifC0te8CG_A/master.dat: Status code 409, "<?xml version="1.0" encoding="UTF-8"?><Error><Code>BlobAlreadyExists</Code><Message>CONFLICT</Message></Error>"];
	at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1856)
	at org.opensearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:356)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1623)
Caused by: java.nio.file.FileAlreadyExistsException: tests-1Gw9amxARjeifC0te8CG_A/master.dat: Status code 409, "<?xml version="1.0" encoding="UTF-8"?><Error><Code>BlobAlreadyExists</Code><Message>CONFLICT</Message></Error>"
	at org.opensearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:379)
	at org.opensearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:133)
	at org.opensearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:141)
	at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1848)
	... 6 more
  1. https://build.ci.opensearch.org/job/gradle-check/31603/consoleFull
org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests > testRequestStats FAILED
    java.lang.AssertionError: SDK sent [{PutBlock=4, PutBlob=37, ListBlobs=7, GetBlob=35, GetBlobProperties=3, PutBlockList=4}] calls and handler measured [{PutBlock=4, ListBlobs=7, PutBlob=38, GetBlob=35, GetBlobProperties=3, PutBlockList=4}] calls expected:<{PutBlock=4, ListBlobs=7, PutBlob=38, GetBlob=35, GetBlobProperties=3, PutBlockList=4}> but was:<{PutBlock=4, PutBlob=37, ListBlobs=7, GetBlob=35, GetBlobProperties=3, PutBlockList=4}>

@saratvemulapalli
Copy link
Member

saratvemulapalli commented Jan 30, 2024

Opened a PR #12079 to add additional logging incase we see the failure again.
Feel free to re-open again.

@shwetathareja
Copy link
Member

Failed again - #12773 (comment)

@peternied
Copy link
Member

[Triage - attendees 1 2 3 4 5 6 7 8]
@shwetathareja Thanks for reopening this issue

@reta
Copy link
Collaborator

reta commented Jun 19, 2024

Closing in favour of #14291

@reta reta closed this as completed Jun 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants