Skip to content

Conversation

mkhludnev
Copy link
Contributor

  • I replaced RandomIndexWriter to plain IndexWriter to avoid unexpected segments in test which are unable to handle more than single segment
  • for other test I've added explicit segmenting. TODO It needs to be reviewed.
  • Perhaps StreamingTermsAggregator should reject second segment via assert or there should be a proper publish/reset flow. TODO, WIP

so far this commit fixes the known reason for flackines (I belive)

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

- I replaced RandomIndexWriter to plain IndexWriter to avoid unexpected segments in test which are unable to handle more than single segment
- for other test I've added explicit segmenting. TODO It needs to be reviewed.
- Perhaps StreamingTermsAggregator should reject second segment via assert or there should be a proper publish/reset flow. TODO, WIP

so far this commit fixes the known reason for flackines (I belive)

Signed-off-by: Mikhail Khludnev <[email protected]>
@mkhludnev
Copy link
Contributor Author

Got reproducible failure on the single segment. It doesn't comply with the current hypothesis.

REPRODUCE WITH: ./gradlew 'null' --tests 'org.opensearch.search.aggregations.bucket.terms.StreamStringTermsAggregatorTests.testReduceSingleAggregation' -Dtests.seed=71C9C5CB31106539 -Dtests.locale=ee -Dtests.timezone=ACT -Druntime.java=21

java.lang.AssertionError: 
Expected: <3>
     but: was <1>
Expected :<3>
Actual   :<1>
<Click to see difference>

	at __randomizedtesting.SeedInfo.seed([71C9C5CB31106539:9E6BEB57758E488D]:0)
	at org.opensearch.search.aggregations.bucket.terms.StreamStringTermsAggregatorTests.testReduceSingleAggregation(StreamStringTermsAggregatorTests.java:```

Copy link
Contributor

❕ Gradle check result for b5cd364: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Copy link

codecov bot commented Sep 24, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.93%. Comparing base (6fdb010) to head (b76c2a2).
⚠️ Report is 7 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #19404      +/-   ##
============================================
+ Coverage     72.90%   72.93%   +0.02%     
+ Complexity    69915    69859      -56     
============================================
  Files          5675     5675              
  Lines        320841   320865      +24     
  Branches      46387    46388       +1     
============================================
+ Hits         233909   234011     +102     
+ Misses        68031    67843     -188     
- Partials      18901    19011     +110     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rishabhmaurya
Copy link
Contributor

@mkhludnev do you want to address TODO as part of different PR? If yes, we can probably merge this one

@mkhludnev
Copy link
Contributor Author

fixed.

testReduceSingleAggregation
newIndexWriterConfig() sets MaxDocsBuffered randomly and caused index segmentation. Replaced it with newIndexWriterConfig()

address TODO as part of different PR? If yes, we can probably merge this one

Right. We can treat this PR as urgent fix for reducing CI noize.

Signed-off-by: Mikhail Khludnev <[email protected]>
Copy link
Contributor

❌ Gradle check result for bca5be7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Mikhail Khludnev <[email protected]>
Copy link
Contributor

✅ Gradle check result for b76c2a2: SUCCESS

@mkhludnev
Copy link
Contributor Author

TODO as part of different PR?

attempt #19416

@rishabhmaurya rishabhmaurya merged commit 7c6b3c5 into opensearch-project:main Sep 25, 2025
33 checks passed
@mkhludnev
Copy link
Contributor Author

this PR left RandomIndexWriter in testBuildAggregationsBatchWithCountOrder and testBuildAggregationsBatchReset which made'em prone to segmentation. I've fixed these tests in #19416

vinaykpud pushed a commit to vinaykpud/OpenSearch that referenced this pull request Sep 26, 2025
…ith plain IndexWriter opensearch-project#18956 WIP (opensearch-project#19404)

* fix opensearch-project#18956 WIP

- I replaced RandomIndexWriter to plain IndexWriter to avoid unexpected segments in test which are unable to handle more than single segment
- for other test I've added explicit segmenting. TODO It needs to be reviewed.
- Perhaps StreamingTermsAggregator should reject second segment via assert or there should be a proper publish/reset flow. TODO, WIP

so far this commit fixes the known reason for flakiness (I believe)

Signed-off-by: Mikhail Khludnev <[email protected]>

* newIndexWriterConfig() causes segmenting via flushes by bufferedDocs

Signed-off-by: Mikhail Khludnev <[email protected]>

* subAggs test goes single segment

Signed-off-by: Mikhail Khludnev <[email protected]>

* asserting single segment

Signed-off-by: Mikhail Khludnev <[email protected]>

---------

Signed-off-by: Mikhail Khludnev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants