Async Batch shards changes for GatewayAllocator #8746

Gaurav614 · 2023-07-18T11:28:52Z

Description

This pull request is part of the improvement #5098
And mainly is around creation of batches, cleaning of batcher and other changes related to AsyncBatchShardFetch in GatewayAllocator

The PR is dependent on following PRs:
#8742
#8218
#8356

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
[x] All tests pass
New functionality has been documented.
[x] New functionality has javadoc added
Commits are signed per the DCO using --signoff
~~Commit changes are listed out in CHANGELOG.md file (See: Changelog)~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-07-18T11:37:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20403/
CommitID: fb3e757
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-19T08:47:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20497/
CommitID: c47bc4d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/gateway/GatewayAllocator.java

github-actions · 2023-07-25T10:43:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20893/
CommitID: 18a5d9c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

amkhar

Thanks @Gaurav614 for the changes. Partially gone through it, left minor comments.

server/src/main/java/org/opensearch/cluster/routing/allocation/ExistingShardsAllocator.java

server/src/main/java/org/opensearch/gateway/GatewayAllocator.java

amkhar

Please re-check for any other race conditions if possible.

server/src/main/java/org/opensearch/gateway/GatewayAllocator.java

github-actions · 2023-08-31T02:59:34Z

Compatibility status:

Checks if related components are compatible with change 8153a69

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/sql.git]

github-actions · 2023-08-31T03:07:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23904/
CommitID: f0cdd68
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-06T07:52:18Z

Compatibility status:

Checks if related components are compatible with change f4b32b2

Incompatible components

Skipped components

Compatible components

github-actions · 2023-09-06T07:57:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24582/
CommitID: f4b32b2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-06T08:46:24Z

Compatibility status:

Checks if related components are compatible with change b9407cc

Incompatible components

Skipped components

Compatible components

github-actions · 2023-09-06T08:51:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24590/
CommitID: b9407cc
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-28T07:54:37Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26439/
CommitID: f03e99e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

shiv0408 · 2023-11-30T12:08:19Z

Added the required changes to GatewayAllocator for running the afterPrimaryBeforeReplica method to this PR as this PR has all the changes required to run the AsyncShardBatch flow.

github-actions · 2023-11-30T12:11:41Z

❌ Gradle check result for b776483: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/internalClusterTest/java/org/opensearch/gateway/RecoveryFromGatewayIT.java

server/src/main/java/org/opensearch/cluster/routing/allocation/ExistingShardsAllocator.java

server/src/main/java/org/opensearch/gateway/BaseGatewayShardAllocator.java

server/src/main/java/org/opensearch/gateway/GatewayAllocator.java

Signed-off-by: Shivansh Arora <[email protected]>

# Conflicts: # server/src/internalClusterTest/java/org/opensearch/gateway/RecoveryFromGatewayIT.java # server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationService.java # server/src/main/java/org/opensearch/common/settings/ClusterSettings.java # server/src/main/java/org/opensearch/gateway/ReplicaShardBatchAllocator.java # server/src/main/java/org/opensearch/indices/store/TransportNodesListShardStoreMetadataBatch.java # server/src/main/java/org/opensearch/node/Node.java # server/src/test/java/org/opensearch/gateway/ShardBatchCacheTests.java # test/framework/src/main/java/org/opensearch/test/gateway/TestGatewayAllocator.java

Signed-off-by: Shivansh Arora <[email protected]>

mgodwan · 2024-04-25T17:59:46Z

@Gaurav614 Could you please ensure your branch has the latest changes from main? The failing test [org.opensearch.index.codec.fuzzy/FuzzyFilterPostingsFormatTests/classMethod/](https://build.ci.opensearch.org/job/gradle-check/37728/testReport/org.opensearch.index.codec.fuzzy/FuzzyFilterPostingsFormatTests/classMethod/) has been fixed a week ago in main, and syncing from main should help fix this in your gradle check.

Signed-off-by: Shivansh Arora <[email protected]>

github-actions · 2024-04-25T18:49:50Z

❌ Gradle check result for 3f103c1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-25T19:25:58Z

✅ Gradle check result for 154d13b: SUCCESS

codecov · 2024-04-25T19:28:11Z

Codecov Report

Attention: Patch coverage is 57.36196% with 139 lines in your changes are missing coverage. Please review.

Project coverage is 71.43%. Comparing base (b15cb0c) to head (080c3ba).
Report is 224 commits behind head on main.

Files	Patch %	Lines
...pensearch/gateway/ShardsBatchGatewayAllocator.java	60.97%	98 Missing and 14 partials ⚠️
...ore/TransportNodesListShardStoreMetadataBatch.java	0.00%	10 Missing ⚠️
.../cluster/routing/allocation/AllocationService.java	38.46%	7 Missing and 1 partial ⚠️
...er/routing/allocation/ExistingShardsAllocator.java	12.50%	7 Missing ⚠️
...va/org/opensearch/cluster/routing/RoutingNode.java	0.00%	1 Missing ⚠️
.../opensearch/gateway/BaseGatewayShardAllocator.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8746      +/-   ##
============================================
+ Coverage     71.42%   71.43%   +0.01%     
- Complexity    59978    60789     +811     
============================================
  Files          4985     5045      +60     
  Lines        282275   286042    +3767     
  Branches      40946    41427     +481     
============================================
+ Hits         201603   204327    +2724     
- Misses        63999    64832     +833     
- Partials      16673    16883     +210

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Shivansh Arora <[email protected]>

github-actions · 2024-04-26T03:02:32Z

✅ Gradle check result for 080c3ba: SUCCESS

opensearch-trigger-bot · 2024-04-26T03:17:04Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-8746-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a69cd0864a673579a9cb2190da765bc1ebfd0a22
# Push it to GitHub
git push --set-upstream origin backport/backport-8746-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8746-to-2.x.

) Changes for create/update/delete batches for batch mode for async fetch for both primary & replica. It also added the node scope setting to enable/ disable batch mode. Signed-off-by: Gaurav Chandani <[email protected]> Signed-off-by: Shivansh Arora <[email protected]> Signed-off-by: Aman Khare <[email protected]> (cherry picked from commit a69cd08)

* Async Batch shards changes for GatewayAllocator (#8746) Changes for create/update/delete batches for batch mode for async fetch for both primary & replica. It also added the node scope setting to enable/ disable batch mode. Signed-off-by: Gaurav Chandani <[email protected]> Signed-off-by: Shivansh Arora <[email protected]> Signed-off-by: Aman Khare <[email protected]> (cherry picked from commit a69cd08)

Gaurav614 mentioned this pull request Jul 19, 2023

BaseGatewayShardAllocator changes for Assigning the batch of shards #8776

Merged

6 tasks

amkhar reviewed Jul 19, 2023

View reviewed changes

server/src/main/java/org/opensearch/gateway/GatewayAllocator.java Outdated Show resolved Hide resolved

amkhar reviewed Jul 20, 2023

View reviewed changes

server/src/main/java/org/opensearch/gateway/GatewayAllocator.java Outdated Show resolved Hide resolved

This was referenced Jul 25, 2023

FetchData changes for primaries and replicas #8865

Closed

Allocation service changes for batch assignment #8888

Open

This was referenced Jul 26, 2023

Added transport action for bulk async shard fetch for primary shards #8218

Merged

[BUG] Async shard fetches taking up GBs of memory causing ClusterManager JVM to spike with large no. of shards (> 50k) #5098

Open

amkhar reviewed Jul 26, 2023

View reviewed changes

amkhar mentioned this pull request Jul 31, 2023

[META] Cluster Manager Async Shard Fetch Revamp #8098

Open

13 tasks

amkhar reviewed Aug 9, 2023

View reviewed changes

server/src/main/java/org/opensearch/gateway/GatewayAllocator.java Outdated Show resolved Hide resolved

shiv0408 mentioned this pull request Oct 17, 2023

Fixed Allocation Explain API in batch mode #10348

Closed

7 tasks

khushbr reviewed Dec 1, 2023

View reviewed changes

Gaurav614 marked this pull request as ready for review December 5, 2023 12:35

Gaurav614 requested review from abbashus, adnapibar, anasalkouz, andrross and Bukhtawar as code owners December 5, 2023 12:35

shiv0408 added 8 commits April 25, 2024 22:55

Keep initializing shards in batch

1afa3f4

Signed-off-by: Shivansh Arora <[email protected]>

Spotless apply

3b5751e

Signed-off-by: Shivansh Arora <[email protected]>

Update ensureAsyncFetchStorePrimaryRecency and refresh batch flow

8ae90fc

Signed-off-by: Shivansh Arora <[email protected]>

Revert ensureAsyncFetchStorePrimaryRecency update

16d6c59

Signed-off-by: Shivansh Arora <[email protected]>

Enable batch mode temporarily for gradle check run

0693b9a

Signed-off-by: Shivansh Arora <[email protected]>

Disabled batch mode by default

1ff4588

Signed-off-by: Shivansh Arora <[email protected]>

Spotless apply

3f103c1

Signed-off-by: Shivansh Arora <[email protected]>

shiv0408 force-pushed the GA-batcher-PR branch from c4cb153 to 3f103c1 Compare April 25, 2024 18:00

Fix for missed merge conflict

154d13b

Signed-off-by: Shivansh Arora <[email protected]>

Added changelog

080c3ba

Signed-off-by: Shivansh Arora <[email protected]>

shwetathareja added Cluster Manager and removed skip-changelog labels Apr 26, 2024

shwetathareja approved these changes Apr 26, 2024

View reviewed changes

shwetathareja added the backport 2.x Backport to 2.x branch label Apr 26, 2024

shwetathareja merged commit a69cd08 into opensearch-project:main Apr 26, 2024
32 of 35 checks passed

opensearch-trigger-bot bot added the backport-failed label Apr 26, 2024

shiv0408 mentioned this pull request Apr 28, 2024

[Backport 2.x] Async Batch shards changes for GatewayAllocator #13431

Merged

Pranshu-S mentioned this pull request Apr 30, 2024

[DOC] Update documentation for cluster settings to include Async shard batch fetch settings. opensearch-project/documentation-website#7055

Closed

4 tasks

BrewTestBot mentioned this pull request May 14, 2024

opensearch 2.14.0 Homebrew/homebrew-core#171722

Merged

shiv0408 assigned shiv0408 and Gaurav614 Aug 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Async Batch shards changes for GatewayAllocator #8746

Async Batch shards changes for GatewayAllocator #8746

Gaurav614 commented Jul 18, 2023 •

edited by shwetathareja

Loading

github-actions bot commented Jul 18, 2023

github-actions bot commented Jul 19, 2023

github-actions bot commented Jul 25, 2023

amkhar left a comment

amkhar left a comment

github-actions bot commented Aug 31, 2023 •

edited

Loading

github-actions bot commented Aug 31, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 28, 2023

shiv0408 commented Nov 30, 2023

github-actions bot commented Nov 30, 2023

mgodwan commented Apr 25, 2024

github-actions bot commented Apr 25, 2024

github-actions bot commented Apr 25, 2024

codecov bot commented Apr 25, 2024 •

edited

Loading

github-actions bot commented Apr 26, 2024

opensearch-trigger-bot bot commented Apr 26, 2024

Async Batch shards changes for GatewayAllocator #8746

Async Batch shards changes for GatewayAllocator #8746

Conversation

Gaurav614 commented Jul 18, 2023 • edited by shwetathareja Loading

Description

Related Issues

Check List

github-actions bot commented Jul 18, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 25, 2023

Gradle Check (Jenkins) Run Completed with:

amkhar left a comment

Choose a reason for hiding this comment

amkhar left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 31, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 6, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 6, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 28, 2023

Gradle Check (Jenkins) Run Completed with:

shiv0408 commented Nov 30, 2023

github-actions bot commented Nov 30, 2023

mgodwan commented Apr 25, 2024

github-actions bot commented Apr 25, 2024

github-actions bot commented Apr 25, 2024

codecov bot commented Apr 25, 2024 • edited Loading

Codecov Report

github-actions bot commented Apr 26, 2024

opensearch-trigger-bot bot commented Apr 26, 2024

Gaurav614 commented Jul 18, 2023 •

edited by shwetathareja

Loading

github-actions bot commented Aug 31, 2023 •

edited

Loading

codecov bot commented Apr 25, 2024 •

edited

Loading