Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Async Batch shards changes for GatewayAllocator #8746

Merged
merged 46 commits into from
Apr 26, 2024
Merged

Async Batch shards changes for GatewayAllocator #8746

merged 46 commits into from
Apr 26, 2024

Commits on Apr 25, 2024

  1. Draft changes for creation of batches 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    7f13975 View commit details
    Browse the repository at this point in the history

  2. WIP changes for GA changes for fetching 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    637fca0 View commit details
    Browse the repository at this point in the history

  3. Batcher changes in GA 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    49f7ce1 View commit details
    Browse the repository at this point in the history

  4. Binding of Trasnport Classes to Injector 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    9b51dfb View commit details
    Browse the repository at this point in the history

  5. Updated ShardsBatch to take ShardRouting into account instead of Shar… 

    …dIds

Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    0075bb2 View commit details
    Browse the repository at this point in the history

  6. Cleaned Shards from batches after allocation attempt and moved keys t… 

    …o ShardIds in ShardsBatch

1. Cleaning of shards from batches in shard started and shardFailed
2. ShardRouting Equals method have nodeId and unassignedInfo for equals.
The state changes after assignment so removing it from batcher class becomes difficult
Therefore added ShardsId as map key in Shardsbatch class and new EntryClass for it to hold the
values for it

Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    38205cf View commit details
    Browse the repository at this point in the history

  7. Allocation service changes for batch assignment 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    9bbba2e View commit details
    Browse the repository at this point in the history

  8. Added in-flight fetches for batchMode 

    Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    818ef1d View commit details
    Browse the repository at this point in the history

  9. Fixed bugs 

    1. updating shard routing in batches in accordance with previous behaviour
2. Closing of AsyncShardFetch in cleanCaches()
3. Updated in getNumberOfInflightFetches for batch maps
4. Renamed variable names

Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    1eabedf View commit details
    Browse the repository at this point in the history

  10. Fixed typos and added tests 

    Signed-off-by: Gaurav Chandani <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    80afedf View commit details
    Browse the repository at this point in the history

  11. Added batch mode setting 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    9238ebf View commit details
    Browse the repository at this point in the history

  12. Added missed clusterModule changes 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    728167c View commit details
    Browse the repository at this point in the history

  13. Modify GA for running RSBA 

    Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    7fe6709 View commit details
    Browse the repository at this point in the history

  14. Addressed PR comments 

    1. Moved setting of batch enable disable in this PR
2. Added java docs

Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    214aaa4 View commit details
    Browse the repository at this point in the history

  15. Addressed PR comments 

    1. Added a serparate class for batch mode of GA
2. nit comments of renaming variables
3. Added ClupritShardTest

Signed-off-by: Shivansh Arora <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    5992235 View commit details
    Browse the repository at this point in the history

  16. Added Changes for ShardBatchGatewayAllocator 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    7eb12b9 View commit details
    Browse the repository at this point in the history

  17. Fixed culprit shard test 

    Signed-off-by: Gaurav Chandani <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    d741081 View commit details
    Browse the repository at this point in the history

  18. Cosmetic changes in AllocationService 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    809c99f View commit details
    Browse the repository at this point in the history

  19. PR comments 

    1. Made changes so that Allocation Service run only default implementation of batch mode
2. Renamed methods
3. Added and modified documenatation

Signed-off-by: Gaurav Chandani <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    340abe4 View commit details
    Browse the repository at this point in the history

  20. Changes to implement interface for ShardBatchGatewayAllocator 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    4ee2040 View commit details
    Browse the repository at this point in the history

  21. Fixed PR comments around documentation 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    9f6284c View commit details
    Browse the repository at this point in the history

  22. Renamed allocateUnassignedBatch to allocateAllUnassignedShards 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    38692a0 View commit details
    Browse the repository at this point in the history

  23. Removed batchMode variable from ShardsBatchGatewayAllocator 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    20312bd View commit details
    Browse the repository at this point in the history

  24. Fix TestShardBatchGatewayAllocator 

    Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    8880e80 View commit details
    Browse the repository at this point in the history

  25. Renamed variables and added documentations 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    79a99e0 View commit details
    Browse the repository at this point in the history

  26. Use new AsyncShardFetchBatch class for creating cache for batch trans… 

    …port actions

Signed-off-by: Aman Khare <[email protected]>
    @shiv0408
    Aman Khare authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    c2da82c View commit details
    Browse the repository at this point in the history

  27. Correct version check with 3.0.0 

    Signed-off-by: Aman Khare <[email protected]>
    @shiv0408
    Aman Khare authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    d2e5522 View commit details
    Browse the repository at this point in the history

  28. Local lookup map for shardId to batchId, more code comments 

    Signed-off-by: Aman Khare <[email protected]>
    @shiv0408
    Aman Khare authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    4e4fa06 View commit details
    Browse the repository at this point in the history

  29. Remove shards from batches if they are not present in unassigned list… 

    … from allocation object

Signed-off-by: Aman Khare <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    Aman Khare authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    3b851b9 View commit details
    Browse the repository at this point in the history

  30. Resolved PR comments 

    1.Stop data nodes in test cases deterministically instead of random
2.Minor rename of variables

Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    edc66dd View commit details
    Browse the repository at this point in the history

  31. Incorporate AsyncShardBatchFetch class changes 

    Signed-off-by: Aman Khare <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    Aman Khare authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    26a90ce View commit details
    Browse the repository at this point in the history

  32. Fixed GatewayAllocatorTests 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    3da0af3 View commit details
    Browse the repository at this point in the history

  33. Delete batch if empty when removing shards from batch 

    Signed-off-by: Aman Khare <[email protected]>
    @shiv0408
    Aman Khare authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    dbd51b5 View commit details
    Browse the repository at this point in the history

  34. Fix index deletion test 

    Signed-off-by: Aman Khare <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    Aman Khare authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    eff1d2e View commit details
    Browse the repository at this point in the history

  35. Spotless Apply 

    Signed-off-by: Gaurav Chandani <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    6f52f9f View commit details
    Browse the repository at this point in the history

  36. Refactored to reuse duplicate code 

    Signed-off-by: Shivansh Arora <[email protected]>
    @Gaurav614 @shiv0408
    Gaurav614 authored and shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    7298a08 View commit details
    Browse the repository at this point in the history

  37. Keep initializing shards in batch 

    Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    1afa3f4 View commit details
    Browse the repository at this point in the history

  38. Spotless apply 

    Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    3b5751e View commit details
    Browse the repository at this point in the history

  39. Update ensureAsyncFetchStorePrimaryRecency and refresh batch flow 

    Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    8ae90fc View commit details
    Browse the repository at this point in the history

  40. Revert ensureAsyncFetchStorePrimaryRecency update 

    Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    16d6c59 View commit details
    Browse the repository at this point in the history

  41. Enable batch mode temporarily for gradle check run 

    Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    0693b9a View commit details
    Browse the repository at this point in the history

  42. Disabled batch mode by default 

    Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    1ff4588 View commit details
    Browse the repository at this point in the history

  43. Merge branch 'main' into GA-batcher-PR 

    # Conflicts:
#	server/src/internalClusterTest/java/org/opensearch/gateway/RecoveryFromGatewayIT.java
#	server/src/main/java/org/opensearch/cluster/routing/allocation/AllocationService.java
#	server/src/main/java/org/opensearch/common/settings/ClusterSettings.java
#	server/src/main/java/org/opensearch/gateway/ReplicaShardBatchAllocator.java
#	server/src/main/java/org/opensearch/indices/store/TransportNodesListShardStoreMetadataBatch.java
#	server/src/main/java/org/opensearch/node/Node.java
#	server/src/test/java/org/opensearch/gateway/ShardBatchCacheTests.java
#	test/framework/src/main/java/org/opensearch/test/gateway/TestGatewayAllocator.java
    @shiv0408
    shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    95cec33 View commit details
    Browse the repository at this point in the history

  44. Spotless apply 

    Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    3f103c1 View commit details
    Browse the repository at this point in the history

  45. Fix for missed merge conflict 

    Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    shiv0408 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    154d13b View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. Added changelog 

    Signed-off-by: Shivansh Arora <[email protected]>
    @shiv0408
    shiv0408 committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    080c3ba View commit details
    Browse the repository at this point in the history