Implementing pagination for _cat/shards #14641

gargharsh3134 · 2024-07-04T05:25:52Z

Description

This change is targeted towards implementing pagination for _cat/shards API. A new parameter, nextToken, has been introduced as both the query parameter as well as in the response body.

Functional Testing Details:

Started a 9 data nodes local cluster having 3 indices (each with 2 primaries and 3 replicas).

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_cluster/health?pretty"
{
  "cluster_name" : "gkharsh-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 9,
  "number_of_data_nodes" : 6,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 6,
  "active_shards" : 24,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

_cat/shards response


gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_cat/shards"                             
test-index2 0 r STARTED 0 208b 127.0.0.1 data6
test-index2 0 p STARTED 0 208b 127.0.0.1 data2
test-index2 0 r STARTED 0 208b 127.0.0.1 data4
test-index2 0 r STARTED 0 208b 127.0.0.1 data3
test-index2 1 p STARTED 0 208b 127.0.0.1 data1
test-index2 1 r STARTED 0 208b 127.0.0.1 data5
test-index2 1 r STARTED 0 208b 127.0.0.1 data4
test-index2 1 r STARTED 0 208b 127.0.0.1 data3
test-index  0 r STARTED 0 208b 127.0.0.1 data6
test-index  0 p STARTED 0 208b 127.0.0.1 data2
test-index  0 r STARTED 0 208b 127.0.0.1 data5
test-index  0 r STARTED 0 208b 127.0.0.1 data3
test-index  1 p STARTED 0 208b 127.0.0.1 data1
test-index  1 r STARTED 0 208b 127.0.0.1 data6
test-index  1 r STARTED 0 208b 127.0.0.1 data5
test-index  1 r STARTED 0 208b 127.0.0.1 data4
index4-test 0 p STARTED 0 208b 127.0.0.1 data1
index4-test 0 r STARTED 0 208b 127.0.0.1 data2
index4-test 0 r STARTED 0 208b 127.0.0.1 data5
index4-test 0 r STARTED 0 208b 127.0.0.1 data3
index4-test 1 p STARTED 0 208b 127.0.0.1 data1
index4-test 1 r STARTED 0 208b 127.0.0.1 data6
index4-test 1 r STARTED 0 208b 127.0.0.1 data2
index4-test 1 r STARTED 0 208b 127.0.0.1 data4

_list/shards with plain text response:


gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/shards"
index4-test 0 p STARTED 0 208b 127.0.0.1 data1
index4-test 0 r STARTED 0 208b 127.0.0.1 data2
index4-test 0 r STARTED 0 208b 127.0.0.1 data5
index4-test 0 r STARTED 0 208b 127.0.0.1 data3
index4-test 1 p STARTED 0 208b 127.0.0.1 data1
index4-test 1 r STARTED 0 208b 127.0.0.1 data6
index4-test 1 r STARTED 0 208b 127.0.0.1 data2
index4-test 1 r STARTED 0 208b 127.0.0.1 data4
test-index2 0 r STARTED 0 208b 127.0.0.1 data6
test-index2 0 p STARTED 0 208b 127.0.0.1 data2
test-index2 0 r STARTED 0 208b 127.0.0.1 data4
test-index2 0 r STARTED 0 208b 127.0.0.1 data3
test-index2 1 p STARTED 0 208b 127.0.0.1 data1
test-index2 1 r STARTED 0 208b 127.0.0.1 data5
test-index2 1 r STARTED 0 208b 127.0.0.1 data4
test-index2 1 r STARTED 0 208b 127.0.0.1 data3
test-index  0 r STARTED 0 208b 127.0.0.1 data6
test-index  0 p STARTED 0 208b 127.0.0.1 data2
test-index  0 r STARTED 0 208b 127.0.0.1 data5
test-index  0 r STARTED 0 208b 127.0.0.1 data3
test-index  1 p STARTED 0 208b 127.0.0.1 data1
test-index  1 r STARTED 0 208b 127.0.0.1 data6
test-index  1 r STARTED 0 208b 127.0.0.1 data5
test-index  1 r STARTED 0 208b 127.0.0.1 data4
next_token null

_list/shards with JSON response format:


gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/shards?format=json&pretty"
{
  "next_token" : null,
  "shards" : [
    {
      "index" : "index4-test",
      "shard" : "0",
      "prirep" : "p",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data1"
    },
    {
      "index" : "index4-test",
      "shard" : "0",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data2"
    },
    {
      "index" : "index4-test",
      "shard" : "0",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data5"
    },
    {
      "index" : "index4-test",
      "shard" : "0",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data3"
    },
    {
      "index" : "index4-test",
      "shard" : "1",
      "prirep" : "p",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data1"
    },
    {
      "index" : "index4-test",
      "shard" : "1",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data6"
    },
    {
      "index" : "index4-test",
      "shard" : "1",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data2"
    },
    {
      "index" : "index4-test",
      "shard" : "1",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data4"
    },
    {
      "index" : "test-index2",
      "shard" : "0",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data6"
    },
    {
      "index" : "test-index2",
      "shard" : "0",
      "prirep" : "p",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data2"
    },
    {
      "index" : "test-index2",
      "shard" : "0",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data4"
    },
    {
      "index" : "test-index2",
      "shard" : "0",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data3"
    },
    {
      "index" : "test-index2",
      "shard" : "1",
      "prirep" : "p",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data1"
    },
    {
      "index" : "test-index2",
      "shard" : "1",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data5"
    },
    {
      "index" : "test-index2",
      "shard" : "1",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data4"
    },
    {
      "index" : "test-index2",
      "shard" : "1",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data3"
    },
    {
      "index" : "test-index",
      "shard" : "0",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data6"
    },
    {
      "index" : "test-index",
      "shard" : "0",
      "prirep" : "p",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data2"
    },
    {
      "index" : "test-index",
      "shard" : "0",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data5"
    },
    {
      "index" : "test-index",
      "shard" : "0",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data3"
    },
    {
      "index" : "test-index",
      "shard" : "1",
      "prirep" : "p",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data1"
    },
    {
      "index" : "test-index",
      "shard" : "1",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data6"
    },
    {
      "index" : "test-index",
      "shard" : "1",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data5"
    },
    {
      "index" : "test-index",
      "shard" : "1",
      "prirep" : "r",
      "state" : "STARTED",
      "docs" : "0",
      "store" : "208b",
      "ip" : "127.0.0.1",
      "node" : "data4"
    }
  ]
}

Related Issues

Resolves #14257

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-07-04T05:34:52Z

❌ Gradle check result for 671bb92: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/indices/cluster/IndicesClusterStateService.java

server/src/main/java/org/opensearch/cluster/metadata/Metadata.java

github-actions · 2024-07-09T06:08:02Z

✅ Gradle check result for 549427a: SUCCESS

codecov · 2024-07-09T06:09:07Z

Codecov Report

Attention: Patch coverage is 79.21569% with 53 lines in your changes missing coverage. Please review.

Project coverage is 72.01%. Comparing base (7a58f5e) to head (10f82d2).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...admin/cluster/shards/TransportCatShardsAction.java	0.00%	20 Missing ⚠️
...g/opensearch/rest/action/cat/RestShardsAction.java	47.05%	9 Missing ⚠️
...a/org/opensearch/action/pagination/PageParams.java	75.00%	1 Missing and 5 partials ⚠️
...rch/action/pagination/IndexPaginationStrategy.java	78.26%	0 Missing and 5 partials ⚠️
...va/org/opensearch/action/pagination/PageToken.java	75.00%	1 Missing and 4 partials ⚠️
...action/admin/cluster/shards/CatShardsResponse.java	81.81%	2 Missing and 2 partials ⚠️
...rch/action/pagination/ShardPaginationStrategy.java	95.55%	1 Missing and 3 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14641      +/-   ##
============================================
+ Coverage     71.88%   72.01%   +0.12%     
- Complexity    64546    64714     +168     
============================================
  Files          5296     5298       +2     
  Lines        301754   301952     +198     
  Branches      43588    43627      +39     
============================================
+ Hits         216910   217439     +529     
+ Misses        67040    66723     -317     
+ Partials      17804    17790      -14

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

server/src/main/java/org/opensearch/rest/action/cat/RestTable.java

server/src/main/java/org/opensearch/rest/action/cat/RestShardsAction.java

dblock · 2024-07-10T15:24:01Z

Please make sure to queue up an update to the OpenAPI spec in https://github.com/opensearch-project/opensearch-api-specification with this.

opensearch-trigger-bot · 2024-08-26T15:21:53Z

This PR is stalled because it has been open for 30 days with no activity.

Signed-off-by: Harsh Garg <[email protected]>

github-actions · 2024-09-26T11:18:44Z

❌ Gradle check result for 437b87f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/action/admin/cluster/shards/CatShardsRequest.java

server/src/main/java/org/opensearch/rest/pagination/ShardPaginationStrategy.java

Signed-off-by: Harsh Garg <[email protected]>

github-actions · 2024-09-27T11:43:44Z

❌ Gradle check result for 6c855ce: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Harsh Garg <[email protected]>

github-actions · 2024-09-30T01:06:02Z

✅ Gradle check result for d723978: SUCCESS

Signed-off-by: gargharsh3134 <[email protected]>

github-actions · 2024-10-01T06:37:26Z

❌ Gradle check result for d51a282: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/action/pagination/ShardPaginationStrategy.java

server/src/main/java/org/opensearch/action/pagination/IndexPaginationStrategy.java

server/src/main/java/org/opensearch/action/pagination/ShardPaginationStrategy.java

server/src/main/java/org/opensearch/rest/action/cat/RestShardsAction.java

server/src/test/java/org/opensearch/action/pagination/ShardPaginationStrategyTests.java

Signed-off-by: Harsh Garg <[email protected]>

github-actions · 2024-10-03T07:14:42Z

❌ Gradle check result for e9993a1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Harsh Garg <[email protected]>

github-actions · 2024-10-03T13:17:06Z

✅ Gradle check result for ede0700: SUCCESS

server/src/main/java/org/opensearch/action/pagination/IndexPaginationStrategy.java

Signed-off-by: Harsh Garg <[email protected]>

github-actions · 2024-10-03T19:33:55Z

❌ Gradle check result for 757b837: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Harsh Garg <[email protected]>

github-actions · 2024-10-03T21:20:47Z

❕ Gradle check result for 4698858: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

server/src/main/java/org/opensearch/action/admin/cluster/shards/CatShardsResponse.java

server/src/main/java/org/opensearch/action/pagination/ShardPaginationStrategy.java

Signed-off-by: Harsh Garg <[email protected]>

github-actions · 2024-10-07T07:24:14Z

✅ Gradle check result for 10f82d2: SUCCESS

opensearch-trigger-bot · 2024-10-07T07:26:39Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-14641-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a81b868facc186728b3d35fd65a6d95621f1dd82
# Push it to GitHub
git push --set-upstream origin backport/backport-14641-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-14641-to-2.x.

vikasvb90 suggested changes Jul 5, 2024

View reviewed changes

server/src/main/java/org/opensearch/indices/cluster/IndicesClusterStateService.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/cluster/metadata/Metadata.java Outdated Show resolved Hide resolved

gargharsh3134 force-pushed the catShardsPaginationPR branch from 671bb92 to 549427a Compare July 9, 2024 05:18

github-actions bot added Cluster Manager enhancement Enhancement or improvement to existing feature or request labels Jul 9, 2024

gargharsh3134 changed the title ~~Implementing pagination for _cat/shards and metadata changes~~ Implementing pagination for _cat/shards Jul 9, 2024

vikasvb90 reviewed Jul 10, 2024

View reviewed changes

github-actions bot added the v2.16.0 Issues and PRs related to version 2.16.0 label Jul 11, 2024

rwali-aws added v2.17.0 and removed v2.16.0 Issues and PRs related to version 2.16.0 labels Jul 26, 2024

opensearch-trigger-bot bot added stalled Issues that have stalled and removed stalled Issues that have stalled labels Aug 26, 2024

gargharsh3134 force-pushed the catShardsPaginationPR branch from 549427a to 578d510 Compare September 26, 2024 10:55

Adding _list/shards API

437b87f

Signed-off-by: Harsh Garg <[email protected]>

gargharsh3134 force-pushed the catShardsPaginationPR branch from 578d510 to 437b87f Compare September 26, 2024 11:02

rajiv-kv reviewed Sep 26, 2024

View reviewed changes

Refactoring and bug fixes for ShardPaginationStrategy

6c855ce

Signed-off-by: Harsh Garg <[email protected]>

Harsh Garg added 2 commits September 30, 2024 05:37

Adding UTs for shard based pagination

cbc4c06

Signed-off-by: Harsh Garg <[email protected]>

Merge branch 'main' into catShardsPaginationPR

d723978

Signed-off-by: Harsh Garg <[email protected]>

gargharsh3134 marked this pull request as ready for review September 30, 2024 04:28

gargharsh3134 requested review from anasalkouz, andrross and ashking94 as code owners September 30, 2024 04:28

Merge branch 'main' into catShardsPaginationPR

d51a282

Signed-off-by: gargharsh3134 <[email protected]>

rajiv-kv reviewed Oct 1, 2024

View reviewed changes

opensearch-ci-bot mentioned this pull request Oct 1, 2024

[AUTOCUT] Gradle Check Flaky Test Report for SpecificClusterManagerNodesIT #15944

Open

Harsh Garg added 2 commits October 3, 2024 11:38

Refactoring tests

30ea52a

Signed-off-by: Harsh Garg <[email protected]>

Merge branch 'main' into catShardsPaginationPR

e9993a1

Signed-off-by: Harsh Garg <[email protected]>

Harsh Garg added 2 commits October 3, 2024 14:47

Merge branch 'main' into catShardsPaginationPR

fe6601c

Signed-off-by: Harsh Garg <[email protected]>

Changing computation of PageToken in ShardStrategy

ede0700

Signed-off-by: Harsh Garg <[email protected]>

rajiv-kv reviewed Oct 3, 2024

View reviewed changes

server/src/main/java/org/opensearch/action/pagination/IndexPaginationStrategy.java Outdated Show resolved Hide resolved

Separating metadata filtering for pagination strategies

757b837

Signed-off-by: Harsh Garg <[email protected]>

gargharsh3134 force-pushed the catShardsPaginationPR branch from 82786d2 to 757b837 Compare October 3, 2024 18:32

This was referenced Oct 3, 2024

[AUTOCUT] Gradle Check Flaky Test Report for MinimumClusterManagerNodesIT #14289

Open

[AUTOCUT] Gradle Check Flaky Test Report for SearchOnlyReplicaIT #15812

Open

Retry Build

4698858

Signed-off-by: Harsh Garg <[email protected]>

rajiv-kv approved these changes Oct 4, 2024

View reviewed changes

shwetathareja reviewed Oct 4, 2024

View reviewed changes

Harsh Garg added 2 commits October 7, 2024 11:33

Merge branch 'main' into catShardsPaginationPR

dd9df05

Signed-off-by: Harsh Garg <[email protected]>

Updating exception message in ShardPaginationStrategy

10f82d2

Signed-off-by: Harsh Garg <[email protected]>

shwetathareja approved these changes Oct 7, 2024

View reviewed changes

shwetathareja added backport 2.x Backport to 2.x branch and removed v2.17.0 labels Oct 7, 2024

shwetathareja merged commit a81b868 into opensearch-project:main Oct 7, 2024
38 of 40 checks passed

opensearch-trigger-bot bot added the backport-failed label Oct 7, 2024

opensearch-ci-bot mentioned this pull request Oct 8, 2024

[AUTOCUT] Gradle Check Flaky Test Report for SearchRestCancellationIT #14311

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementing pagination for _cat/shards #14641

Implementing pagination for _cat/shards #14641

gargharsh3134 commented Jul 4, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024

github-actions bot commented Jul 9, 2024

codecov bot commented Jul 9, 2024 •

edited

Loading

dblock commented Jul 10, 2024

opensearch-trigger-bot bot commented Aug 26, 2024

github-actions bot commented Sep 26, 2024

github-actions bot commented Sep 27, 2024

github-actions bot commented Sep 30, 2024

github-actions bot commented Oct 1, 2024

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 7, 2024

opensearch-trigger-bot bot commented Oct 7, 2024

Implementing pagination for _cat/shards #14641

Implementing pagination for _cat/shards #14641

Conversation

gargharsh3134 commented Jul 4, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Jul 4, 2024

github-actions bot commented Jul 9, 2024

codecov bot commented Jul 9, 2024 • edited Loading

Codecov Report

dblock commented Jul 10, 2024

opensearch-trigger-bot bot commented Aug 26, 2024

github-actions bot commented Sep 26, 2024

github-actions bot commented Sep 27, 2024

github-actions bot commented Sep 30, 2024

github-actions bot commented Oct 1, 2024

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 7, 2024

opensearch-trigger-bot bot commented Oct 7, 2024

gargharsh3134 commented Jul 4, 2024 •

edited

Loading

codecov bot commented Jul 9, 2024 •

edited

Loading