feat: Add a means to query sharding state more natively/directly

[JustinPihony]

Possibly an HTTP endpoint added to query shard allocations and statistics using the existing protocol for that. It may have limitations (it could return no shards allocated (thanks to a Future.sequence), which at least indicates a problem).

[patriknw]

This would be implemented in Akka Management. https://doc.akka.io/libraries/akka-management/current/cluster-http-management.html

[johanandren]

If the existing two endpoints are not enough, it would be good to describe in the issue what is missing and should be added:

• https://doc.akka.io/libraries/akka-management/current/cluster-http-management.html#get-cluster-shard-responses
• https://doc.akka.io/libraries/akka-management/current/cluster-http-management.html#get-cluster-shards-name-responses

[leviramsey]

The Future.sequence in the coordinator's handling of GetClusterShardingStats where it fans out asks to the shard region manifests as "no shard regions" (which indicates a problem, to be fair, though it could illuminate more): so perhaps adding an unresponsiveRegions field to ClusterShardingStats (and plumbing that into the schema for the HTTP responses) is low-hanging fruit here.

(ShardRegionStats's exposing scala.Int in it's Java API is also a minor wart).

[patriknw]

Fixing the ShardRegionStat Integer in #32780

[janikdotzel]

Current State

Currently we have the two endpoints johan mentioned.

Endpoint	Returns
GET /cluster/shards	List of shard type names (e.g., ["Orders", "Users"])
GET /cluster/shards/{name}	Shard IDs with entity counts per shard

What's Missing

• Shards that aren't responding are not exposed
  As Levi pointed out, the internal Akka API returns a failed field for shards that didn't respond, but this information is not propagated back to the public API.
• There is no cluster-wide query
  The current endpoint queries only the local ShardRegion. There's no way to see which node hosts which shards. This could be useful to check the load distribution or debug routing issues.
• There is no query that returns entity ID's
  You can see entity counts but not which entities are actually active where they are located. This could make debugging specific entity issues easier.

The internal Akka API already provides these infos, they are just not made public at the moment.

Proposal

1. Add a failed field to the existing endpoint:

GET /cluster/shards/{name}

{
  "regions": [{"shardId": "shard-001", "numEntities": 15}],
  "failed": ["shard-003"]
}

2. New endpoint for cluster-wide statistics:

GET /cluster/shards/{name}/stats using GetClusterShardingStats:

{
  "regions": {
    "akka://System@10.0.0.1:25520": {
      "shards": [{"shardId": "shard-001", "numEntities": 15}],
      "failed": []
    },
    "akka://System@10.0.0.2:25520": {
      "shards": [{"shardId": "shard-002", "numEntities": 23}],
      "failed": []
    }
  },
  "totalEntities": 38,
  "totalShards": 2
}

3. New endpoint for entity-level state:

GET /cluster/shards/{name}/state using GetShardRegionState:

{
  "shards": [
    {"shardId": "shard-001", "entityIds": ["order-123", "order-456"]}
  ],
  "failed": []
}

[janikdotzel]

As I was already down the Sharding rabbit hole and got the time, I went ahead implemented the proposal.
Please Let me know what you think.
If you like it I can push a PR, otherwise I can incorporate any changes.

[johanandren]

I think the first two sounds good, the last sounds problematic because in real use cases there can be hundreds of thousands of entity ids, I think that requires more thinking and making sure it is not something causing problems in itself, before it could be a feature we support (note the Intended for testing purpose on GetShardRegionState).

[janikdotzel]

Agreed. Maybe the first two changes are sufficient.

If information about a specific entity would be useful for debugging issues we could eventually reverse the search. Instead of finding all Entities for a Shard type name, we could lookup a specific entityId.

GET /cluster/shards/{name}/entity/{entityId}

{
  "entityId": "order-12345",
  "shardId": "shard-42",
  "node": "akka://System@10.0.0.2:25520",
  "active": true
}

1. Find the shard ID: I currently don't know how to get the shard by Id
2. Find the node location: Use GetClusterShardingStats to find which node hosts that shard
3. Active check: Query GetShardRegionState only on the relevant shard, not cluster-wide