Presenting zero token nodes

[tzach]

scylladb/scylladb#19684 adds zero token nodes to ScyllaDB. In this project, we need to:

• Make sure zero token nodes do not break the current panels.
  For example, the average storage chart should not include zero token nodes.

• Visually present which nodes have tokens and which are not

First is more urgent;
Second, it needs some (subtle?) design.

[amnonh]

@tzach, we need a metric that shows such nodes. We can see nodes that have zero tablets, but I'm not sure if that's enough

[mykaul]

I'm not sure what's the value here (in monitoring) to show them uniquely. The value is perhaps for debugging, to see why they do not participate in any workload? If anything, we might consider how we can filter them out from some averages, etc.?

[amnonh]

Maybe in the nodes table should appear differently?

[tzach]

Its true that for most cases, and most of the time, a node is a node, but as a Scylla user, the impact of a data node down, or a zero-token down is different.

[mykaul]

Its true that for most cases, and most of the time, a node is a node, but as a Scylla user, the impact of a data node down, or a zero-token down is different.

But also in the metrics of anything that is averaged, such as load or whatnot at the cluster level.

[amnonh]

@patjed41 do we have some indications in the metrics for such a node?

[patjed41]

@patjed41 do we have some indications in the metrics for such a node?

I don't think we have a metric that could be used to distinguish zero-token nodes from token-owning nodes easily and precisely. We can add a new metric if needed. It could show the number of tokens or just be two-valued: 0 for zero-token nodes, 1 for token-owning nodes.

[amnonh]

If we add a metric we can make it a counter of the number of tokens, we can probably have one per node