Healthcheck: correctly label nodes removed from metrics

[Michal-Leszczynski]

Commits a75df89 and 3cdd20d added and started setting new labels (dc/rack) in healthcheck metrics.
Unfortunately, those labels weren't taken into consideration when removing metrics of removed clusters/nodes:

scylla-manager/pkg/service/healthcheck/runner.go

Lines 116 to 121 in cafa851

	hl := prometheus.Labels{
	clusterKey: clusterID.String(),
	hostKey: host,
	}
	r.metrics.status.Delete(hl)
	r.metrics.rtt.Delete(hl)

This resulted in not removing those metrics until SM restart.

This commit fixes this issue and brings a small refactor to the way in which healthcheck labels are applied, so that it's more difficult to make such mistake in the future.

Fixes #4017

[Michal-Leszczynski]

The fix was tested manually by decommissioning a node from a fresh testing cluster.

Before the fix:

After the fix:

Note that the metrics for the removed node will keep on showing -1 until it's removed from the config cache svc, which can take up to 5 minutes, but I guess that's not really a problem. Before the fix, no matter how long I waited, the metrics weren't removed at all. After the fix it's clearly visible that after some time the metrics were removed.

[Michal-Leszczynski]

@karol-kokoszka @VAveryanov8 not sure about automated integration testing.
It would probably require some bash scripts using docker commands - I don't think that we have the infrastructure for that in place and that it would be too much hassle to properly implement it.

[karol-kokoszka]

If there is a test for that in integration-tests, it would catch this bug before releasing scylla-manager with the bug.
But I agree it's too much effort to add it and to cover that scenario. Especially that we are not really testing metrics in e2e scenarios.