Add pool spread topology key feature

[sinhaashish]

PoolSpreadTopologyKey: "performance" - is a exclusion
1: "performance: Gold"
2: "performance: Silver"
3: "performance: Emerald"

[tiagolobocastro]

hmm I think there's another location where you need to do the spread check, when we add replicas to existing volume?
Which also kinda of reminds me that the BDD tests are not covering replica replacement nor replica count increase?

[tiagolobocastro]

Why did you remove the snapshot check?

[tiagolobocastro]

Suggested change

	let is_spread_label_same = match &context.volume.as_ref().topology {
	None => false,
	let Some(topology) = context.volume.topology.as_ref() else {
	return Ok(false);
	};

[tiagolobocastro]

again let Some else to reduce nesting..

[tiagolobocastro]

Can we not pass this information through the candidates type?

[tiagolobocastro]

Need to update the top of the file to include spread?

[dsharma-dc]

@sinhaashish What's the reason this PR is open? Is the change redundant or subsumed in any other merged PRs ?

[sinhaashish]

No poolSpreadTopologyKey is not feasible
The test case

# The labels to be applied to the pools.
###############################################################################################
#   Description           ||   Pool Name   ||         Label           ||        Node        ||
#==============================================================================================
#     "pool1" has         ||   node1pool1  ||     zone-us=us-west-1   ||     io-engine-1    ||
#       the label         ||   node2pool1  ||     zone-us=us-west-1   ||     io-engine-2    ||
#   "zone-us=us-west-1"   ||   node3pool1  ||     zone-us=us-west-1   ||     io-engine-3    ||
#==============================================================================================
#     "pool2" has        ||   node1pool2  ||     zone-ap=ap-south-1   ||     io-engine-1   ||
#      the label         ||   node2pool2  ||     zone-ap=ap-south-1  ||     io-engine-2    ||
#   "zone-ap=ap-south-1" ||   node3pool2  ||     zone-ap=ap-south-1  ||     io-engine-3    ||
#============================================================================================
#     "pool3" has         ||   node1pool3  ||     zone-eu=eu-west-3   ||     io-engine-1    ||
#      the label          ||   node2pool3  ||     zone-eu=eu-west-3   ||     io-engine-2    ||
#   "zone-eu=eu-west-3"   ||   node3pool3  ||     zone-eu=eu-west-3   ||     io-engine-3    ||
#===========================================================================================
#     "poolX" has        ||   node1poolX  ||     zone-eu=eu-west-4   ||     io-engine-1    ||
#      the label         ||               ||                         ||                    ||
#   "zone-eu=eu-west-X"  ||               ||                         ||                    ||
#===========================================================================================

For pool spread topology key as zone-eu :

In a 2 replica volume, if the first replica is placed on node1 or node2 on pool3 where pool label is zone-eu=eu-west-3 the second replica goes on node3 which has the label zone-eu=eu-west-X which is correct and the test case passes.
But if the first replica sits on node3pool3 which has label `zone-eu=eu-west-3the other replica doesn't have the pool to get provisioned because poolX is on node3 itself and the test case fails

There is a discrepancy .
This is for poolSpreadTopologyKey , but wont it happen for nodeSpreadTopolofyKey as the replica will pe placed on different nodes only.

Thus closing this PR