Latest weekly helm chart does not support spread minimizing token generation

[rknightion]

When using the latest weekly helm chart (but setting the tag to 2.12.0 rc1) and setting the appropriate (following the migration documentation):

tokens_file_path: ""
token_generation_strategy: spread-minimizing
spread_minimizing_zones: "1a,1b,1c"
spread_minimizing_join_ring_in_order: true

The ingesters do start up successfully and register to the ring, but the distributors fail to start up with
error validating config: invalid ingester config: zone is not valid

From a quick look it seems as if the change in ring configuration isn't making its way to the distributors so the zone is being seen as "" rather than the 3 zones I have?

Any ideas? Or is this expected until the helm chart has been updated?

[davidkohr]

My guess is that the issue is when you enable zone awareness for ingesters, it does this by injecting this runtime config to each ingester:

# ingesters in zone-a:
    - args:
        - '-ingester.ring.instance-availability-zone=zone-a'

# ingesters in zone-b:
    - args:
        - '-ingester.ring.instance-availability-zone=zone-b'

Because this is being dynamically handled by the chart, you aren't the one configuring this in structuredConfig. Because of this, all of your Mimir components, at least upon init, think that the value of ingester.ring.instance-availability-zone is empty, hence the error message zone is not valid (notice the 2 spaces after zone).

Again, this is my guess, and I would love someone to verify this.

To get around this, we had to use Kustomize to inject runtime configs to each zone to enable spread-min token generation. This feels very hacky, so I'm looking forward to this being fixed in the chart, unless there's already a better way.

[dimitarvdimitrov]

looks like a legit bug - opened #7932 #7822