Why spread-minimizing tokens could not work?

[deniszh]

We're not using shuffle sharding yet, and I noticed quite severe memory disbalance in ingesters, so, we decided to migrate to spread-minimizing tokens. So, I did migration as described in https://grafana.com/docs/mimir/latest/configure/configure-spread-minimizing-tokens/ ... and nothing happens?
I still see same disbalance in cortex_ingester_memory_series (ignore pod name - only ingester deployed there)

Writers and distributors have following flags enabled:

 -ingester.ring.spread-minimizing-join-ring-in-order=true
 -ingester.ring.spread-minimizing-zones=zone-a,zone-b,zone-c
 -ingester.ring.token-generation-strategy=spread-minimizing
 -ingester.ring.tokens-file-path=

Can it be because I was not able to remove /data/tokens at last step because distroless image now?
On the other hand disbalance didn't go after migrating first zone, so, don't think it's related.

[deniszh]

Still don't know what was the issue, but after rerolling cluster issue is gone 🤷

[gespi1]

and yet the /data/tokens file was never removed?

[gespi1]

what do you mean specifically with rerolling cluster ?

[deniszh]

and yet the /data/tokens file was never removed?

No, I didn't touch pvc

what do you mean specifically with rerolling cluster ?

deleted all resources (through kubectl delete -f <path to manifests generated from helm>) and created them again.
please note that pvc were not affected.

[gespi1]

updated to spread-minimizing tokens on our dev environment and at first the number of series per ingester were not balanced but after a few hours they balanced themselves out.

[deniszh]

Yes, I also can't reproduce. But spreading-minimised tokens were definitely not working - now I see perfectly aligned percentages on ingester ring web ui, before it was not. So, let's assume it was some strange bug. 🤷