Zone aware consistency

[LasseHels]

Hey!

Consider a zone aware Mimir setup with three zones: A, B and C.

In this example, it is my understanding that:

• A write request must succeed for two out of three ingesters in order for the overall request to be considered successful.
• Read requests reach out to two out of three zones and return the sum of results from both zones¹.

Assuming this to be correct, can the following happen?

1. A write request succeeds for zone A but fails for B and C.
2. The data ingested by zone A will now show up in queries at random, depending on whether the query happened to query zone A.

Footnotes

1. minimize_ingester_requests is set to true. ↩

[dimitarvdimitrov]

yes, this is a very possible scenario

The reason it's perhaps not as bad as it sounds is that if the client is Prometheus or Grafana Agent, then it will retry with the same sample again fairly quickly, so data will be complete again.