Describe read-only mode of ingesters. (#9132)

* Describe read-only mode of ingesters. Signed-off-by: Peter Štibraný <[email protected]> * Document new downscale endpoints in ingester. Signed-off-by: Peter Štibraný <[email protected]> * CHANGELOG entries. Signed-off-by: Peter Štibraný <[email protected]> * Fix link. Signed-off-by: Peter Štibraný <[email protected]> * make doc Signed-off-by: Peter Štibraný <[email protected]> * Add link to api endpoint. Signed-off-by: Peter Štibraný <[email protected]> * Fix link. Signed-off-by: Peter Štibraný <[email protected]> * Accept doc suggestions. Co-authored-by: Taylor C <[email protected]> * Fix typo. Signed-off-by: Peter Štibraný <[email protected]> --------- Signed-off-by: Peter Štibraný <[email protected]> Co-authored-by: Taylor C <[email protected]>
grafana · Sep 18, 2024 · 355dcb6 · 355dcb6
1 parent 796dd38
commit 355dcb6
Show file tree

Hide file tree

Showing 3 changed files with 52 additions and 0 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -220,6 +220,8 @@
 * [ENHANCEMENT] Provide more detailed recommendations on how to migrate from classic to native histograms. #8864
 * [ENHANCEMENT] Clarify that `{namespace}` and `{groupName}` path segments in the ruler config API should be URL-escaped. #8969
 * [ENHANCEMENT] Include stalled compactor network drive information in runbooks. #9297
+* [ENHANCEMENT] Document `/ingester/prepare-partition-downscale` and `/ingester/prepare-instance-ring-downscale` endpoints. #9132
+* [ENHANCEMENT] Describe read-only mode of ingesters in component documentation. #9132
 
 ### Tools
 

diff --git a/docs/sources/mimir/references/architecture/components/ingester.md b/docs/sources/mimir/references/architecture/components/ingester.md
@@ -97,3 +97,15 @@ For more information on shuffle sharding, refer to [Configuring shuffle sharding
 Out-of-order samples are discarded by default. If the system writing samples to Mimir produces out-of-order samples, you can enable ingestion of such samples.
 
 For more information about out-of-order samples ingestion, refer to [Configuring out of order samples ingestion]({{< relref "../../../configure/configure-out-of-order-samples-ingestion" >}}).
+
+## Read-only mode
+
+You can put ingesters in "read-only" mode through calling the [Prepare Instance Ring Downscale]({{< relref "../../http-api/index.md#prepare-instance-ring-downscale" >}}) API endpoint.
+Ingesters in read-only mode don't receive write requests, but can still receive read requests.
+Ingesters in read-only mode are not part of the shuffle shard for the write operation.
+
+Read-only mode is useful in downscaling scenarios and is a preparation for later shutdown of the ingester.
+
+Ingester ring states like JOINING, ACTIVE, or LEAVING are separate from read-only mode.
+An ingester that is both ACTIVE and read-only does not receive write requests.
+Read-only mode is stored in the hash ring as well.
diff --git a/docs/sources/mimir/references/http-api/index.md b/docs/sources/mimir/references/http-api/index.md
@@ -49,6 +49,8 @@ This document groups API endpoints by service. Note that the API endpoints are e
 | [Flush chunks / blocks](#flush-chunks--blocks) | Ingester | `GET,POST /ingester/flush` |
 | [Prepare for Shutdown](#prepare-for-shutdown) | Ingester | `GET,POST,DELETE /ingester/prepare-shutdown` |
 | [Shutdown](#shutdown) | Ingester | `POST /ingester/shutdown` |
+| [Prepare Partition Downscale](#prepare-partition-downscale) | Ingester | `GET,POST,DELETE /ingester/prepare-partition-downscale` |
+| [Prepare Instance Ring Downscale](#prepare-instance-ring-downscale) | Ingester | `GET,POST,DELETE /ingester/prepare-instance-ring-downscale` |
 | [Ingesters ring status](#ingesters-ring-status) | Distributor,Ingester | `GET /ingester/ring` |
 | [Ingester tenants](#ingester-tenants) | Ingester | `GET /ingester/tenants` |
 | [Ingester tenant TSDB](#ingester-tenant-tsdb) | Ingester | `GET /ingester/tsdb/{tenant}` |
@@ -420,6 +422,42 @@ This endpoint unregisters the ingester from the ring even if you disable `-inges
 
 This API endpoint is usually used by scale down automations.
 
+### Prepare partition downscale
+
+```
+GET,POST,DELETE /ingester/prepare-partition-downscale
+```
+
+This endpoint prepares the ingester's partition for downscaling by setting it to the `INACTIVE` state.
+
+A `GET` call to this endpoint returns a timestamp of when the partition was switched to the `INACTIVE` state, or 0, if the partition is not in the `INACTIVE` state.
+
+A `POST` call switches this ingester's partition to the `INACTIVE` state, if it isn't `INACTIVE` already, and returns the timestamp of when the switch to the `INACTIVE` state occurred.
+
+A `DELETE` call sets the partition back from the `INACTIVE` to the `ACTIVE` state.
+
+If the ingester is not configured to use ingest-storage, any call to this endpoint fails.
+
+This API endpoint is usually used by scale down automation, e.g. rollout-operator.
+
+### Prepare instance ring downscale
+
+```
+GET,POST,DELETE /ingester/prepare-instance-ring-downscale
+```
+
+This endpoint prepares the ingester for downscaling by setting it to read-only mode.
+
+A `GET` call to this endpoint returns a timestamp of when the ingester was switched to read-only mode, or 0, if the ingester is not in read-only mode.
+
+A `POST` call switches this ingester's partition to read-only mode, if it isn't read-only already, and returns the timestamp of when the switch to read-only mode occurred.
+
+A `DELETE` call sets the ingester back to read-write mode.
+
+If the ingester is configured to use ingest-storage, any call to this endpoint fails.
+
+This API endpoint is usually used by scale down automation, e.g. rollout-operator.
+
 ### Prepare for unregister
 
 ```