diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index e47143a4d8f..b023881fae4 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -23,11 +23,6 @@ A clear and concise description of what you expected to happen. - Infrastructure: [e.g., Kubernetes, bare-metal, laptop] - Deployment tool: [e.g., helm, jsonnet] -**Storage Engine** - -- [ ] Blocks -- [ ] Chunks - **Additional Context** diff --git a/.github/issue_commands.json b/.github/issue_and_pr_commands.json similarity index 100% rename from .github/issue_commands.json rename to .github/issue_and_pr_commands.json diff --git a/.github/workflows-doc.md b/.github/workflows-doc.md deleted file mode 100644 index 6c7b1932ddc..00000000000 --- a/.github/workflows-doc.md +++ /dev/null @@ -1,22 +0,0 @@ -# GitHub Actions CI/CD - -The purpose of this workflow is to run all continuous integration (CI) and continuous deployment (CD) jobs when needed while respecting their internal dependencies. The continuous integration jobs serve to ensure new code passes linting, unit tests and integration tests before reaching the main branch. The continuous deployment jobs serve to deploy the latest version of the code to mimir and the website when merged with main. - -## Contributing - -If you wish to add a new CI or CD job, add it to the existing current test-build-deploy workflow and make sure it does not prevent any of the other jobs from passing. If you wish to change any of the build or testing images, update it in all sections are containers are often reused. If you wish to add an entirely new workflow, create a new yml file with separate triggers and filters. In all cases, clearly document any changes made to the workflows, images and dependencies below. - -## Test, Build and Deploy - -test-build-deploy.yml specifies a workflow that runs all Mimir continuous integration and continuous deployment jobs. The workflow is triggered on every pull request and commit to main, however the CD jobs only run when changes are merged onto main. The workflow combines both CI and CD jobs, because the CD jobs are dependent on artifacts produced the CI jobs. - -## Specific Jobs - -| Job | Description | Type | -| -------------- | ----------------------------------------------------------------------------------------------------------------------------- | ---- | -| lint | Runs linting and ensures vendor directory, protos and generated documentation are consistent. | CI | -| test | Runs units tests on Cassandra testing framework. | CI | -| integration | Runs integration tests after upgrading golang, pulling necessary docker images and downloading necessary module dependencies. | CI | -| build | Builds and saves an up-to-date Mimir image and website. | CI | -| deploy_website | Deploys the latest version of Mimir website to gh-pages branch. Triggered within workflow. | CD | -| deploy | Deploys the latest Mimir image. | CD | diff --git a/.github/workflows/issue_commands.yml b/.github/workflows/issue_and_pr_commands.yml similarity index 78% rename from .github/workflows/issue_commands.yml rename to .github/workflows/issue_and_pr_commands.yml index 57a05f677cc..cefd3762110 100644 --- a/.github/workflows/issue_commands.yml +++ b/.github/workflows/issue_and_pr_commands.yml @@ -1,7 +1,9 @@ -name: Run commands when issues are labeled +name: Run commands when issues or PRs are labeled on: issues: types: [labeled] + pull_request: + types: [labeled] jobs: main: runs-on: ubuntu-latest @@ -18,4 +20,4 @@ jobs: uses: ./actions/commands with: token: ${{secrets.GH_BOT_ACCESS_TOKEN}} - configPath: issue_commands \ No newline at end of file + configPath: issue_and_pr_commands \ No newline at end of file diff --git a/.github/workflows/test-build-deploy.yml b/.github/workflows/test-build-deploy.yml index 477376458a6..7c94c7b3f1f 100644 --- a/.github/workflows/test-build-deploy.yml +++ b/.github/workflows/test-build-deploy.yml @@ -19,7 +19,7 @@ jobs: lint: runs-on: ubuntu-20.04 container: - image: us.gcr.io/kubernetes-dev/mimir-build-image:import-jsonnet-readme-2418fd778-WIP + image: us.gcr.io/kubernetes-dev/mimir-build-image:remove-website-c9babc116 credentials: username: _json_key password: ${{ secrets.gcr_json_key }} @@ -55,7 +55,7 @@ jobs: lint-jsonnet: runs-on: ubuntu-20.04 container: - image: us.gcr.io/kubernetes-dev/mimir-build-image:import-jsonnet-readme-2418fd778-WIP + image: us.gcr.io/kubernetes-dev/mimir-build-image:remove-website-c9babc116 credentials: username: _json_key password: ${{ secrets.gcr_json_key }} @@ -85,7 +85,7 @@ jobs: test: runs-on: ubuntu-20.04 container: - image: us.gcr.io/kubernetes-dev/mimir-build-image:import-jsonnet-readme-2418fd778-WIP + image: us.gcr.io/kubernetes-dev/mimir-build-image:remove-website-c9babc116 credentials: username: _json_key password: ${{ secrets.gcr_json_key }} @@ -112,7 +112,7 @@ jobs: build-mimir: runs-on: ubuntu-20.04 container: - image: us.gcr.io/kubernetes-dev/mimir-build-image:import-jsonnet-readme-2418fd778-WIP + image: us.gcr.io/kubernetes-dev/mimir-build-image:remove-website-c9babc116 credentials: username: _json_key password: ${{ secrets.gcr_json_key }} @@ -144,7 +144,7 @@ jobs: build-tools: runs-on: ubuntu-20.04 container: - image: us.gcr.io/kubernetes-dev/mimir-build-image:import-jsonnet-readme-2418fd778-WIP + image: us.gcr.io/kubernetes-dev/mimir-build-image:remove-website-c9babc116 credentials: username: _json_key password: ${{ secrets.gcr_json_key }} @@ -160,7 +160,7 @@ jobs: - name: Build Images # Build everything except mimir (run by build-mimir job) and build image (not managed by CI). run: | - make list-image-targets | grep -v -E '/mimir-build-image/|/cmd/mimir/' | xargs -I {} make BUILD_IN_CONTAINER=false {} + make list-image-targets | grep -v -E '/mimir-build-image/|/cmd/mimir/|/mimir-mixin-tools/' | xargs -I {} make BUILD_IN_CONTAINER=false {} - name: Save Images run: | mkdir /tmp/images @@ -227,7 +227,7 @@ jobs: if: (startsWith(github.ref, 'refs/tags/') || startsWith(github.ref, 'refs/heads/r') ) && github.event_name == 'push' && github.repository == 'grafana/mimir' runs-on: ubuntu-20.04 container: - image: us.gcr.io/kubernetes-dev/mimir-build-image:import-jsonnet-readme-2418fd778-WIP + image: us.gcr.io/kubernetes-dev/mimir-build-image:remove-website-c9babc116 credentials: username: _json_key password: ${{ secrets.gcr_json_key }} @@ -261,8 +261,13 @@ jobs: if [ -n "$GCR_REGISTRY_PASSWORD" ]; then printenv GCR_REGISTRY_PASSWORD | docker login -u "$GCR_REGISTRY_USER" --password-stdin us.gcr.io fi + if [ -n "$DOCKER_PASSWORD" ]; then + printenv DOCKER_PASSWORD | docker login -u "$DOCKER_USERNAME" --password-stdin + fi export IMAGE_TAG=$(make image-tag) ./push-images env: GCR_REGISTRY_USER: _json_key GCR_REGISTRY_PASSWORD: ${{ secrets.gcr_json_key }} + DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }} + DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }} diff --git a/.gitignore b/.gitignore index af18ad724d9..875a642028a 100644 --- a/.gitignore +++ b/.gitignore @@ -2,14 +2,12 @@ cmd/cortextool/cortextool cmd/test-exporter/test-exporter cmd/mimir/mimir cmd/query-tee/query-tee -cmd/thanosconvert/thanosconvert +cmd/metaconvert/metaconvert .uptodate .pkg .cache +.swp docker-images/ -website/public -website/resources -website/content/en/docs e2e_integration_test* active-query-tracker dist/ diff --git a/.gitmodules b/.gitmodules deleted file mode 100644 index 4fa5bddda5f..00000000000 --- a/.gitmodules +++ /dev/null @@ -1,4 +0,0 @@ - -[submodule "themes/docsy"] - path = website/themes/docsy - url = https://github.com/google/docsy diff --git a/.golangci.yml b/.golangci.yml index a4fc782cf78..2920fe7beef 100644 --- a/.golangci.yml +++ b/.golangci.yml @@ -24,3 +24,4 @@ run: build-tags: - netgo - requires_docker + - requires_libpcap diff --git a/.lintignore b/.lintignore index 7ff53ab2751..0b1626a02f6 100644 --- a/.lintignore +++ b/.lintignore @@ -3,6 +3,3 @@ ./vendor* ./pkg/configs/legacy_promql* ./.pkg* -./website/themes* -./website/public* -./website/resources* diff --git a/ADOPTERS.md b/ADOPTERS.md deleted file mode 100644 index 5edfcabbe1f..00000000000 --- a/ADOPTERS.md +++ /dev/null @@ -1,19 +0,0 @@ -# Cortex Adopters - -This is the list of organisations that are using Cortex in **production environments** to power their metrics and monitoring systems. Please send PRs to add or remove organisations. - -- [Amazon Web Services (AWS)](https://aws.amazon.com/prometheus) -- [Aspen Mesh](https://aspenmesh.io/) -- [Buoyant](https://buoyant.io/) -- [DigitalOcean](https://www.digitalocean.com/) -- [Electronic Arts](https://www.ea.com/) -- [Etsy](https://www.etsy.com/) -- [EverQuote](https://everquote.com/) -- [GoJek](https://www.gojek.io/) -- [GrafanaLabs](https://grafana.com/) -- [MayaData](https://mayadata.io/) -- [Opstrace](https://opstrace.com/) -- [Platform9](https://platform9.com/) -- [REWE Digital](https://rewe-digital.com/) -- [SysEleven](https://www.syseleven.de/) -- [Weaveworks](https://www.weave.works/) diff --git a/CHANGELOG.md b/CHANGELOG.md index 756149d336d..f06087c04d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## Mimir - main / unreleased +* [CHANGE] Add two new metrics `cortex_ruler_list_rules_seconds` and `cortex_ruler_load_rule_groups_seconds` to the ruler. #906 +* [CHANGE] Remove `-alertmanager.configs.auto-webhook-root` #977 * [CHANGE] Removed deprecated limits for rejecting old samples #799 This removes the following flags: * `-validation.reject-old-samples` @@ -55,12 +57,13 @@ * CLI flag `-blocks-storage.tsdb.max-exemplars` is renamed to `-ingester.max-global-exemplars-per-user`. * YAML `max_exemplars` is moved from `tsdb` to `overrides` and renamed to `max_global_exemplars_per_user`. * [CHANGE] The metric `cortex_deprecated_flags_inuse_total` has been renamed to `deprecated_flags_inuse_total` as part of using grafana/dskit functionality. #185 +* [CHANGE] API: The `-api.response-compression-enabled` flag has been removed, and GZIP response compression is always enabled except on `/api/v1/push` and `/push` endpoints. #880 * [CHANGE] Alertmanager: Don't count user-not-found errors from replicas as failures in the `cortex_alertmanager_state_fetch_replica_state_failed_total` metric. #190 * [CHANGE] Alertmanager: Use distributor for non-API routes when sharding is enabled. #213 * [CHANGE] Query-frontend: added `sharded` label to `cortex_query_seconds_total` metric. #235 * [CHANGE] Query-frontend: changed the flag name for controlling query sharding total shards from `-querier.total-shards` to `-frontend.query-sharding-total-shards`. #230 * [CHANGE] Querier/ruler: Option `-querier.ingester-streaming` has been removed. Querier/ruler now always use streaming method to query ingesters. #204 -* [CHANGE] Limits: Option `-ingester.max-samples-per-query` is now deprecated. YAML field `max_samples_per_query` is no longer supported. It required `-querier.ingester-streaming` option to be set to false, but since `-querier.ingester-streaming` is removed (always defaulting to true), the limit using it was removed as well. #204 +* [CHANGE] Limits: Option `-ingester.max-samples-per-query` and its YAML field `max_samples_per_query` have been removed. It required `-querier.ingester-streaming` option to be set to false, but since `-querier.ingester-streaming` is removed (always defaulting to true), the limit using it was removed as well. #204 #1132 * [CHANGE] Compactor: removed the `cortex_compactor_group_vertical_compactions_total` metric. #278 * [CHANGE] Limits: Set the default max number of inflight ingester push requests (`-ingester.instance-limits.max-inflight-push-requests`) to 30000 in order to prevent clusters from being overwhelmed by request volume or temporary slow-downs. #259 * [CHANGE] Compactor no longer waits for initial blocks cleanup to finish before starting compactions. #282 @@ -143,7 +146,7 @@ * [CHANGE] Store-gateway: index cache now includes tenant in cache keys, this invalidates previous cached entries. #607 * [CHANGE] Removed the deprecated `-.fifocache.size` flag. #618 * [CHANGE] Ruler: removed the support for the deprecated storage configuration via `-ruler.storage.*` CLI flags (and their respective YAML config options). Use `-ruler-storage.*` instead. #628 -* [CHANGE] Querier: always fetch labels from store and respect start/end times in request; the option `-querier.query-store-for-labels-enabled` is now always on. #518 +* [CHANGE] Querier: always fetch labels from store and respect start/end times in request; the option `-querier.query-store-for-labels-enabled` has been removed and is now always on. #518 #1132 * [CHANGE] Query Frontend: range query response now omits the `data` field when it's empty (error case) like Prometheus does, previously it was `"data":{"resultType":"","result":null}`. #629 * [CHANGE] Query Frontend: instant queries now honor the `-frontend.max-retries-per-request` flag. #630 * [CHANGE] Alertmanager: removed `-alertmanager.storage.*` configuration options, with the exception of the CLI flags `-alertmanager.storage.path` and `-alertmanager.storage.retention`. Use `-alertmanager-storage.*` instead. #632 @@ -214,6 +217,84 @@ * `-query-frontend.cache-unaligned-requests` renamed to `-frontend.cache-unaligned-requests` * [CHANGE] Ruler: set new default limits for rule groups: `ruler.max_rules_per_rule_group` to 20 (previously 0, disabled) and `ruler.max_rule_groups_per_tenant` to 70 (previously 0, disabled). #847 * [CHANGE] Compactor is now included in `all` target (single-binary). #866 +* [CHANGE] Shuffle-sharding: + * `-distributor.sharding-strategy` option has been removed, and shuffle sharding is enabled by default. Default shard size is set to 0, which disables shuffle sharding for the tenant (all ingesters will receive tenants's samples). #888 + * `-ruler.sharding-strategy` option has been removed from ruler. Ruler now uses shuffle-sharding by default, but respects `ruler_tenant_shard_size`, which defaults to 0 (ie. use all rulers for tenant). #889 + * `-store-gateway.sharding-strategy` option has been removed store-gateways. Store-gateway now uses shuffle-sharding by default, but respects `store_gateway_tenant_shard_size` for tenant, and this value defaults to 0. #891 +* [CHANGE] Server: `-server.http-listen-port` (yaml: `server.http_listen_port`) now defaults to `8080` (previously `80`). #871 +* [CHANGE] Changed the default value of `blocks-storage.bucket-store.ignore-deletion-marks-delay` from 6h to 1h. #892 +* [CHANGE] Querier/ruler/query-frontend: the experimental `-querier.at-modifier-enabled` CLI flag has been removed and the PromQL `@` modifier is always enabled. #941 +* [CHANGE] Ruler: removed `-ruler.enable-sharding` option, and change default value of `-ruler.ring.store` to `memberlist`. #943 +* [CHANGE] Ruler: `-ruler.alertmanager-use-v2` has been removed. The ruler will always use the `v2` endpoints. #954 #1100 +* [CHANGE] Compactor: Removed `-compactor.sharding-enabled` option. Sharding in compactor is now always enabled. Default value of `-compactor.ring.store` has changed from `consul` to `memberlist`. Default value of `-compactor.ring.wait-stability-min-duration` is now 0, which disables the feature. #956 +* [CHANGE] Changed default settings for memcached clients: #959 #1000 + * The default value for the following config options has changed from `10000` to `25000`: + * `-blocks-storage.bucket-store.chunks-cache.memcached.max-async-buffer-size` + * `-blocks-storage.bucket-store.index-cache.memcached.max-async-buffer-size` + * `-blocks-storage.bucket-store.metadata-cache.memcached.max-async-buffer-size` + * `-frontend.results-cache.memcached.max-async-buffer-size` + * The default value for the following config options has changed from `0` (unlimited) to `100`: + * `-blocks-storage.bucket-store.chunks-cache.memcached.max-get-multi-batch-size` + * `-blocks-storage.bucket-store.index-cache.memcached.max-get-multi-batch-size` + * `-blocks-storage.bucket-store.metadata-cache.memcached.max-get-multi-batch-size` + * `-frontend.results-cache.memcached.max-get-multi-batch-size` + * The default value for the following config options has changed from `16` to `100`: + * `-blocks-storage.bucket-store.chunks-cache.memcached.max-idle-connections` + * `-blocks-storage.bucket-store.index-cache.memcached.max-idle-connections` + * `-blocks-storage.bucket-store.metadata-cache.memcached.max-idle-connections` + * `-frontend.results-cache.memcached.max-idle-connections` + * The default value for the following config options has changed from `100ms` to `200ms`: + * `-blocks-storage.bucket-store.metadata-cache.memcached.timeout` + * `-blocks-storage.bucket-store.index-cache.memcached.timeout` + * `-blocks-storage.bucket-store.chunks-cache.memcached.timeout` + * `-frontend.results-cache.memcached.timeout` +* [CHANGE] Querier: removed `-querier.worker-match-max-concurrent` and `-querier.worker-parallelism` CLI flags (and their respective YAML config options). Mimir now behaves like if `-querier.worker-match-max-concurrent` is always enabled and you should configure the max concurrency per querier process using `-querier.max-concurrent` instead. #958 +* [CHANGE] Distributor: change default value of `-distributor.instance-limits.max-inflight-push-requests` to `2000`. #964 +* [CHANGE] Distributor: change default value of `-distributor.remote-timeout` from `2s` to `20s`. #970 +* [CHANGE] Store-gateway: options `-store-gateway.sharding-enabled` and `-querier.store-gateway-addresses` were removed. Default value of `-store-gateway.sharding-ring.store` is now `memberlist` and default value for `-store-gateway.sharding-ring.wait-stability-min-duration` changed from `1m` to `0` (disabled). #976 +* [CHANGE] Ingester: change default value of `-ingester.final-sleep` from `30s` to `0s`. #981 +* [CHANGE] Ruler: `-experimental.ruler.enable-api` flag has been renamed to `-ruler.enable-api` and is now stable. The default value has also changed from `false` to `true`, so both ruler and alertmanager API are enabled by default. #913 #1065 +* [CHANGE] Alertmanager: `-experimental.alertmanager.enable-api` flag has been renamed to `-alertmanager.enable-api` and is now stable. #913 +* [CHANGE] Ingester: changed default value of `-blocks-storage.tsdb.retention-period` from `6h` to `24h`. #966 +* [CHANGE] Changed default value of `-querier.query-ingesters-within` and `-blocks-storage.tsdb.close-idle-tsdb-timeout` from `0` to `13h`. #966 +* [CHANGE] Changed default value of `-distributor.ring.store` (Distributor ring) and `-ring.store` (Ingester ring) to `memberlist`. #1046 +* [CHANGE] Ruler: `-ruler.alertmanager-discovery` flag has been removed. URLs following the prior SRV format, will be treated as a static target. To continue using service discovery for these URLs prepend `dnssrvnoa+` to them. #993 +* [CHANGE] Ruler: add support for [DNS service discovery format](./docs/sources/configuration/arguments.md#dns-service-discovery) for `-ruler.alertmanager-url`. #993 +* [CHANGE] Ruler: the following metrics for Alertmanager DNS service discovery are replaced: #993 + * `prometheus_sd_dns_lookups_total` replaced by `cortex_dns_lookups_total{component="ruler"}` + * `prometheus_sd_dns_lookup_failures_total` replaced by `cortex_dns_failures_total{component="ruler"}` +* [CHANGE] Memberlist: the `name` label on metrics `cortex_dns_failures_total`, `cortex_dns_lookups_total` and `cortex_dns_provider_results` was renamed to `component`. #993 +* [CHANGE] Changed the default value of `-blocks-storage.bucket-store.bucket-index.enabled` to `true`. The default configuration must now run the compactor in order to write the bucket index or else queries to long term storage will fail. #924 +* [CHANGE] Alertmanager: now always runs with sharding enabled; other modes of operation are removed. #1044 #1126 + * The following configuration options are removed: + * `-alertmanager.sharding-enabled` + * `-alertmanager.cluster.advertise-address` + * `-alertmanager.cluster.gossip-interval` + * `-alertmanager.cluster.listen-address` + * `-alertmanager.cluster.peers` + * `-alertmanager.cluster.push-pull-interval` + * The following configuration options are renamed: + * `-alertmanager.cluster.peer-timeout` to `-alertmanager.peer-timeout` +* [CHANGE] Alertmanager: default value of `-alertmanager.web.external-url` has changed from `http://localhost` to `http://localhost:8080/alertmanager`. #1067 +* [CHANGE] Option `-auth.enabled` has been renamed to `-auth.multitenancy-enabled`. #1130 +* [CHANGE] Default tenant ID used with disabled auth (`-auth.multitenancy-enabled=false`) has changed from `fake` to `anonymous`. This tenant ID can now be changed with `-auth.no-auth-tenant` option. #1063 +* [CHANGE] `thanosconvert` tool has been renamed to `metaconvert`. `-config.file` option has been removed, while it now requires `-tenant` option to work on single tenant only. It now also preserves labels recognized by Mimir. #1120 +* [CHANGE] The default values for the following local directories have changed: #1071 + * `-alertmanager.storage.path` default value changed to `./data-alertmanager/` + * `-compactor.data-dir` default value changed to `./data-compactor/` + * `-ruler.rule-path` default value changed to `./data-ruler/` +* [CHANGE] Query-frontend: the default value of `-frontend.split-queries-by-interval` has changed from `0` to `24h`. #1131 +* [CHANGE] Remove the test-exporter tool. #1133 +* [CHANGE] The default value for gRPC max send message size has been changed from 16MB to 100MB. This affects the following parameters: #1152 + * `-frontend.grpc-client-config.grpc-max-send-msg-size` + * `-ingester.client.grpc-max-send-msg-size` + * `-querier.frontend-client.grpc-max-send-msg-size` + * `-query-scheduler.grpc-client-config.grpc-max-send-msg-size` + * `-ruler.client.grpc-max-send-msg-size` +* [CHANGE] The following ingester low level settings have been removed: #1153 + * `-ingester-client.expected-labels` + * `-ingester-client.expected-samples-per-series` + * `-ingester-client.expected-timeseries` * [FEATURE] Query Frontend: Add `cortex_query_fetched_chunks_total` per-user counter to expose the number of chunks fetched as part of queries. This metric can be enabled with the `-frontend.query-stats-enabled` flag (or its respective YAML config option `query_stats_enabled`). #31 * [FEATURE] Query Frontend: Add experimental querysharding for the blocks storage (instant and range queries). You can now enable querysharding for blocks storage (`-store.engine=blocks`) by setting `-frontend.parallelize-shardable-queries` to `true`. The following additional config and exported metrics have been added. #79 #80 #100 #124 #140 #148 #150 #151 #153 #154 #155 #156 #157 #158 #159 #160 #163 #169 #172 #196 #205 #225 #226 #227 #228 #230 #235 #240 #239 #246 #244 #319 #330 #371 #385 #400 #458 #586 #630 #660 #707 * New config options: @@ -244,7 +325,7 @@ * The number of shards is adjusted to be compatible with number of compactor shards that are used by a split-and-merge compactor. The querier can use this to avoid querying blocks that cannot have series in a given query shard. * [FEATURE] PromQL: added `present_over_time` support. #139 * [FEATURE] Ingester: can expose metrics on active series matching custom trackers configured via `-ingester.active-series-custom-trackers` (or its respective YAML config option). When configured, active series for custom trackers are exposed by the `cortex_ingester_active_series_custom_tracker` metric. #42 #672 -* [FEATURE] Ingester: Enable snapshotting of in-memory TSDB on disk during shutdown via `-blocks-storage.tsdb.memory-snapshot-on-shutdown`. #249 +* [FEATURE] Ingester: Enable snapshotting of in-memory TSDB on disk during shutdown via `-blocks-storage.tsdb.memory-snapshot-on-shutdown` (experimental). #249 * [FEATURE] Compactor: compactor now uses new algorithm that we call "split-and-merge". Previous compaction strategy was removed. With the `split-and-merge` compactor source blocks for a given tenant are grouped into `-compactor.split-groups` number of groups. Each group of blocks is then compacted separately, and is split into `-compactor.split-and-merge-shards` shards (configurable on a per-tenant basis). Compaction of each tenant shards can be horizontally scaled. Number of compactors that work on jobs for single tenant can be limited by using `-compactor.compactor-tenant-shard-size` parameter, or per-tenant `compactor_tenant_shard_size` override. #275 #281 #282 #283 #288 #290 #303 #307 #317 #323 #324 #328 #353 #368 #479 #820 * [FEATURE] Querier: Added label names cardinality endpoint `/api/v1/cardinality/label_names` that is disabled by default. Can be enabled/disabled via the CLI flag `-querier.cardinality-analysis-enabled` or its respective YAML config option. Configurable on a per-tenant basis. #301 #377 #474 * [FEATURE] Distributor: Added `-api.skip-label-name-validation-header-enabled` option to allow skipping label name validation on the HTTP write path based on `X-Mimir-SkipLabelNameValidation` header being `true` or not. #390 @@ -255,12 +336,45 @@ * [FEATURE] Compactor: Added `-compactor.max-compaction-time` to control how long can compaction for a single tenant take. If compactions for a tenant take longer, no new compactions are started in the same compaction cycle. Running compactions are not stopped however, and may take much longer. #523 * [FEATURE] Compactor: When compactor finds blocks with out-of-order chunks, it will mark them for no-compaction. Blocks marked for no-compaction are ignored in future compactions too. Added metric `cortex_compactor_blocks_marked_for_no_compaction_total` to track number of blocks marked for no-compaction. Added `CortexCompactorSkippedBlocksWithOutOfOrderChunks` alert based on new metric. Markers are only checked from `/markers` location, but uploaded to the block directory too. #520 #535 #550 * [FEATURE] Compactor: multiple blocks are now downloaded and uploaded at once, which can shorten compaction process. #552 -* [FEATURE] Ingester: Added `-blocks-storage.tsdb.head-chunks-write-queue-size` flag, which allows setting the size of the queue used by the TSDB before m-mapping chunks. #591 +* [FEATURE] Ingester: Added `-blocks-storage.tsdb.head-chunks-write-queue-size` flag, which allows setting the size of the queue used by the TSDB before m-mapping chunks (experimental). #591 * Added `cortex_ingester_tsdb_mmap_chunk_write_queue_operations_total` metric to track different operations of this queue. * [FEATURE] Ruler: Added federated rule groups. #533 * Added `-ruler.tenant-federation.enabled` config flag. * Added support for `source_tenants` field on rule groups. -* [FEATURE] Mimir: Added "Activity tracker" feature which can log ongoing activities from previous Mimir run in case of the crash. It is enabled by default and controlled by the `-activity-tracker.filepath` flag. It can be disabled by setting this path to an empty string. Currently, the Store-gateway, Ruler, Querier, and Query-frontend components uses this feature. #631 #782 #822 +* [FEATURE] Mimir: Added "Activity tracker" feature which can log ongoing activities from previous Mimir run in case of a crash. It is enabled by default and controlled by the `-activity-tracker.filepath` flag. It can be disabled by setting this path to an empty string. Currently, the Store-gateway, Ruler, Querier, Query-frontend and Ingester components use this feature to track queries. #631 #782 #822 #1121 +* [FEATURE] Mimir: Divide configuration parameters into categories "basic", "advanced", and "experimental". Only flags in the basic category are shown when invoking `-help`, whereas `-help-all` will include flags in all categories (basic, advanced, experimental). #840 +* [FEATURE] Store-gateway: Added `/store-gateway/tenants` and `/store-gateway/tenant/{tenant}/blocks` endpoints that provide functionality that was provided by `tools/listblocks`. #911 +* [FEATURE] The following features have been moved from experimental to stable: #913 #1002 + * Alertmanager API + * Alertmanager receiver firewall + * Alertmanager sharding + * Azure blob storage support + * Blocks storage bucket index + * Disable the ring health check in the readiness endpoint (`-ingester.readiness-check-ring-health=false`) + * Distributor: do not extend writes on unhealthy ingesters + * Do not unregister ingesters from ring on shutdown (`-ingester.unregister-on-shutdown=false`) + * HA Tracker: cleanup of old replicas from KV Store + * Instance limits in ingester and distributor + * OpenStack Swift storage support + * Query-frontend: query stats tracking + * Query-scheduler + * Querier: tenant federation + * Ruler API + * S3 Server Side Encryption (SSE) using KMS + * TLS configuration for gRPC, HTTP and etcd clients + * Zone-aware replication + * `/labels` API using matchers (`-querier.query-label-names-with-matchers-enabled`) + * The following querier limits: + * `-querier.max-fetched-chunks-per-query` + * `-querier.max-fetched-chunk-bytes-per-query` + * `-querier.max-fetched-series-per-query` + * The following alertmanager limits: + * Notification rate (`-alertmanager.notification-rate-limit` and `-alertmanager.notification-rate-limit-per-integration`) + * Dispatcher groups (`-alertmanager.max-dispatcher-aggregation-groups`) + * User config size (`-alertmanager.max-config-size-bytes`) + * Templates count in user config (`-alertmanager.max-templates-count`) + * Max template size (`-alertmanager.max-template-size-bytes`) +* [FEATURE] Querier: Added support for tenant federation to exemplar endpoints. #927 * [ENHANCEMENT] Query-frontend: added `cortex_query_frontend_workers_enqueued_requests_total` metric to track the number of requests enqueued in each query-scheduler. #384 * [ENHANCEMENT] Add a flag (`--proxy.compare-use-relative-error`) in the query-tee to compare floating point values using relative error. #208 * [ENHANCEMENT] Add a flag (`--proxy.compare-skip-recent-samples`) in the query-tee to skip comparing recent samples. By default samples not older than 1 minute are skipped. #234 @@ -317,6 +431,8 @@ * [ENHANCEMENT] Store-gateway: wait for ring tokens stability instead of ring stability to speed up startup and tests #620 * [ENHANCEMENT] Query-scheduler: exported summary `cortex_query_scheduler_inflight_requests` tracking total number of inflight requests (both enqueued and processing) in percentile buckets. #675 * [ENHANCEMENT] Ingester: Expose ingester ring page on ingesters. #654 +* [ENHANCEMENT] Querier: retry store-gateway in case of unexpected failure, instead of failing the query. #1003 +* [ENHANCEMENT] Added a new metric `mimir_build_info` to coincide with `cortex_build_info`. #1022 * [BUGFIX] Frontend: Fixes @ modifier functions (start/end) when splitting queries by time. #206 * [BUGFIX] Fixes a panic in the query-tee when comparing result. #207 * [BUGFIX] Upgrade Prometheus. TSDB now waits for pending readers before truncating Head block, fixing the `chunk not found` error and preventing wrong query results. #16 @@ -335,6 +451,10 @@ * [BUGFIX] Querier: Disable query scheduler SRV DNS lookup. #689 * [BUGFIX] Query-frontend: fix API error messages that were mentioning Prometheus `--enable-feature=promql-negative-offset` and `--enable-feature=promql-at-modifier` flags. #688 * [BUGFIX] Query-frontend: worker's cancellation channels are now buffered to ensure that all request cancellations are properly handled. #741 +* [BUGFIX] Compactor: compactor should now be able to correctly mark blocks for deletion and no-compaction, if such marking was previously interrupted. #1015 +* [BUGFIX] Overrides-exporter: successfully startup even if runtime config is not set. #1056 +* [BUGFIX] Multi-KV: runtime config changes are now propagated to all rings, not just ingester ring. #1047 +* [BUGFIX] Ruler: do not log `unable to read rules directory` at startup if the directory hasn't been created yet. #1058 ### Mixin (changes since `grafana/cortex-jsonnet` `1.9.0`) @@ -391,8 +511,10 @@ * [CHANGE] Removed `cortex_overrides_metric` from `_config`. #871 * [CHANGE] Renamed recording rule groups (`cortex_` prefix changed to `mimir_`). #871 * [CHANGE] Alerts name prefix has been changed from `Cortex` to `Mimir` (eg. alert `CortexIngesterUnhealthy` has been renamed to `MimirIngesterUnhealthy`). #879 +* [CHANGE] Enabled resources dashboards by default. Can be disabled setting `resources_dashboards_enabled` config field to `false`. #920 * [FEATURE] Added `Cortex / Overrides` dashboard, displaying default limits and per-tenant overrides applied to Mimir. #673 * [FEATURE] Added `Mimir / Tenants` and `Mimir / Top tenants` dashboards, displaying user-based metrics. #776 +* [FEATURE] Added querier autoscaling panels and alerts. #1006 #1016 * [ENHANCEMENT] cortex-mixin: Make `cluster_namespace_deployment:kube_pod_container_resource_requests_{cpu_cores,memory_bytes}:sum` backwards compatible with `kube-state-metrics` v2.0.0. [#317](https://github.com/grafana/cortex-jsonnet/pull/317) * [ENHANCEMENT] Cortex-mixin: Include `cortex-gw-internal` naming variation in default `gateway` job names. [#328](https://github.com/grafana/cortex-jsonnet/pull/328) * [ENHANCEMENT] Ruler dashboard: added object storage metrics. [#354](https://github.com/grafana/cortex-jsonnet/pull/354) @@ -432,6 +554,7 @@ * [ENHANCEMENT] Added "Recover accidentally deleted blocks (Google Cloud specific)" playbook. #475 * [ENHANCEMENT] Added support to multi-zone store-gateway deployments. #608 #615 * [ENHANCEMENT] Show supplementary alertmanager services in the Rollout Progress dashboard. #738 #855 +* [ENHANCEMENT] Added `mimir` to default job names. This makes dashboards and alerts working when Mimir is installed in single-binary mode and the deployment is named `mimir`. #921 * [BUGFIX] Fixed `CortexIngesterHasNotShippedBlocks` alert false positive in case an ingester instance had ingested samples in the past, then no traffic was received for a long period and then it started receiving samples again. [#308](https://github.com/grafana/cortex-jsonnet/pull/308) * [BUGFIX] Fixed `CortexInconsistentRuntimeConfig` metric. [#335](https://github.com/grafana/cortex-jsonnet/pull/335) * [BUGFIX] Fixed scaling dashboard to correctly work when a Cortex service deployment spans across multiple zones (a zone is expected to have the `zone-[a-z]` suffix). [#365](https://github.com/grafana/cortex-jsonnet/pull/365) @@ -442,6 +565,7 @@ * [BUGFIX] Span the annotation.message in alerts as YAML multiline strings. [#412](https://github.com/grafana/cortex-jsonnet/pull/412) * [BUGFIX] Fixed "Instant queries / sec" in "Cortex / Reads" dashboard. #445 * [BUGFIX] Fixed and added missing KV store panels in Writes, Reads, Ruler and Compactor dashboards. #448 +* [BUGFIX] Fixed Alertmanager dashboard when alertmanager is running as part of single binary. #1064 ### Jsonnet (changes since `grafana/cortex-jsonnet` `1.9.0`) @@ -507,7 +631,58 @@ compactor_tenant_shard_size: 2, compactor_split_groups: 2, ``` +* [CHANGE] The entrypoint file to include has been renamed from `cortex.libsonnet` to `mimir.libsonnet`. #897 +* [CHANGE] The default image config field has been renamed from `cortex` to `mimir`. #896 + ``` + { + _images+:: { + mimir: '...', + }, + } + ``` +* [CHANGE] Removed `cortex_` prefix from config fields. #898 + * The following config fields have been renamed: + * `cortex_bucket_index_enabled` renamed to `bucket_index_enabled` + * `cortex_compactor_cleanup_interval` renamed to `compactor_cleanup_interval` + * `cortex_compactor_data_disk_class` renamed to `compactor_data_disk_class` + * `cortex_compactor_data_disk_size` renamed to `compactor_data_disk_size` + * `cortex_compactor_max_concurrency` renamed to `compactor_max_concurrency` + * `cortex_distributor_allow_multiple_replicas_on_same_node` renamed to `distributor_allow_multiple_replicas_on_same_node` + * `cortex_ingester_data_disk_class` renamed to `ingester_data_disk_class` + * `cortex_ingester_data_disk_size` renamed to `ingester_data_disk_size` + * `cortex_querier_allow_multiple_replicas_on_same_node` renamed to `querier_allow_multiple_replicas_on_same_node` + * `cortex_query_frontend_allow_multiple_replicas_on_same_node` renamed to `query_frontend_allow_multiple_replicas_on_same_node` + * `cortex_query_sharding_enabled` renamed to `query_sharding_enabled` + * `cortex_query_sharding_msg_size_factor` renamed to `query_sharding_msg_size_factor` + * `cortex_ruler_allow_multiple_replicas_on_same_node` renamed to `ruler_allow_multiple_replicas_on_same_node` + * `cortex_store_gateway_data_disk_class` renamed to `store_gateway_data_disk_class` + * `cortex_store_gateway_data_disk_size` renamed to `store_gateway_data_disk_size` +* [CHANGE] The overrides configmap default mountpoint has changed from `/etc/cortex` to `/etc/mimir`. It can be customized via the `overrides_configmap_mountpoint` config field. #899 +* [CHANGE] Enabled in the querier the features to query label names with matchers, PromQL at modifier and query long-term storage for labels. #905 +* [CHANGE] Reduced TSDB blocks retention on ingesters disk from 96h to 24h. #905 +* [CHANGE] Enabled closing of idle TSDB in ingesters. #905 +* [CHANGE] Disabled TSDB isolation in ingesters for better performances. #905 +* [CHANGE] Changed log level of querier, query-frontend, query-scheduler and alertmanager from `debug` to `info`. #905 +* [CHANGE] Enabled attributes in-memory cache in store-gateway. #905 +* [CHANGE] Configured store-gateway to not load blocks containing samples more recent than 10h (because such samples are queried from ingesters). #905 +* [CHANGE] Dynamically compute `-compactor.deletion-delay` based on other settings, in order to reduce the deletion delay as much as possible and lower the number of live blocks in the storage. #907 +* [CHANGE] The config field `distributorConfig` has been renamed to `ingesterRingClientConfig`. Config field `ringClient` has been removed in favor of `ingesterRingClientConfig`. #997 #1057 +* [CHANGE] Gossip.libsonnet has been fixed to modify all ring configurations, not only the ingester ring config. Furthermore it now supports migration via multi KV store. #1057 #1099 +* [CHANGE] Changed the default of `bucket_index_enabled` to `true`. #924 +* [CHANGE] Remove the support for the test-exporter. #1133 * [FEATURE] Added query sharding support. It can be enabled setting `cortex_query_sharding_enabled: true` in the `_config` object. #653 +* [FEATURE] Added shuffle-sharding support. It can be enabled and configured using the following config: #902 + ``` + _config+:: { + shuffle_sharding:: { + ingester_write_path_enabled: true, + ingester_read_path_enabled: true, + querier_enabled: true, + ruler_enabled: true, + store_gateway_enabled: true, + }, + } + ``` * [ENHANCEMENT] Add overrides config to compactor. This allows setting retention configs per user. [#386](https://github.com/grafana/cortex-jsonnet/pull/386) * [ENHANCEMENT] Added 256MB memory ballast to querier. [#369](https://github.com/grafana/cortex-jsonnet/pull/369) * [ENHANCEMENT] Update `etcd-operator` to latest version (see https://github.com/grafana/jsonnet-libs/pull/480). [#263](https://github.com/grafana/cortex-jsonnet/pull/263) @@ -524,6 +699,7 @@ * [BUGFIX] Fixed `-distributor.extend-writes` setting on ruler when `unregister_ingesters_on_shutdown` is disabled. [#369](https://github.com/grafana/cortex-jsonnet/pull/369) * [BUGFIX] Treat `compactor_blocks_retention_period` type as string rather than int.[#395](https://github.com/grafana/cortex-jsonnet/pull/395) * [BUGFIX] Pass `-ruler-storage.s3.endpoint` to ruler when using S3. [#421](https://github.com/grafana/cortex-jsonnet/pull/421) +* [BUGFIX] Remove service selector on label `gossip_ring_member` from other services than `gossip-ring`. [#1008](https://github.com/grafana/mimir/pull/1008) ### Query-tee diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 00000000000..3d4caa4f5cf --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,46 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to make participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation. + +## Our Standards + +Examples of behavior that contributes to creating a positive environment include: + +- Using welcoming and inclusive language +- Being respectful of differing viewpoints and experiences +- Gracefully accepting constructive criticism +- Focusing on what is best for the community +- Showing empathy towards other community members + +Examples of unacceptable behavior by participants include: + +- The use of sexualized language or imagery and unwelcome sexual attention or advances +- Trolling, insulting/derogatory comments, and personal or political attacks +- Public or private harassment +- Publishing others' private information, such as a physical or electronic address, without explicit permission +- Other conduct which could reasonably be considered inappropriate in a professional setting + +## Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at conduct@grafana.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version] + +[homepage]: http://contributor-covenant.org +[version]: http://contributor-covenant.org/version/1/4/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1186e1ce336..72070db44d2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,3 +1,3 @@ -# Contributing to Cortex +# Contributing to Grafana Mimir -See [https://cortexmetrics.io/docs/contributing/](https://cortexmetrics.io/docs/contributing/). +See [https://github.com/grafana/mimir/tree/main/docs/sources/contributing](https://github.com/grafana/mimir/tree/main/docs/sources/contributing). diff --git a/GOVERNANCE.md b/GOVERNANCE.md index e1df4f51a00..cb6ec41f94e 100644 --- a/GOVERNANCE.md +++ b/GOVERNANCE.md @@ -48,12 +48,23 @@ In case a member leaves, the [offboarding](#offboarding) procedure is applied. The current team members are: -- Bryan Boreham - [bboreham](https://github.com/bboreham) ([Grafana Labs](https://grafana.com/)) -- Goutham Veeramachaneni - [gouthamve](https://github.com/gouthamve) ([Grafana Labs](https://grafana.com/)) -- Jacob Lisi - [jtlisi](https://github.com/jtlisi) ([Grafana Labs](https://grafana.com/)) -- Marco Pracucci - [pracucci](https://github.com/pracucci) ([Grafana Labs](https://grafana.com/)) -- Peter Štibraný - [pstibrany](https://github.com/pstibrany) ([Grafana Labs](https://grafana.com/)) -- Tom Wilkie - [tomwilkie](https://github.com/tomwilkie) ([Grafana Labs](https://grafana.com/)) +- Arve Knudsen — [@aknuds1](https://github.com/aknuds1) ([Grafana Labs](https://grafana.com/)) +- Bryan Boreham — [@bboreham](https://github.com/bboreham) ([Grafana Labs](https://grafana.com/)) +- George Krajcsovits - [@krajorama](https://github.com/krajorama) ([Grafana Labs](https://grafana.com/)) +- Johanna Ratliff — [@johannaratliff](https://github.com/johannaratliff) ([Grafana Labs](https://grafana.com/)) +- Marco Pracucci — [@pracucci](https://github.com/pracucci) ([Grafana Labs](https://grafana.com/)) +- Mauro Stettler — [@replay](https://github.com/replay) ([Grafana Labs](https://grafana.com/)) +- Nick Pillitteri — [@56quarters](https://github.com/56quarters) ([Grafana Labs](https://grafana.com/)) +- Oleg Zaytsev — [@colega](https://github.com/colega) ([Grafana Labs](https://grafana.com/)) +- Peter Štibraný — [@pstibrany](https://github.com/pstibrany) ([Grafana Labs](https://grafana.com/)) +- Steve Simpson - [@stevesg](https://github.com/stevesg) ([Grafana Labs](https://grafana.com/)) +- Tyler Reid — [@treid314](https://github.com/treid314) ([Grafana Labs](https://grafana.com/)) + +Previous team members: + +- Goutham Veeramachaneni — [@gouthamve](https://github.com/gouthamve) +- Jacob Lisi — [@jtlisi](https://github.com/jtlisi) +- Tom Wilkie — [@tomwilkie](https://github.com/tomwilkie) ### Maintainers diff --git a/MAINTAINERS.md b/MAINTAINERS.md index 91c8aa9bdeb..9727a7ad031 100644 --- a/MAINTAINERS.md +++ b/MAINTAINERS.md @@ -1 +1,24 @@ -There are currently no default maintainer or codebase-specific maintainers. Please see the list of maintainers in our [governance](GOVERNANCE.md). +The following are the main/default maintainers: + +- Arve Knudsen — [@aknuds1](https://github.com/aknuds1) ([Grafana Labs](https://grafana.com/)) +- Bryan Boreham — [@bboreham](https://github.com/bboreham) ([Grafana Labs](https://grafana.com/)) +- George Krajcsovits - [@krajorama](https://github.com/krajorama) ([Grafana Labs](https://grafana.com/)) +- Johanna Ratliff — [@johannaratliff](https://github.com/johannaratliff) ([Grafana Labs](https://grafana.com/)) +- Marco Pracucci — [@pracucci](https://github.com/pracucci) ([Grafana Labs](https://grafana.com/)) +- Mauro Stettler — [@replay](https://github.com/replay) ([Grafana Labs](https://grafana.com/)) +- Nick Pillitteri — [@56quarters](https://github.com/56quarters) ([Grafana Labs](https://grafana.com/)) +- Oleg Zaytsev — [@colega](https://github.com/colega) ([Grafana Labs](https://grafana.com/)) +- Peter Štibraný — [@pstibrany](https://github.com/pstibrany) ([Grafana Labs](https://grafana.com/)) +- Steve Simpson - [@stevesg](https://github.com/stevesg) ([Grafana Labs](https://grafana.com/)) +- Tyler Reid — [@treid314](https://github.com/treid314) ([Grafana Labs](https://grafana.com/)) + +Some parts of the codebase have other maintainers, the package paths also include all sub-packages: + +- `docs`: + - Ursula Kallio — [@osg-grafana](https://github.com/osg-grafana) ([Grafana Labs](https://grafana.com/)) +- `pkg/ruler`: + - Josh Abreu Mesa — [@gotjosh](https://github.com/gotjosh) ([Grafana Labs](https://grafana.com/)) +- `pkg/alertmanager`: + - Josh Abreu Mesa — [@gotjosh](https://github.com/gotjosh) ([Grafana Labs](https://grafana.com/)) +- `pkg/frontend`: + - Cyril Tovena — [@cyriltovena](https://github.com/cyriltovena) ([Grafana Labs](https://grafana.com/)) # limited to reviews diff --git a/Makefile b/Makefile index 9f42e55bfee..90e49fa20c0 100644 --- a/Makefile +++ b/Makefile @@ -1,3 +1,5 @@ +SHELL = /usr/bin/env bash + # Adapted from https://www.thapaliya.com/en/writings/well-documented-makefiles/ .PHONY: help help: ## Display this help and any documented user-facing targets. Other undocumented targets may be present in the Makefile. @@ -8,7 +10,7 @@ help: # WARNING: do not commit to a repository! -include Makefile.local -.PHONY: all test test-with-race integration-tests cover clean images protos exes dist doc clean-doc check-doc push-multiarch-build-image license check-license format check-mixin check-mixin-jb check-mixin-mixtool checkin-mixin-playbook build-mixin format-mixin check-jsonnet-manifests format-jsonnet-manifests push-multiarch-mimir list-image-targets check-jsonnet-getting-started +.PHONY: all test test-with-race integration-tests cover clean images protos exes dist doc clean-doc check-doc push-multiarch-build-image license check-license format check-mixin check-mixin-jb check-mixin-mixtool checkin-mixin-playbook build-mixin format-mixin check-jsonnet-manifests format-jsonnet-manifests push-multiarch-mimir list-image-targets check-jsonnet-getting-started mixin-screenshots .DEFAULT_GOAL := all # Version number @@ -16,6 +18,8 @@ VERSION=$(shell cat "./VERSION" 2> /dev/null) GOOS ?= $(shell go env GOOS) GOARCH ?= $(shell go env GOARCH) +HOSTNAME := $(shell hostname) + # Don't export GOOS and GOARCH as environment variables. They get exported when passed via CLI options, # but that breaks tools ran via "go run". We use GOOS/GOARCH explicitly in places where needed. unexport GOOS @@ -112,7 +116,7 @@ push-multiarch-build-image: # We don't want find to scan inside a bunch of directories, to accelerate the # 'make: Entering directory '/go/src/github.com/grafana/mimir' phase. -DONT_FIND := -name vendor -prune -o -name .git -prune -o -name .cache -prune -o -name .pkg -prune -o -name packaging -prune -o +DONT_FIND := -name vendor -prune -o -name .git -prune -o -name .cache -prune -o -name .pkg -prune -o -name mimir-mixin-tools -prune -o MAKEFILES = $(shell find . $(DONT_FIND) \( -name 'Makefile' -o -name '*.mk' \) -print) @@ -162,19 +166,25 @@ all: $(UPTODATE_FILES) test: protos test-with-race: protos mod-check: protos -lint: lint-packaging-scripts protos +lint: protos mimir-build-image/$(UPTODATE): mimir-build-image/* # All the boiler plate for building golang follows: SUDO := $(shell docker info >/dev/null 2>&1 || echo "sudo -E") BUILD_IN_CONTAINER := true -LATEST_BUILD_IMAGE_TAG ?= import-jsonnet-readme-2418fd778-WIP +LATEST_BUILD_IMAGE_TAG ?= remove-website-c9babc116 # TTY is parameterized to allow Google Cloud Builder to run builds, # as it currently disallows TTY devices. This value needs to be overridden # in any custom cloudbuild.yaml files TTY := --tty -GO_FLAGS := -ldflags "-X main.Branch=$(GIT_BRANCH) -X main.Revision=$(GIT_REVISION) -X main.Version=$(VERSION) -extldflags \"-static\" -s -w" -tags netgo +MIMIR_VERSION := github.com/grafana/mimir/pkg/util/version + +GO_FLAGS := -ldflags "\ + -X $(MIMIR_VERSION).Branch=$(GIT_BRANCH) \ + -X $(MIMIR_VERSION).Revision=$(GIT_REVISION) \ + -X $(MIMIR_VERSION).Version=$(VERSION) \ + -extldflags \"-static\" -s -w" -tags netgo ifeq ($(BUILD_IN_CONTAINER),true) @@ -185,7 +195,7 @@ GOVOLUMES= -v $(shell pwd)/.cache:/go/cache:delegated,z \ # Mount local ssh credentials to be able to clone private repos when doing `mod-check` SSHVOLUME= -v ~/.ssh/:/root/.ssh:delegated,z -exes $(EXES) protos $(PROTO_GOS) lint lint-packaging-scripts test test-with-race cover shell mod-check check-protos web-build web-pre web-deploy doc format: mimir-build-image/$(UPTODATE) +exes $(EXES) protos $(PROTO_GOS) lint test test-with-race cover shell mod-check check-protos doc format: mimir-build-image/$(UPTODATE) @mkdir -p $(shell pwd)/.pkg @mkdir -p $(shell pwd)/.cache @echo @@ -206,10 +216,7 @@ protos: $(PROTO_GOS) @# to configure all such relative paths. protoc -I $(GOPATH)/src:./vendor/github.com/thanos-io/thanos/pkg:./vendor/github.com/gogo/protobuf:./vendor:./$(@D) --gogoslick_out=plugins=grpc,Mgoogle/protobuf/any.proto=github.com/gogo/protobuf/types,:./$(@D) ./$(patsubst %.pb.go,%.proto,$@) -lint-packaging-scripts: packaging/deb/control/postinst packaging/deb/control/prerm packaging/rpm/control/post packaging/rpm/control/preun - shellcheck $? - -lint: lint-packaging-scripts check-makefiles +lint: check-makefiles misspell -error docs/sources # Configured via .golangci.yml. @@ -298,25 +305,17 @@ mod-check: check-protos: clean-protos protos @git diff --exit-code -- $(PROTO_GOS) -web-pre: - cd website && git submodule update --init --recursive - ./tools/website/web-pre.sh - -web-build: web-pre - cd website && HUGO_ENV=production hugo --config config.toml --minify -v - -web-deploy: - ./tools/website/web-deploy.sh - -# Generates the config file documentation. +doc: ## Generates the config file documentation. doc: clean-doc - go run ./tools/doc-generator ./docs/sources/configuration/config-file-reference.template > ./docs/sources/configuration/config-file-reference.md - go run ./tools/doc-generator ./docs/sources/blocks-storage/compactor.template > ./docs/sources/blocks-storage/compactor.md - go run ./tools/doc-generator ./docs/sources/blocks-storage/store-gateway.template > ./docs/sources/blocks-storage/store-gateway.md - go run ./tools/doc-generator ./docs/sources/blocks-storage/querier.template > ./docs/sources/blocks-storage/querier.md - go run ./tools/doc-generator ./docs/sources/guides/encryption-at-rest.template > ./docs/sources/guides/encryption-at-rest.md - embedmd -w docs/sources/operations/requests-mirroring-to-secondary-cluster.md + go run ./tools/doc-generator ./docs/sources/configuration/reference-configuration-parameters.template > ./docs/sources/configuration/reference-configuration-parameters.md + go run ./tools/doc-generator ./docs/sources/architecture/compactor.template > ./docs/sources/architecture/compactor.md + go run ./tools/doc-generator ./docs/sources/architecture/store-gateway.template > ./docs/sources/architecture/store-gateway.md + go run ./tools/doc-generator ./docs/sources/architecture/querier.template > ./docs/sources/architecture/querier.md + go run ./tools/doc-generator ./docs/sources/operating-grafana-mimir/encrypt-data-at-rest.template > ./docs/sources/operating-grafana-mimir/encrypt-data-at-rest.md + embedmd -w docs/sources/configuration/using-the-query-frontend-with-prometheus.md + embedmd -w docs/sources/operating-grafana-mimir/mirror-requests-to-a-second-cluster.md embedmd -w docs/sources/guides/overrides-exporter.md + embedmd -w docs/sources/getting-started/_index.md embedmd -w operations/mimir/README.md # Make up markdown files prettier. When running with check-doc target, it will fail if this produces any change. @@ -324,7 +323,7 @@ doc: clean-doc # Add license header to files. license: - go run ./tools/add-license ./cmd ./integration ./pkg ./tools ./packaging ./development ./mimir-build-image + go run ./tools/add-license ./cmd ./integration ./pkg ./tools ./development ./mimir-build-image check-license: license @git diff --exit-code || (echo "Please add the license header running 'make BUILD_IN_CONTAINER=false license'" && false) @@ -338,7 +337,7 @@ check-makefiles: format-makefiles .PHONY: format-makefiles format-makefiles: ## Format all Makefiles. format-makefiles: $(MAKEFILES) - sed -i -e 's/^\(\t*\) /\1\t/g' -e 's/^\(\t*\) /\1/' -- $? + $(SED) -i -e 's/^\(\t*\) /\1\t/g' -e 's/^\(\t*\) /\1/' -- $? clean: $(SUDO) docker rmi $(IMAGE_NAMES) >/dev/null 2>&1 || true @@ -393,13 +392,20 @@ check-doc: doc ./docs/sources/configuration/config-file-reference.md \ ./docs/sources/blocks-storage/*.md \ ./docs/sources/configuration/*.md \ - ./docs/sources/operations/*.md \ + ./docs/sources/operating-grafana-mimir/*.md \ ./operations/mimir/*.md \ ./operations/mimir-mixin/docs/sources/*.md \ || (echo "Please update generated documentation by running 'make doc'" && false) +.PHONY: reference-help +reference-help: cmd/mimir/mimir + @(./cmd/mimir/mimir -h || true) > cmd/mimir/help.txt.tmpl + @$(SED) -i s/$(HOSTNAME)/\{\{.Hostname\}\}/g cmd/mimir/help.txt.tmpl + @(./cmd/mimir/mimir -help-all || true) > cmd/mimir/help-all.txt.tmpl + @$(SED) -i s/$(HOSTNAME)/\{\{.Hostname\}\}/g cmd/mimir/help-all.txt.tmpl + clean-white-noise: - @find . -path ./.pkg -prune -o -path ./vendor -prune -o -path ./website -prune -or -type f -name "*.md" -print | \ + @find . -path ./.pkg -prune -o -path ./vendor -prune -or -type f -name "*.md" -print | \ SED_BIN="$(SED)" xargs ./tools/cleanup-white-noise.sh check-white-noise: clean-white-noise @@ -426,13 +432,20 @@ check-mixin-playbook: build-mixin build-mixin: check-mixin-jb @rm -rf $(MIXIN_OUT_PATH) && mkdir $(MIXIN_OUT_PATH) - @mixtool generate all --output-alerts $(MIXIN_OUT_PATH)/alerts.yaml --output-rules $(MIXIN_OUT_PATH)/rules.yaml --directory $(MIXIN_OUT_PATH)/dashboards ${MIXIN_PATH}/mixin.libsonnet + @mixtool generate all --output-alerts $(MIXIN_OUT_PATH)/alerts.yaml --output-rules $(MIXIN_OUT_PATH)/rules.yaml --directory $(MIXIN_OUT_PATH)/dashboards ${MIXIN_PATH}/mixin-compiled.libsonnet @cd $(MIXIN_OUT_PATH)/.. && zip -q -r mimir-mixin.zip $$(basename "$(MIXIN_OUT_PATH)") @echo "The mixin has been compiled to $(MIXIN_OUT_PATH) and archived to $$(realpath --relative-to=$$(pwd) $(MIXIN_OUT_PATH)/../mimir-mixin.zip)" format-mixin: @find $(MIXIN_PATH) -type f -name '*.libsonnet' -print -o -name '*.jsonnet' -print | xargs jsonnetfmt -i +mixin-serve: ## Runs Grafana (listening on port 3000) loading the mixin dashboards compiled at operations/mimir-mixin-compiled. + @./operations/mimir-mixin-tools/serve/run.sh + +mixin-screenshots: ## Generates mixin dashboards screenshots. + @rm -f docs/sources/images/dashboards/*.png + @./operations/mimir-mixin-tools/screenshots/run.sh + check-jsonnet-manifests: format-jsonnet-manifests @echo "Checking diff:" @git diff --exit-code -- $(JSONNET_MANIFESTS_PATH) || (echo "Please format jsonnet manifests by running 'make format-jsonnet-manifests'" && false) @@ -459,100 +472,43 @@ check-jsonnet-tests: build-jsonnet-tests check-tsdb-blocks-storage-s3-docker-compose-yaml: cd development/tsdb-blocks-storage-s3 && make check -web-serve: - cd website && hugo --config config.toml --minify -v server - # Generate binaries for a Mimir release dist: dist/$(UPTODATE) dist/$(UPTODATE): rm -fr ./dist mkdir -p ./dist - for os in linux darwin; do \ - for arch in amd64 arm64; do \ + # Build binaries for various architectures and operating systems. Only + # mimirtool supports Windows for now. Also darwin/386 is not a valid + # architecture. + for os in linux darwin windows; do \ + for arch in 386 amd64 arm64; do \ + suffix="" ; \ + if [ "$$os" = "windows" ]; then \ + suffix=".exe" ; \ + fi; \ + if [ "$$os" = "darwin" ] && [ "$$arch" = "386" ]; then \ + continue; \ + fi; \ + echo "Building mimirtool for $$os/$$arch"; \ + GOOS=$$os GOARCH=$$arch CGO_ENABLED=0 go build $(GO_FLAGS) -o ./dist/mimirtool-$$os-$$arch$$suffix ./cmd/mimirtool; \ + sha256sum ./dist/mimirtool-$$os-$$arch$$suffix | cut -d ' ' -f 1 > ./dist/mimirtool-$$os-$$arch$$suffix-sha-256; \ + if [ "$$os" = "windows" ]; then \ + continue; \ + fi; \ echo "Building Mimir for $$os/$$arch"; \ - GOOS=$$os GOARCH=$$arch CGO_ENABLED=0 go build $(GO_FLAGS) -o ./dist/mimir-$$os-$$arch ./cmd/mimir; \ - sha256sum ./dist/mimir-$$os-$$arch | cut -d ' ' -f 1 > ./dist/mimir-$$os-$$arch-sha-256; \ + GOOS=$$os GOARCH=$$arch CGO_ENABLED=0 go build $(GO_FLAGS) -o ./dist/mimir-$$os-$$arch$$suffix ./cmd/mimir; \ + sha256sum ./dist/mimir-$$os-$$arch$$suffix | cut -d ' ' -f 1 > ./dist/mimir-$$os-$$arch$$suffix-sha-256; \ echo "Building query-tee for $$os/$$arch"; \ - GOOS=$$os GOARCH=$$arch CGO_ENABLED=0 go build $(GO_FLAGS) -o ./dist/query-tee-$$os-$$arch ./cmd/query-tee; \ - sha256sum ./dist/query-tee-$$os-$$arch | cut -d ' ' -f 1 > ./dist/query-tee-$$os-$$arch-sha-256; \ + GOOS=$$os GOARCH=$$arch CGO_ENABLED=0 go build $(GO_FLAGS) -o ./dist/query-tee-$$os-$$arch$$suffix ./cmd/query-tee; \ + sha256sum ./dist/query-tee-$$os-$$arch$$suffix | cut -d ' ' -f 1 > ./dist/query-tee-$$os-$$arch$$suffix-sha-256; \ done; \ done; \ touch $@ -# Generate packages for a Mimir release. -FPM_OPTS := fpm -s dir -v $(VERSION) -n mimir -f \ - --license "Apache 2.0" \ - --url "https://github.com/grafana/mimir" - -PACKAGE_IN_CONTAINER := true -PACKAGE_IMAGE ?= $(IMAGE_PREFIX)fpm -ifeq ($(PACKAGE_IN_CONTAINER), true) - -.PHONY: packages -packages: dist packaging/fpm/$(UPTODATE) - @mkdir -p $(shell pwd)/.pkg - @mkdir -p $(shell pwd)/.cache - @echo ">>>> Entering build container: $@" - $(SUDO) time docker run --rm $(TTY) \ - -v $(shell pwd):/src/github.com/grafana/mimir:delegated,z \ - -i $(PACKAGE_IMAGE) $@; - -else - -packages: dist/$(UPTODATE)-packages - -dist/$(UPTODATE)-packages: dist $(wildcard packaging/deb/**) $(wildcard packaging/rpm/**) - for arch in amd64 arm64; do \ - rpm_arch=x86_64; \ - deb_arch=x86_64; \ - if [ "$$arch" = "arm64" ]; then \ - rpm_arch=aarch64; \ - deb_arch=arm64; \ - fi; \ - $(FPM_OPTS) -t deb \ - --architecture $$deb_arch \ - --after-install packaging/deb/control/postinst \ - --before-remove packaging/deb/control/prerm \ - --package dist/mimir-$(VERSION)_$$arch.deb \ - dist/mimir-linux-$$arch=/usr/local/bin/mimir \ - docs/sources/chunks-storage/single-process-config.yaml=/etc/mimir/single-process-config.yaml \ - packaging/deb/default/mimir=/etc/default/mimir \ - packaging/deb/systemd/mimir.service=/etc/systemd/system/mimir.service; \ - $(FPM_OPTS) -t rpm \ - --architecture $$rpm_arch \ - --after-install packaging/rpm/control/post \ - --before-remove packaging/rpm/control/preun \ - --package dist/mimir-$(VERSION)_$$arch.rpm \ - dist/mimir-linux-$$arch=/usr/local/bin/mimir \ - docs/sources/chunks-storage/single-process-config.yaml=/etc/mimir/single-process-config.yaml \ - packaging/rpm/sysconfig/mimir=/etc/sysconfig/mimir \ - packaging/rpm/systemd/mimir.service=/etc/systemd/system/mimir.service; \ - done - for pkg in dist/*.deb dist/*.rpm; do \ - sha256sum $$pkg | cut -d ' ' -f 1 > $${pkg}-sha-256; \ - done; \ - touch $@ - -endif - integration-tests: cmd/mimir/$(UPTODATE) go test -tags=requires_docker ./integration/... -# Build both arm64 and amd64 images, so that we can test deb/rpm packages for both architectures. -packaging/rpm/centos-systemd/$(UPTODATE): packaging/rpm/centos-systemd/Dockerfile - $(SUDO) docker build --platform linux/amd64 --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) -t $(IMAGE_PREFIX)$(shell basename $(@D)):amd64 $(@D)/ - $(SUDO) docker build --platform linux/arm64 --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) -t $(IMAGE_PREFIX)$(shell basename $(@D)):arm64 $(@D)/ - touch $@ - -packaging/deb/debian-systemd/$(UPTODATE): packaging/deb/debian-systemd/Dockerfile - $(SUDO) docker build --platform linux/amd64 --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) -t $(IMAGE_PREFIX)$(shell basename $(@D)):amd64 $(@D)/ - $(SUDO) docker build --platform linux/arm64 --build-arg=revision=$(GIT_REVISION) --build-arg=goproxyValue=$(GOPROXY_VALUE) -t $(IMAGE_PREFIX)$(shell basename $(@D)):arm64 $(@D)/ - touch $@ - -.PHONY: test-packages -test-packages: packages packaging/rpm/centos-systemd/$(UPTODATE) packaging/deb/debian-systemd/$(UPTODATE) - ./tools/packaging/test-packages $(IMAGE_PREFIX) $(VERSION) - -include docs.mk +include docs/docs.mk +DOCS_DIR = docs/sources docs: doc diff --git a/RELEASE.md b/RELEASE.md index 7823faf40b8..850c1e933b5 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -94,7 +94,6 @@ To publish a release candidate: 1. Create a pre-release in GitHub - Write the release notes (including a copy-paste of the changelog) - Build binaries with `make dist` and attach them to the release - - Build packages with `make packages`, test them with `make test-packages` and attach them to the release ### Publish a stable release @@ -105,12 +104,15 @@ To publish a stable release: 1. Update the Cortex version in the following locations: - Kubernetes manifests located at `k8s/` - Documentation located at `docs/` +1. Update dashboard screenshots + 1. Run `make mixin-screenshots` + 1. Review all updated screenshots and ensure no sensitive data is disclosed + 1. Open a PR 1. After merging your PR to release branch, `git tag` the new release (see [How to tag a release](#how-to-tag-a-release)) from release branch. 1. Wait until CI pipeline succeeded (once a tag is created, the release process through GitHub actions will be triggered for this tag) 1. Create a release in GitHub - Write the release notes (including a copy-paste of the changelog) - Build binaries with `make dist` and attach them to the release - - Build packages with `make packages`, test them with `make test-packages` and attach them to the release 1. Merge the release branch `release-x.y` to `master` - Create `merge-release-X.Y-to-master` branch **from `release-X.Y` branch** locally - Merge upstream `master` branch into your `merge-release-X.Y-to-master` and resolve conflicts diff --git a/SECURITY.md b/SECURITY.md index ba07c5344e3..b8697c164d6 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,87 +1,20 @@ -# Cortex Security and Disclosure Information +# Reporting security issues -As with any complex system, it is certain that bugs will be found, some of them security-relevant. If you find a _security bug_ please report it privately to the maintainers listed in the MAINTAINERS of the relevant repository and CC cortex-team@googlegroups.com. We will fix the issue as soon as possible and coordinate a release date with you. You will be able to choose if you want public acknowledgement of your effort and if you want to be mentioned by name. +If you think you have found a security vulnerability, please send a report to [security@grafana.com](mailto:security@grafana.com). This address can be used for all of Grafana Labs's open source and commercial products (including but not limited to Grafana, Grafana Cloud, Grafana Enterprise, and grafana.com). We can accept only vulnerability reports at this address. -## Public Disclosure Timing +Please encrypt your message to us; please use our PGP key. The key fingerprint is: -The public disclosure date is agreed between the Cortex Team and the bug submitter. We prefer to fully disclose the bug as soon as possible, but only after a mitigation or fix is available. We will ask for delay if the bug or the fix is not yet fully understood or the solution is not tested to our standards yet. While there is no fixed timeframe for fix & disclosure, we will try our best to be quick and do not expect to need the usual 90 days most companies ask or. For a vulnerability with a straightforward mitigation, we expect report date to disclosure date to be on the order of 7 days. +F988 7BEA 027A 049F AE8E 5CAA D125 8932 BE24 C5CA ---- +The key is available from [keyserver.ubuntu.com](https://keyserver.ubuntu.com/pks/lookup?search=0xF9887BEA027A049FAE8E5CAAD1258932BE24C5CA&fingerprint=on&op=index). -## Private Vendors List +Grafana Labs will send you a response indicating the next steps in handling your report. After the initial reply to your report, the security team will keep you informed of the progress towards a fix and full announcement, and may ask for additional information or guidance. -We also have a list for vendors who provide Cortex to their users. -This list is not intended for individuals to find out about security issues. +**Important:** We ask you to not disclose the vulnerability before it have been fixed and announced, unless you received a response from the Grafana Labs security team that you can do so. -### Embargo Policy +## Security announcements -The information members receive on cortex-vendors-announce@googlegroups.com -must not be made public, shared, nor even hinted at anywhere -beyond the need-to-know within your specific team except with the list's -explicit approval. This holds true until the public disclosure date/time that was -agreed upon by the list. Members of the list and others may not use the information -for anything other than getting the issue fixed for your respective vendor’s users. +We maintain a category on the community site called [Security Announcements](https://community.grafana.com/c/support/security-announcements), +where we will post a summary, remediation, and mitigation details for any patch containing security fixes. -Before any information from the list is shared with respective members of your -team required to fix said issue, they must agree to the same terms and only -find out information on a need-to-know basis. - -In the unfortunate event you share the information beyond what is allowed by -this policy, you _must_ urgently inform the cortex-team@googlegroups.com -mailing list of exactly what information -leaked and to whom. A retrospective will take place after the leak so -we can assess how to not make the same mistake in the future. - -If you continue to leak information and break the policy outlined here, you -will be removed from the list. - -### Contributing Back - -This is a team effort. As a member of the list you must carry some water. This -could be in the form of the following: - -**Technical** - -- Review and/or test the proposed patches and point out potential issues with - them (such as incomplete fixes for the originally reported issues, additional - issues you might notice, and newly introduced bugs), and inform the list of the - work done even if no issues were encountered. - -**Administrative** - -- Help draft emails to the public disclosure mailing list. -- Help with release notes. - -### Membership - -Group membership is managed by the Cortex maintainers using Google Groups. - -### Membership Criteria - -To be eligible for the cortex-vendors-announce@googlegroups.com mailing list, your -company should: - -0. Have an actively monitored security email alias for our project. -1. Have a public hosted version/distribution of Cortex. -2. Have a user base not limited to your own organization. -3. Not be a downstream or rebuild of another vendor. -4. Be a participant and active contributor in the community. -5. Accept the [Embargo Policy](#embargo-policy) that is outlined above. -6. Be willing to [contribute back](#contributing-back) as outlined above. - -**Removal**: If your vendor stops meeting one or more of these criteria -after joining the list then you will be unsubscribed. - -### Request to Join - -Please open an issue to the repo with the details to prove that you're eligible based on -the above criteria. The team will review the application and will reach out to you in case -of any clarifications. Once the application is accepted you'll be added to the mailing-list -and also to the list below. - -### Members of the list - -Members of the list are listed below: - -- Cortex Maintainers -- AWS Security Announce (email private) +You can also subscribe to email updates to this category if you have a grafana.com account and sign on to the community site or track updates via an [RSS feed](https://community.grafana.com/c/support/security-announcements.rss). diff --git a/cmd/thanosconvert/Dockerfile b/cmd/metaconvert/Dockerfile similarity index 74% rename from cmd/thanosconvert/Dockerfile rename to cmd/metaconvert/Dockerfile index 35f1c550ef1..c27f4ba4f22 100644 --- a/cmd/thanosconvert/Dockerfile +++ b/cmd/metaconvert/Dockerfile @@ -5,10 +5,10 @@ FROM alpine:3.13 RUN apk add --no-cache ca-certificates -COPY thanosconvert / -ENTRYPOINT ["/thanosconvert"] +COPY metaconvert / +ENTRYPOINT ["/metaconvert"] ARG revision -LABEL org.opencontainers.image.title="thanosconvert" \ - org.opencontainers.image.source="https://github.com/grafana/mimir/tree/main/tools/thanosconvert" \ +LABEL org.opencontainers.image.title="metaconvert" \ + org.opencontainers.image.source="https://github.com/grafana/mimir/tree/main/cmd/metaconvert" \ org.opencontainers.image.revision="${revision}" diff --git a/cmd/metaconvert/main.go b/cmd/metaconvert/main.go new file mode 100644 index 00000000000..70c60afbdeb --- /dev/null +++ b/cmd/metaconvert/main.go @@ -0,0 +1,156 @@ +// SPDX-License-Identifier: AGPL-3.0-only +// Provenance-includes-location: https://github.com/cortexproject/cortex/blob/master/cmd/thanosconvert/main.go +// Provenance-includes-location: https://github.com/cortexproject/cortex/blob/master/tools/thanosconvert/thanosconvert.go +// Provenance-includes-license: Apache-2.0 +// Provenance-includes-copyright: The Cortex Authors. + +package main + +import ( + "bytes" + "context" + "flag" + "fmt" + "os" + "os/signal" + "path" + "syscall" + + gklog "github.com/go-kit/log" + "github.com/go-kit/log/level" + "github.com/pkg/errors" + "github.com/thanos-io/thanos/pkg/block" + "github.com/thanos-io/thanos/pkg/block/metadata" + "github.com/thanos-io/thanos/pkg/objstore" + "github.com/weaveworks/common/logging" + + "github.com/grafana/mimir/pkg/storage/bucket" + mimir_tsdb "github.com/grafana/mimir/pkg/storage/tsdb" + "github.com/grafana/mimir/pkg/util/log" +) + +type config struct { + LogLevel logging.Level + LogFormat logging.Format + BucketConfig bucket.Config + DryRun bool + Tenant string +} + +func main() { + cfg := config{} + + cfg.LogLevel.RegisterFlags(flag.CommandLine) + cfg.LogFormat.RegisterFlags(flag.CommandLine) + cfg.BucketConfig.RegisterFlags(flag.CommandLine) + + flag.BoolVar(&cfg.DryRun, "dry-run", false, "Don't make changes; only report what needs to be done") + flag.StringVar(&cfg.Tenant, "tenant", "", "Tenant to process") + flag.Usage = func() { + fmt.Fprintln(flag.CommandLine.Output(), os.Args[0], "%s is a tool to update meta.json files to conform to Mimir requirements.") + fmt.Fprintln(flag.CommandLine.Output(), "Flags:") + flag.PrintDefaults() + } + flag.Parse() + + logger := log.NewDefaultLogger(cfg.LogLevel, cfg.LogFormat) + + if cfg.Tenant == "" { + exitWithMessage("Use -tenant parameter to specify tenant, or -h to get list of available options.") + } + + if err := cfg.BucketConfig.Validate(); err != nil { + exitWithMessage("Bucket config is invalid: %v", err) + } + + ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT) + defer cancel() + + bkt, err := bucket.NewClient(ctx, cfg.BucketConfig, "metaconvert", logger, nil) + if err != nil { + exitWithMessage("Failed to create bucket client: %v", err) + } + + userBucketClient := bucket.NewUserBucketClient(cfg.Tenant, bkt, nil) + err = convertTenantBlocks(ctx, userBucketClient, cfg.Tenant, cfg.DryRun, logger) + if err != nil { + exitWithMessage("Update of meta.json files failed: %v", err) + } +} + +func exitWithMessage(msg string, args ...interface{}) { + fmt.Fprintf(os.Stderr, msg+"\n", args...) + os.Exit(1) +} + +func convertTenantBlocks(ctx context.Context, userBucketClient objstore.Bucket, tenant string, dryRun bool, logger gklog.Logger) error { + logger = gklog.With(logger, "tenant", tenant) + + return userBucketClient.Iter(ctx, "", func(dir string) error { + blockID, ok := block.IsBlockDir(dir) + if !ok { + // not a block + return nil + } + + // retrieve meta.json + meta, err := block.DownloadMeta(ctx, logger, userBucketClient, blockID) + if err != nil { + level.Error(logger).Log("msg", "failed to download meta.json file for block", "block", blockID.String(), "err", err.Error()) + return nil + } + + if meta.Thanos.Labels == nil { + meta.Thanos.Labels = map[string]string{} + } + + updated := false + + metaOrgID := meta.Thanos.Labels[mimir_tsdb.TenantIDExternalLabel] + if metaOrgID != tenant { + level.Warn(logger).Log("msg", "updating tenant label", "block", blockID.String(), "old_value", metaOrgID, "new_value", tenant) + updated = true + meta.Thanos.Labels[mimir_tsdb.TenantIDExternalLabel] = tenant + } + + for l, v := range meta.Thanos.Labels { + switch l { + case mimir_tsdb.TenantIDExternalLabel, mimir_tsdb.IngesterIDExternalLabel, mimir_tsdb.CompactorShardIDExternalLabel: + continue + } + + level.Warn(logger).Log("msg", "removing unknown label", "block", blockID.String(), "label", l, "value", v) + updated = true + delete(meta.Thanos.Labels, l) + } + + if !updated { + level.Info(logger).Log("msg", "no changes required", "block", blockID.String()) + return nil + } + + if dryRun { + level.Warn(logger).Log("msg", "changes required, not uploading back due to dry run", "block", blockID.String()) + return nil + } + + // convert and upload if appropriate + level.Info(logger).Log("msg", "changes required, uploading meta.json file", "block", blockID.String()) + + if err := uploadMetadata(ctx, userBucketClient, meta, path.Join(blockID.String(), block.MetaFilename)); err != nil { + return errors.Wrapf(err, "failed to upload meta.json for block %s", blockID.String()) + } + + level.Info(logger).Log("msg", "meta.json file uploaded successfully", "block", blockID.String()) + return nil + }) +} + +func uploadMetadata(ctx context.Context, bkt objstore.Bucket, meta metadata.Meta, path string) error { + var body bytes.Buffer + if err := meta.Write(&body); err != nil { + return errors.Wrap(err, "encode meta.json") + } + + return bkt.Upload(ctx, path, &body) +} diff --git a/cmd/metaconvert/main_test.go b/cmd/metaconvert/main_test.go new file mode 100644 index 00000000000..29b8b0babbc --- /dev/null +++ b/cmd/metaconvert/main_test.go @@ -0,0 +1,306 @@ +// SPDX-License-Identifier: AGPL-3.0-only + +package main + +import ( + "context" + "fmt" + "path" + "strings" + "testing" + + "github.com/go-kit/log" + "github.com/grafana/dskit/concurrency" + "github.com/oklog/ulid" + "github.com/prometheus/prometheus/tsdb" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/thanos-io/thanos/pkg/block" + "github.com/thanos-io/thanos/pkg/block/metadata" + + "github.com/grafana/mimir/pkg/storage/bucket/filesystem" + mimir_tsdb "github.com/grafana/mimir/pkg/storage/tsdb" +) + +func TestConvertTenantBlocks(t *testing.T) { + dir := t.TempDir() + bkt, err := filesystem.NewBucketClient(filesystem.Config{Directory: dir}) + require.NoError(t, err) + + ctx := context.Background() + + blockWithNoLabelsButManyOtherFields := ulid.MustNew(1, nil) + blockWithWrongTenant := ulid.MustNew(2, nil) + blockWithManyMimirLabels := ulid.MustNew(3, nil) + blockWithNoChangesRequired := ulid.MustNew(4, nil) + + const tenant = "target_tenant" + + inputMetas := map[ulid.ULID]metadata.Meta{ + blockWithNoLabelsButManyOtherFields: { + BlockMeta: tsdb.BlockMeta{ + ULID: blockWithNoLabelsButManyOtherFields, + + MinTime: 100, + MaxTime: 200, + Version: 1, + + Compaction: tsdb.BlockMetaCompaction{ + Level: 5, + Sources: []ulid.ULID{blockWithNoLabelsButManyOtherFields}, + }, + }, + + Thanos: metadata.Thanos{ + Version: 10, + Downsample: metadata.ThanosDownsample{ + Resolution: 15, + }, + Source: "ingester", + }, + }, + + blockWithWrongTenant: { + BlockMeta: tsdb.BlockMeta{ + ULID: blockWithWrongTenant, + }, + + Thanos: metadata.Thanos{ + Labels: map[string]string{ + "test": "label", + mimir_tsdb.TenantIDExternalLabel: "wrong tenant", + }, + }, + }, + + blockWithManyMimirLabels: { + BlockMeta: tsdb.BlockMeta{ + ULID: blockWithManyMimirLabels, + }, + + Thanos: metadata.Thanos{ + Labels: map[string]string{ + mimir_tsdb.TenantIDExternalLabel: "fake", + mimir_tsdb.CompactorShardIDExternalLabel: "1_of_10", + mimir_tsdb.IngesterIDExternalLabel: "ingester-1", + }, + }, + }, + + blockWithNoChangesRequired: { + BlockMeta: tsdb.BlockMeta{ + ULID: blockWithNoChangesRequired, + }, + + Thanos: metadata.Thanos{ + Labels: map[string]string{ + mimir_tsdb.TenantIDExternalLabel: tenant, + }, + }, + }, + } + + for b, m := range inputMetas { + require.NoError(t, uploadMetadata(ctx, bkt, m, path.Join(b.String(), metadata.MetaFilename))) + } + + logs := &concurrency.SyncBuffer{} + logger := log.NewLogfmtLogger(logs) + + // Run conversion + assert.NoError(t, convertTenantBlocks(ctx, bkt, tenant, false, logger)) + + expected := map[ulid.ULID]metadata.Meta{ + blockWithNoLabelsButManyOtherFields: { + BlockMeta: tsdb.BlockMeta{ + ULID: blockWithNoLabelsButManyOtherFields, + + MinTime: 100, + MaxTime: 200, + Version: 1, + + Compaction: tsdb.BlockMetaCompaction{ + Level: 5, + Sources: []ulid.ULID{blockWithNoLabelsButManyOtherFields}, + }, + }, + + Thanos: metadata.Thanos{ + Version: 10, + Labels: map[string]string{ + mimir_tsdb.TenantIDExternalLabel: tenant, + }, + Downsample: metadata.ThanosDownsample{ + Resolution: 15, + }, + Source: "ingester", + }, + }, + + blockWithWrongTenant: { + BlockMeta: tsdb.BlockMeta{ + ULID: blockWithWrongTenant, + }, + + Thanos: metadata.Thanos{ + Labels: map[string]string{ + mimir_tsdb.TenantIDExternalLabel: tenant, + }, + }, + }, + + blockWithManyMimirLabels: { + BlockMeta: tsdb.BlockMeta{ + ULID: blockWithManyMimirLabels, + }, + + Thanos: metadata.Thanos{ + Labels: map[string]string{ + mimir_tsdb.TenantIDExternalLabel: tenant, + mimir_tsdb.CompactorShardIDExternalLabel: "1_of_10", + mimir_tsdb.IngesterIDExternalLabel: "ingester-1", + }, + }, + }, + + blockWithNoChangesRequired: { + BlockMeta: tsdb.BlockMeta{ + ULID: blockWithNoChangesRequired, + }, + + Thanos: metadata.Thanos{ + Labels: map[string]string{ + mimir_tsdb.TenantIDExternalLabel: tenant, + }, + }, + }, + } + + for b, m := range expected { + meta, err := block.DownloadMeta(ctx, logger, bkt, b) + require.NoError(t, err) + require.Equal(t, m, meta) + } + + assert.Equal(t, []string{ + `level=warn tenant=target_tenant msg="updating tenant label" block=00000000010000000000000000 old_value= new_value=target_tenant`, + `level=info tenant=target_tenant msg="changes required, uploading meta.json file" block=00000000010000000000000000`, + `level=info tenant=target_tenant msg="meta.json file uploaded successfully" block=00000000010000000000000000`, + `level=warn tenant=target_tenant msg="updating tenant label" block=00000000020000000000000000 old_value="wrong tenant" new_value=target_tenant`, + `level=warn tenant=target_tenant msg="removing unknown label" block=00000000020000000000000000 label=test value=label`, + `level=info tenant=target_tenant msg="changes required, uploading meta.json file" block=00000000020000000000000000`, + `level=info tenant=target_tenant msg="meta.json file uploaded successfully" block=00000000020000000000000000`, + `level=warn tenant=target_tenant msg="updating tenant label" block=00000000030000000000000000 old_value=fake new_value=target_tenant`, + `level=info tenant=target_tenant msg="changes required, uploading meta.json file" block=00000000030000000000000000`, + `level=info tenant=target_tenant msg="meta.json file uploaded successfully" block=00000000030000000000000000`, + `level=info tenant=target_tenant msg="no changes required" block=00000000040000000000000000`, + }, strings.Split(strings.TrimSpace(logs.String()), "\n")) +} + +func TestConvertTenantBlocksDryMode(t *testing.T) { + dir := t.TempDir() + bkt, err := filesystem.NewBucketClient(filesystem.Config{Directory: dir}) + require.NoError(t, err) + + ctx := context.Background() + + blockWithNoLabelsButManyOtherFields := ulid.MustNew(1, nil) + blockWithWrongTenant := ulid.MustNew(2, nil) + blockWithManyMimirLabels := ulid.MustNew(3, nil) + blockWithNoChangesRequired := ulid.MustNew(4, nil) + + const tenant = "target_tenant" + + inputMetas := map[ulid.ULID]metadata.Meta{ + blockWithNoLabelsButManyOtherFields: { + BlockMeta: tsdb.BlockMeta{ + ULID: blockWithNoLabelsButManyOtherFields, + + MinTime: 100, + MaxTime: 200, + Version: 1, + + Compaction: tsdb.BlockMetaCompaction{ + Level: 5, + Sources: []ulid.ULID{blockWithNoLabelsButManyOtherFields}, + }, + }, + + Thanos: metadata.Thanos{ + Version: 10, + Downsample: metadata.ThanosDownsample{ + Resolution: 15, + }, + Source: "ingester", + }, + }, + + blockWithWrongTenant: { + BlockMeta: tsdb.BlockMeta{ + ULID: blockWithWrongTenant, + }, + + Thanos: metadata.Thanos{ + Labels: map[string]string{ + "test": "label", + mimir_tsdb.TenantIDExternalLabel: "wrong tenant", + }, + }, + }, + + blockWithManyMimirLabels: { + BlockMeta: tsdb.BlockMeta{ + ULID: blockWithManyMimirLabels, + }, + + Thanos: metadata.Thanos{ + Labels: map[string]string{ + mimir_tsdb.TenantIDExternalLabel: "fake", + mimir_tsdb.CompactorShardIDExternalLabel: "1_of_10", + mimir_tsdb.IngesterIDExternalLabel: "ingester-1", + }, + }, + }, + + blockWithNoChangesRequired: { + BlockMeta: tsdb.BlockMeta{ + ULID: blockWithNoChangesRequired, + }, + + Thanos: metadata.Thanos{ + Labels: map[string]string{ + mimir_tsdb.TenantIDExternalLabel: tenant, + }, + }, + }, + } + + for b, m := range inputMetas { + require.NoError(t, uploadMetadata(ctx, bkt, m, path.Join(b.String(), metadata.MetaFilename))) + } + + logs := &concurrency.SyncBuffer{} + logger := log.NewLogfmtLogger(logs) + + // Run conversion + assert.NoError(t, convertTenantBlocks(ctx, bkt, tenant, true, logger)) + + for b, m := range inputMetas { + meta, err := block.DownloadMeta(ctx, logger, bkt, b) + require.NoError(t, err) + require.Equal(t, m, meta) + } + + fmt.Println(logs.String()) + + assert.Equal(t, []string{ + `level=warn tenant=target_tenant msg="updating tenant label" block=00000000010000000000000000 old_value= new_value=target_tenant`, + `level=warn tenant=target_tenant msg="changes required, not uploading back due to dry run" block=00000000010000000000000000`, + `level=warn tenant=target_tenant msg="updating tenant label" block=00000000020000000000000000 old_value="wrong tenant" new_value=target_tenant`, + `level=warn tenant=target_tenant msg="removing unknown label" block=00000000020000000000000000 label=test value=label`, + `level=warn tenant=target_tenant msg="changes required, not uploading back due to dry run" block=00000000020000000000000000`, + `level=warn tenant=target_tenant msg="updating tenant label" block=00000000030000000000000000 old_value=fake new_value=target_tenant`, + `level=warn tenant=target_tenant msg="changes required, not uploading back due to dry run" block=00000000030000000000000000`, + `level=info tenant=target_tenant msg="no changes required" block=00000000040000000000000000`, + }, strings.Split(strings.TrimSpace(logs.String()), "\n")) +} diff --git a/cmd/mimir/Dockerfile b/cmd/mimir/Dockerfile index 53d04ffda97..934221b53db 100644 --- a/cmd/mimir/Dockerfile +++ b/cmd/mimir/Dockerfile @@ -12,7 +12,7 @@ ARG BINARY_SUFFIX="_${TARGETOS}_${TARGETARCH}" # Set to non-empty value to use ${TARGET_SUFFIX} when copying mimir binary, leave unset to use no suffix. ARG USE_BINARY_SUFFIX COPY mimir${USE_BINARY_SUFFIX:+${BINARY_SUFFIX}} /bin/mimir -EXPOSE 80 +EXPOSE 8080 ENTRYPOINT [ "/bin/mimir" ] ARG revision diff --git a/cmd/mimir/help-all.txt.tmpl b/cmd/mimir/help-all.txt.tmpl new file mode 100644 index 00000000000..9ce3e348b63 --- /dev/null +++ b/cmd/mimir/help-all.txt.tmpl @@ -0,0 +1,1617 @@ +Usage of ./cmd/mimir/mimir: + -activity-tracker.filepath string + File where ongoing activities are stored. If empty, activity tracking is disabled. (default "./metrics-activity.log") + -activity-tracker.max-entries int + Max number of concurrent activities that can be tracked. Used to size the file in advance. Additional activities are ignored. (default 1024) + -alertmanager-storage.azure.account-key value + Azure storage account key + -alertmanager-storage.azure.account-name string + Azure storage account name + -alertmanager-storage.azure.container-name string + Azure storage container name + -alertmanager-storage.azure.endpoint-suffix string + Azure storage endpoint suffix without schema. The account name will be prefixed to this value to create the FQDN. If set to empty string, default endpoint suffix is used. + -alertmanager-storage.azure.max-retries int + Number of retries for recoverable errors (default 20) + -alertmanager-storage.azure.msi-resource string + If set, this URL is used instead of https://. for obtaining ServicePrincipalToken from MSI. + -alertmanager-storage.azure.user-assigned-id string + User assigned identity. If empty, then System assigned identity is used. + -alertmanager-storage.backend string + Backend storage to use. Supported backends are: s3, gcs, azure, swift, filesystem, local. (default "filesystem") + -alertmanager-storage.filesystem.dir string + Local filesystem storage directory. (default "alertmanager") + -alertmanager-storage.gcs.bucket-name string + GCS bucket name + -alertmanager-storage.gcs.service-account value + JSON representing either a Google Developers Console client_credentials.json file or a Google Developers service account key file. If empty, fallback to Google default logic. + -alertmanager-storage.local.path string + Path at which alertmanager configurations are stored. + -alertmanager-storage.s3.access-key-id string + S3 access key ID + -alertmanager-storage.s3.bucket-name string + S3 bucket name + -alertmanager-storage.s3.endpoint string + The S3 bucket endpoint. It could be an AWS S3 endpoint listed at https://docs.aws.amazon.com/general/latest/gr/s3.html or the address of an S3-compatible service in hostname:port format. + -alertmanager-storage.s3.expect-continue-timeout duration + The time to wait for a server's first response headers after fully writing the request headers if the request has an Expect header. 0 to send the request body immediately. (default 1s) + -alertmanager-storage.s3.http.idle-conn-timeout duration + The time an idle connection will remain idle before closing. (default 1m30s) + -alertmanager-storage.s3.http.insecure-skip-verify + If the client connects to S3 via HTTPS and this option is enabled, the client will accept any certificate and hostname. + -alertmanager-storage.s3.http.response-header-timeout duration + The amount of time the client will wait for a servers response headers. (default 2m0s) + -alertmanager-storage.s3.insecure + If enabled, use http:// for the S3 endpoint instead of https://. This could be useful in local dev/test environments while using an S3-compatible backend storage, like Minio. + -alertmanager-storage.s3.max-connections-per-host int + Maximum number of connections per host. 0 means no limit. + -alertmanager-storage.s3.max-idle-connections int + Maximum number of idle (keep-alive) connections across all hosts. 0 means no limit. (default 100) + -alertmanager-storage.s3.max-idle-connections-per-host int + Maximum number of idle (keep-alive) connections to keep per-host. If 0, a built-in default value is used. (default 100) + -alertmanager-storage.s3.region string + S3 region. If unset, the client will issue a S3 GetBucketLocation API call to autodetect it. + -alertmanager-storage.s3.secret-access-key value + S3 secret access key + -alertmanager-storage.s3.signature-version string + The signature version to use for authenticating against S3. Supported values are: v4, v2. (default "v4") + -alertmanager-storage.s3.sse.kms-encryption-context string + KMS Encryption Context used for object encryption. It expects JSON formatted string. + -alertmanager-storage.s3.sse.kms-key-id string + KMS Key ID used to encrypt objects in S3 + -alertmanager-storage.s3.sse.type string + Enable AWS Server Side Encryption. Supported values: SSE-KMS, SSE-S3. + -alertmanager-storage.s3.tls-handshake-timeout duration + Maximum time to wait for a TLS handshake. 0 means no limit. (default 10s) + -alertmanager-storage.swift.auth-url string + OpenStack Swift authentication URL + -alertmanager-storage.swift.auth-version int + OpenStack Swift authentication API version. 0 to autodetect. + -alertmanager-storage.swift.connect-timeout duration + Time after which a connection attempt is aborted. (default 10s) + -alertmanager-storage.swift.container-name string + Name of the OpenStack Swift container to put chunks in. + -alertmanager-storage.swift.domain-id string + OpenStack Swift user's domain ID. + -alertmanager-storage.swift.domain-name string + OpenStack Swift user's domain name. + -alertmanager-storage.swift.max-retries int + Max retries on requests error. (default 3) + -alertmanager-storage.swift.password string + OpenStack Swift API key. + -alertmanager-storage.swift.project-domain-id string + ID of the OpenStack Swift project's domain (v3 auth only), only needed if it differs the from user domain. + -alertmanager-storage.swift.project-domain-name string + Name of the OpenStack Swift project's domain (v3 auth only), only needed if it differs from the user domain. + -alertmanager-storage.swift.project-id string + OpenStack Swift project ID (v2,v3 auth only). + -alertmanager-storage.swift.project-name string + OpenStack Swift project name (v2,v3 auth only). + -alertmanager-storage.swift.region-name string + OpenStack Swift Region to use (v2,v3 auth only). + -alertmanager-storage.swift.request-timeout duration + Time after which an idle request is aborted. The timeout watchdog is reset each time some data is received, so the timeout triggers after X time no data is received on a request. (default 5s) + -alertmanager-storage.swift.user-domain-id string + OpenStack Swift user's domain ID. + -alertmanager-storage.swift.user-domain-name string + OpenStack Swift user's domain name. + -alertmanager-storage.swift.user-id string + OpenStack Swift user ID. + -alertmanager-storage.swift.username string + OpenStack Swift username. + -alertmanager.alertmanager-client.remote-timeout duration + Timeout for downstream alertmanagers. (default 2s) + -alertmanager.alertmanager-client.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -alertmanager.alertmanager-client.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -alertmanager.alertmanager-client.tls-enabled + Enable TLS in the GRPC client. This flag needs to be enabled when any other TLS flag is set. If set to false, insecure connection to gRPC server will be used. + -alertmanager.alertmanager-client.tls-insecure-skip-verify + Skip validating server certificate. + -alertmanager.alertmanager-client.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -alertmanager.alertmanager-client.tls-server-name string + Override the expected name on the server certificate. + -alertmanager.configs.fallback string + Filename of fallback config to use if none specified for instance. + -alertmanager.configs.poll-interval duration + How frequently to poll Alertmanager configs. (default 15s) + -alertmanager.enable-api + Enable the alertmanager config API. (default true) + -alertmanager.max-alerts-count int + Maximum number of alerts that a single user can have. Inserting more alerts will fail with a log message and metric increment. 0 = no limit. + -alertmanager.max-alerts-size-bytes int + Maximum total size of alerts that a single user can have, alert size is the sum of the bytes of its labels, annotations and generatorURL. Inserting more alerts will fail with a log message and metric increment. 0 = no limit. + -alertmanager.max-config-size-bytes int + Maximum size of configuration file for Alertmanager that tenant can upload via Alertmanager API. 0 = no limit. + -alertmanager.max-dispatcher-aggregation-groups int + Maximum number of aggregation groups in Alertmanager's dispatcher that a tenant can have. Each active aggregation group uses single goroutine. When the limit is reached, dispatcher will not dispatch alerts that belong to additional aggregation groups, but existing groups will keep working properly. 0 = no limit. + -alertmanager.max-recv-msg-size int + Maximum size (bytes) of an accepted HTTP request body. (default 16777216) + -alertmanager.max-template-size-bytes int + Maximum size of single template in tenant's Alertmanager configuration uploaded via Alertmanager API. 0 = no limit. + -alertmanager.max-templates-count int + Maximum number of templates in tenant's Alertmanager configuration uploaded via Alertmanager API. 0 = no limit. + -alertmanager.notification-rate-limit float + Per-user rate limit for sending notifications from Alertmanager in notifications/sec. 0 = rate limit disabled. Negative value = no notifications are allowed. + -alertmanager.notification-rate-limit-per-integration value + Per-integration notification rate limits. Value is a map, where each key is integration name and value is a rate-limit (float). On command line, this map is given in JSON format. Rate limit has the same meaning as -alertmanager.notification-rate-limit, but only applies for specific integration. Allowed integration names: webhook, email, pagerduty, opsgenie, wechat, slack, victorops, pushover, sns. (default {}) + -alertmanager.peer-timeout duration + Time to wait between peers to send notifications. (default 15s) + -alertmanager.persist-interval duration + The interval between persisting the current alertmanager state (notification log and silences) to object storage. This is only used when sharding is enabled. This state is read when all replicas for a shard can not be contacted. In this scenario, having persisted the state more frequently will result in potentially fewer lost silences, and fewer duplicate notifications. (default 15m0s) + -alertmanager.receivers-firewall-block-cidr-networks value + Comma-separated list of network CIDRs to block in Alertmanager receiver integrations. + -alertmanager.receivers-firewall-block-private-addresses + True to block private and local addresses in Alertmanager receiver integrations. It blocks private addresses defined by RFC 1918 (IPv4 addresses) and RFC 4193 (IPv6 addresses), as well as loopback, local unicast and local multicast addresses. + -alertmanager.sharding-ring.consul.acl-token string + ACL Token used to interact with Consul. + -alertmanager.sharding-ring.consul.client-timeout duration + HTTP timeout when talking to Consul (default 20s) + -alertmanager.sharding-ring.consul.consistent-reads + Enable consistent reads to Consul. + -alertmanager.sharding-ring.consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -alertmanager.sharding-ring.consul.watch-burst-size int + Burst size used in rate limit. Values less than 1 are treated as 1. (default 1) + -alertmanager.sharding-ring.consul.watch-rate-limit float + Rate limit when watching key or prefix in Consul, in requests per second. 0 disables the rate limit. (default 1) + -alertmanager.sharding-ring.etcd.dial-timeout duration + The dial timeout for the etcd connection. (default 10s) + -alertmanager.sharding-ring.etcd.endpoints value + The etcd endpoints to connect to. + -alertmanager.sharding-ring.etcd.max-retries int + The maximum number of retries to do for failed ops. (default 10) + -alertmanager.sharding-ring.etcd.password string + Etcd password. + -alertmanager.sharding-ring.etcd.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -alertmanager.sharding-ring.etcd.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -alertmanager.sharding-ring.etcd.tls-enabled + Enable TLS. + -alertmanager.sharding-ring.etcd.tls-insecure-skip-verify + Skip validating server certificate. + -alertmanager.sharding-ring.etcd.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -alertmanager.sharding-ring.etcd.tls-server-name string + Override the expected name on the server certificate. + -alertmanager.sharding-ring.etcd.username string + Etcd username. + -alertmanager.sharding-ring.heartbeat-period duration + Period at which to heartbeat to the ring. 0 = disabled. (default 15s) + -alertmanager.sharding-ring.heartbeat-timeout duration + The heartbeat timeout after which alertmanagers are considered unhealthy within the ring. 0 = never (timeout disabled). (default 1m0s) + -alertmanager.sharding-ring.instance-addr string + IP address to advertise in the ring. + -alertmanager.sharding-ring.instance-availability-zone string + The availability zone where this instance is running. Required if zone-awareness is enabled. + -alertmanager.sharding-ring.instance-id string + Instance ID to register in the ring. (default "{{.Hostname}}") + -alertmanager.sharding-ring.instance-interface-names value + Name of network interface to read address from. (default [eth0 en0]) + -alertmanager.sharding-ring.instance-port int + Port to advertise in the ring (defaults to server.grpc-listen-port). + -alertmanager.sharding-ring.multi.mirror-enabled + Mirror writes to secondary store. + -alertmanager.sharding-ring.multi.mirror-timeout duration + Timeout for storing value to secondary store. (default 2s) + -alertmanager.sharding-ring.multi.primary string + Primary backend storage used by multi-client. + -alertmanager.sharding-ring.multi.secondary string + Secondary backend storage used by multi-client. + -alertmanager.sharding-ring.prefix string + The prefix for the keys in the store. Should end with a /. (default "alertmanagers/") + -alertmanager.sharding-ring.replication-factor int + The replication factor to use when sharding the alertmanager. (default 3) + -alertmanager.sharding-ring.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "consul") + -alertmanager.sharding-ring.zone-awareness-enabled + True to enable zone-awareness and replicate alerts across different availability zones. + -alertmanager.storage.path string + Directory to store Alertmanager state and temporarily configuration files. The content of this directory is not required to be persisted between restarts unless Alertmanager replication has been disabled. (default "./data-alertmanager/") + -alertmanager.storage.retention duration + How long to keep data for. (default 120h0m0s) + -alertmanager.web.external-url value + The URL under which Alertmanager is externally reachable (eg. could be different than -http.alertmanager-http-prefix in case Alertmanager is served via a reverse proxy). This setting is used both to configure the internal requests router and to generate links in alert templates. If the external URL has a path portion, it will be used to prefix all HTTP endpoints served by Alertmanager, both the UI and API. (default http://localhost:8080/alertmanager) + -api.skip-label-name-validation-header-enabled + Allows to skip label name validation via header on the http write path. Use with caution as it breaks PromQL. Allowing this for external clients allows any client to send invalid label names. After enabling it, requests with a specific HTTP header set to true will not have label names validated. + -auth.multitenancy-enabled + When set to true, incoming HTTP requests must specify tenant ID in HTTP X-Scope-OrgId header. When set to false, tenant ID from -auth.no-auth-tenant is used instead. (default true) + -auth.no-auth-tenant string + Tenant ID to use when multitenancy is disabled. (default "anonymous") + -blocks-storage.azure.account-key value + Azure storage account key + -blocks-storage.azure.account-name string + Azure storage account name + -blocks-storage.azure.container-name string + Azure storage container name + -blocks-storage.azure.endpoint-suffix string + Azure storage endpoint suffix without schema. The account name will be prefixed to this value to create the FQDN. If set to empty string, default endpoint suffix is used. + -blocks-storage.azure.max-retries int + Number of retries for recoverable errors (default 20) + -blocks-storage.azure.msi-resource string + If set, this URL is used instead of https://. for obtaining ServicePrincipalToken from MSI. + -blocks-storage.azure.user-assigned-id string + User assigned identity. If empty, then System assigned identity is used. + -blocks-storage.backend string + Backend storage to use. Supported backends are: s3, gcs, azure, swift, filesystem. (default "filesystem") + -blocks-storage.bucket-store.block-sync-concurrency int + Maximum number of concurrent blocks synching per tenant. (default 20) + -blocks-storage.bucket-store.bucket-index.enabled + If enabled, queriers and store-gateways discover blocks by reading a bucket index (created and updated by the compactor) instead of periodically scanning the bucket. (default true) + -blocks-storage.bucket-store.bucket-index.idle-timeout duration + How long a unused bucket index should be cached. Once this timeout expires, the unused bucket index is removed from the in-memory cache. This option is used only by querier. (default 1h0m0s) + -blocks-storage.bucket-store.bucket-index.max-stale-period duration + The maximum allowed age of a bucket index (last updated) before queries start failing because the bucket index is too old. The bucket index is periodically updated by the compactor, while this check is enforced in the querier (at query time). (default 1h0m0s) + -blocks-storage.bucket-store.bucket-index.update-on-error-interval duration + How frequently a bucket index, which previously failed to load, should be tried to load again. This option is used only by querier. (default 1m0s) + -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes int + Size - in bytes - of the largest chunks pool bucket. (default 50000000) + -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes int + Size - in bytes - of the smallest chunks pool bucket. (default 16000) + -blocks-storage.bucket-store.chunks-cache.attributes-in-memory-max-items int + Maximum number of object attribute items to keep in a first level in-memory LRU cache. Metadata will be stored and fetched in-memory before hitting the cache backend. 0 to disable the in-memory cache. + -blocks-storage.bucket-store.chunks-cache.attributes-ttl duration + TTL for caching object attributes for chunks. If the metadata cache is configured, attributes will be stored under this cache backend, otherwise attributes are stored in the chunks cache backend. (default 168h0m0s) + -blocks-storage.bucket-store.chunks-cache.backend string + Backend for chunks cache, if not empty. Supported values: memcached. + -blocks-storage.bucket-store.chunks-cache.max-get-range-requests int + Maximum number of sub-GetRange requests that a single GetRange request can be split into when fetching chunks. Zero or negative value = unlimited number of sub-requests. (default 3) + -blocks-storage.bucket-store.chunks-cache.memcached.addresses string + Comma separated list of memcached addresses. Supported prefixes are: dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup made after that). + -blocks-storage.bucket-store.chunks-cache.memcached.max-async-buffer-size int + The maximum number of enqueued asynchronous operations allowed. (default 25000) + -blocks-storage.bucket-store.chunks-cache.memcached.max-async-concurrency int + The maximum number of concurrent asynchronous operations can occur. (default 50) + -blocks-storage.bucket-store.chunks-cache.memcached.max-get-multi-batch-size int + The maximum number of keys a single underlying get operation should run. If more keys are specified, internally keys are split into multiple batches and fetched concurrently, honoring the max concurrency. If set to 0, the max batch size is unlimited. (default 100) + -blocks-storage.bucket-store.chunks-cache.memcached.max-get-multi-concurrency int + The maximum number of concurrent connections running get operations. If set to 0, concurrency is unlimited. (default 100) + -blocks-storage.bucket-store.chunks-cache.memcached.max-idle-connections int + The maximum number of idle connections that will be maintained per address. (default 100) + -blocks-storage.bucket-store.chunks-cache.memcached.max-item-size int + The maximum size of an item stored in memcached. Bigger items are not stored. If set to 0, no maximum size is enforced. (default 1048576) + -blocks-storage.bucket-store.chunks-cache.memcached.timeout duration + The socket read/write timeout. (default 200ms) + -blocks-storage.bucket-store.chunks-cache.subrange-size int + Size of each subrange that bucket object is split into for better caching. (default 16000) + -blocks-storage.bucket-store.chunks-cache.subrange-ttl duration + TTL for caching individual chunks subranges. (default 24h0m0s) + -blocks-storage.bucket-store.consistency-delay duration + Minimum age of a block before it's being read. Set it to safe value (e.g 30m) if your object storage is eventually consistent. GCS and S3 are (roughly) strongly consistent. + -blocks-storage.bucket-store.ignore-blocks-within duration + Blocks with minimum time within this duration are ignored, and not loaded by store-gateway. Useful when used together with -querier.query-store-after to prevent loading young blocks, because there are usually many of them (depending on number of ingesters) and they are not yet compacted. Negative values or 0 disable the filter. + -blocks-storage.bucket-store.ignore-deletion-marks-delay duration + Duration after which the blocks marked for deletion will be filtered out while fetching blocks. The idea of ignore-deletion-marks-delay is to ignore blocks that are marked for deletion with some delay. This ensures store can still serve blocks that are meant to be deleted but do not have a replacement yet. (default 1h0m0s) + -blocks-storage.bucket-store.index-cache.backend string + The index cache backend type. Supported values: inmemory, memcached. (default "inmemory") + -blocks-storage.bucket-store.index-cache.inmemory.max-size-bytes uint + Maximum size in bytes of in-memory index cache used to speed up blocks index lookups (shared between all tenants). (default 1073741824) + -blocks-storage.bucket-store.index-cache.memcached.addresses string + Comma separated list of memcached addresses. Supported prefixes are: dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup made after that). + -blocks-storage.bucket-store.index-cache.memcached.max-async-buffer-size int + The maximum number of enqueued asynchronous operations allowed. (default 25000) + -blocks-storage.bucket-store.index-cache.memcached.max-async-concurrency int + The maximum number of concurrent asynchronous operations can occur. (default 50) + -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-batch-size int + The maximum number of keys a single underlying get operation should run. If more keys are specified, internally keys are split into multiple batches and fetched concurrently, honoring the max concurrency. If set to 0, the max batch size is unlimited. (default 100) + -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency int + The maximum number of concurrent connections running get operations. If set to 0, concurrency is unlimited. (default 100) + -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections int + The maximum number of idle connections that will be maintained per address. (default 100) + -blocks-storage.bucket-store.index-cache.memcached.max-item-size int + The maximum size of an item stored in memcached. Bigger items are not stored. If set to 0, no maximum size is enforced. (default 1048576) + -blocks-storage.bucket-store.index-cache.memcached.timeout duration + The socket read/write timeout. (default 200ms) + -blocks-storage.bucket-store.index-header-lazy-loading-enabled + If enabled, store-gateway will lazy load an index-header only once required by a query. (default true) + -blocks-storage.bucket-store.index-header-lazy-loading-idle-timeout duration + If index-header lazy loading is enabled and this setting is > 0, the store-gateway will offload unused index-headers after 'idle timeout' inactivity. (default 1h0m0s) + -blocks-storage.bucket-store.max-chunk-pool-bytes uint + Max size - in bytes - of a chunks pool, used to reduce memory allocations. The pool is shared across all tenants. 0 to disable the limit. (default 2147483648) + -blocks-storage.bucket-store.max-concurrent int + Max number of concurrent queries to execute against the long-term storage. The limit is shared across all tenants. (default 100) + -blocks-storage.bucket-store.meta-sync-concurrency int + Number of Go routines to use when syncing block meta files from object storage per tenant. (default 20) + -blocks-storage.bucket-store.metadata-cache.backend string + Backend for metadata cache, if not empty. Supported values: memcached. + -blocks-storage.bucket-store.metadata-cache.block-index-attributes-ttl duration + How long to cache attributes of the block index. (default 168h0m0s) + -blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl duration + How long to cache content of the bucket index. (default 5m0s) + -blocks-storage.bucket-store.metadata-cache.bucket-index-max-size-bytes int + Maximum size of bucket index content to cache in bytes. Caching will be skipped if the content exceeds this size. This is useful to avoid network round trip for large content if the configured caching backend has an hard limit on cached items size (in this case, you should set this limit to the same limit in the caching backend). (default 1048576) + -blocks-storage.bucket-store.metadata-cache.chunks-list-ttl duration + How long to cache list of chunks for a block. (default 24h0m0s) + -blocks-storage.bucket-store.metadata-cache.memcached.addresses string + Comma separated list of memcached addresses. Supported prefixes are: dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup made after that). + -blocks-storage.bucket-store.metadata-cache.memcached.max-async-buffer-size int + The maximum number of enqueued asynchronous operations allowed. (default 25000) + -blocks-storage.bucket-store.metadata-cache.memcached.max-async-concurrency int + The maximum number of concurrent asynchronous operations can occur. (default 50) + -blocks-storage.bucket-store.metadata-cache.memcached.max-get-multi-batch-size int + The maximum number of keys a single underlying get operation should run. If more keys are specified, internally keys are split into multiple batches and fetched concurrently, honoring the max concurrency. If set to 0, the max batch size is unlimited. (default 100) + -blocks-storage.bucket-store.metadata-cache.memcached.max-get-multi-concurrency int + The maximum number of concurrent connections running get operations. If set to 0, concurrency is unlimited. (default 100) + -blocks-storage.bucket-store.metadata-cache.memcached.max-idle-connections int + The maximum number of idle connections that will be maintained per address. (default 100) + -blocks-storage.bucket-store.metadata-cache.memcached.max-item-size int + The maximum size of an item stored in memcached. Bigger items are not stored. If set to 0, no maximum size is enforced. (default 1048576) + -blocks-storage.bucket-store.metadata-cache.memcached.timeout duration + The socket read/write timeout. (default 200ms) + -blocks-storage.bucket-store.metadata-cache.metafile-attributes-ttl duration + How long to cache attributes of the block metafile. (default 168h0m0s) + -blocks-storage.bucket-store.metadata-cache.metafile-content-ttl duration + How long to cache content of the metafile. (default 24h0m0s) + -blocks-storage.bucket-store.metadata-cache.metafile-doesnt-exist-ttl duration + How long to cache information that block metafile doesn't exist. Also used for user deletion mark file. (default 5m0s) + -blocks-storage.bucket-store.metadata-cache.metafile-exists-ttl duration + How long to cache information that block metafile exists. Also used for user deletion mark file. (default 2h0m0s) + -blocks-storage.bucket-store.metadata-cache.metafile-max-size-bytes int + Maximum size of metafile content to cache in bytes. Caching will be skipped if the content exceeds this size. This is useful to avoid network round trip for large content if the configured caching backend has an hard limit on cached items size (in this case, you should set this limit to the same limit in the caching backend). (default 1048576) + -blocks-storage.bucket-store.metadata-cache.tenant-blocks-list-ttl duration + How long to cache list of blocks for each tenant. (default 5m0s) + -blocks-storage.bucket-store.metadata-cache.tenants-list-ttl duration + How long to cache list of tenants in the bucket. (default 15m0s) + -blocks-storage.bucket-store.partitioner-max-gap-bytes uint + Max size - in bytes - of a gap for which the partitioner aggregates together two bucket GET object requests. (default 524288) + -blocks-storage.bucket-store.posting-offsets-in-mem-sampling int + Controls what is the ratio of postings offsets that the store will hold in memory. (default 32) + -blocks-storage.bucket-store.series-hash-cache-max-size-bytes uint + Max size - in bytes - of the in-memory series hash cache. The cache is shared across all tenants and it's used only when query sharding is enabled. (default 1073741824) + -blocks-storage.bucket-store.sync-dir string + Directory to store synchronized TSDB index headers. This directory is not required to be persisted between restarts, but it's highly recommended in order to improve the store-gateway startup time. (default "./tsdb-sync/") + -blocks-storage.bucket-store.sync-interval duration + How frequently to scan the bucket, or to refresh the bucket index (if enabled), in order to look for changes (new blocks shipped by ingesters and blocks deleted by retention or compaction). (default 15m0s) + -blocks-storage.bucket-store.tenant-sync-concurrency int + Maximum number of concurrent tenants synching blocks. (default 10) + -blocks-storage.filesystem.dir string + Local filesystem storage directory. (default "blocks") + -blocks-storage.gcs.bucket-name string + GCS bucket name + -blocks-storage.gcs.service-account value + JSON representing either a Google Developers Console client_credentials.json file or a Google Developers service account key file. If empty, fallback to Google default logic. + -blocks-storage.s3.access-key-id string + S3 access key ID + -blocks-storage.s3.bucket-name string + S3 bucket name + -blocks-storage.s3.endpoint string + The S3 bucket endpoint. It could be an AWS S3 endpoint listed at https://docs.aws.amazon.com/general/latest/gr/s3.html or the address of an S3-compatible service in hostname:port format. + -blocks-storage.s3.expect-continue-timeout duration + The time to wait for a server's first response headers after fully writing the request headers if the request has an Expect header. 0 to send the request body immediately. (default 1s) + -blocks-storage.s3.http.idle-conn-timeout duration + The time an idle connection will remain idle before closing. (default 1m30s) + -blocks-storage.s3.http.insecure-skip-verify + If the client connects to S3 via HTTPS and this option is enabled, the client will accept any certificate and hostname. + -blocks-storage.s3.http.response-header-timeout duration + The amount of time the client will wait for a servers response headers. (default 2m0s) + -blocks-storage.s3.insecure + If enabled, use http:// for the S3 endpoint instead of https://. This could be useful in local dev/test environments while using an S3-compatible backend storage, like Minio. + -blocks-storage.s3.max-connections-per-host int + Maximum number of connections per host. 0 means no limit. + -blocks-storage.s3.max-idle-connections int + Maximum number of idle (keep-alive) connections across all hosts. 0 means no limit. (default 100) + -blocks-storage.s3.max-idle-connections-per-host int + Maximum number of idle (keep-alive) connections to keep per-host. If 0, a built-in default value is used. (default 100) + -blocks-storage.s3.region string + S3 region. If unset, the client will issue a S3 GetBucketLocation API call to autodetect it. + -blocks-storage.s3.secret-access-key value + S3 secret access key + -blocks-storage.s3.signature-version string + The signature version to use for authenticating against S3. Supported values are: v4, v2. (default "v4") + -blocks-storage.s3.sse.kms-encryption-context string + KMS Encryption Context used for object encryption. It expects JSON formatted string. + -blocks-storage.s3.sse.kms-key-id string + KMS Key ID used to encrypt objects in S3 + -blocks-storage.s3.sse.type string + Enable AWS Server Side Encryption. Supported values: SSE-KMS, SSE-S3. + -blocks-storage.s3.tls-handshake-timeout duration + Maximum time to wait for a TLS handshake. 0 means no limit. (default 10s) + -blocks-storage.swift.auth-url string + OpenStack Swift authentication URL + -blocks-storage.swift.auth-version int + OpenStack Swift authentication API version. 0 to autodetect. + -blocks-storage.swift.connect-timeout duration + Time after which a connection attempt is aborted. (default 10s) + -blocks-storage.swift.container-name string + Name of the OpenStack Swift container to put chunks in. + -blocks-storage.swift.domain-id string + OpenStack Swift user's domain ID. + -blocks-storage.swift.domain-name string + OpenStack Swift user's domain name. + -blocks-storage.swift.max-retries int + Max retries on requests error. (default 3) + -blocks-storage.swift.password string + OpenStack Swift API key. + -blocks-storage.swift.project-domain-id string + ID of the OpenStack Swift project's domain (v3 auth only), only needed if it differs the from user domain. + -blocks-storage.swift.project-domain-name string + Name of the OpenStack Swift project's domain (v3 auth only), only needed if it differs from the user domain. + -blocks-storage.swift.project-id string + OpenStack Swift project ID (v2,v3 auth only). + -blocks-storage.swift.project-name string + OpenStack Swift project name (v2,v3 auth only). + -blocks-storage.swift.region-name string + OpenStack Swift Region to use (v2,v3 auth only). + -blocks-storage.swift.request-timeout duration + Time after which an idle request is aborted. The timeout watchdog is reset each time some data is received, so the timeout triggers after X time no data is received on a request. (default 5s) + -blocks-storage.swift.user-domain-id string + OpenStack Swift user's domain ID. + -blocks-storage.swift.user-domain-name string + OpenStack Swift user's domain name. + -blocks-storage.swift.user-id string + OpenStack Swift user ID. + -blocks-storage.swift.username string + OpenStack Swift username. + -blocks-storage.tsdb.block-ranges-period value + TSDB blocks range period. (default 2h0m0s) + -blocks-storage.tsdb.close-idle-tsdb-timeout duration + If TSDB has not received any data for this duration, and all blocks from TSDB have been shipped, TSDB is closed and deleted from local disk. If set to positive value, this value should be equal or higher than -querier.query-ingesters-within flag to make sure that TSDB is not closed prematurely, which could cause partial query results. 0 or negative value disables closing of idle TSDB. (default 13h0m0s) + -blocks-storage.tsdb.dir string + Directory to store TSDBs (including WAL) in the ingesters. This directory is required to be persisted between restarts. (default "./tsdb/") + -blocks-storage.tsdb.flush-blocks-on-shutdown + True to flush blocks to storage on shutdown. If false, incomplete blocks will be reused after restart. + -blocks-storage.tsdb.head-chunks-end-time-variance float + [experimental] How much variance (as percentage between 0 and 1) should be applied to the chunk end time, to spread chunks writing across time. Doesn't apply to the last chunk of the chunk range. 0 means no variance. + -blocks-storage.tsdb.head-chunks-write-buffer-size-bytes int + The write buffer size used by the head chunks mapper. Lower values reduce memory utilisation on clusters with a large number of tenants at the cost of increased disk I/O operations. (default 4194304) + -blocks-storage.tsdb.head-chunks-write-queue-size int + [experimental] The size of the write queue used by the head chunks mapper. Lower values reduce memory utilisation at the cost of potentially higher ingest latency. Value of 0 switches chunks mapper to implementation without a queue. + -blocks-storage.tsdb.head-compaction-concurrency int + Maximum number of tenants concurrently compacting TSDB head into a new block (default 5) + -blocks-storage.tsdb.head-compaction-idle-timeout duration + If TSDB head is idle for this duration, it is compacted. Note that up to 25% jitter is added to the value to avoid ingesters compacting concurrently. 0 means disabled. (default 1h0m0s) + -blocks-storage.tsdb.head-compaction-interval duration + How frequently ingesters try to compact TSDB head. Block is only created if data covers smallest block range. Must be greater than 0 and max 5 minutes. (default 1m0s) + -blocks-storage.tsdb.isolation-enabled + Enables TSDB isolation feature. Disabling may improve performance. (default true) + -blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup int + limit the number of concurrently opening TSDB's on startup (default 10) + -blocks-storage.tsdb.memory-snapshot-on-shutdown + [experimental] True to enable snapshotting of in-memory TSDB data on disk when shutting down. + -blocks-storage.tsdb.retention-period duration + TSDB blocks retention in the ingester before a block is removed. This should be larger than the -blocks-storage.tsdb.block-ranges-period, -querier.query-store-after and large enough to give store-gateways and queriers enough time to discover newly uploaded blocks. (default 24h0m0s) + -blocks-storage.tsdb.series-hash-cache-max-size-bytes uint + Max size - in bytes - of the in-memory series hash cache. The cache is shared across all tenants and it's used only when query sharding is enabled. (default 1073741824) + -blocks-storage.tsdb.ship-concurrency int + Maximum number of tenants concurrently shipping blocks to the storage. (default 10) + -blocks-storage.tsdb.ship-interval duration + How frequently the TSDB blocks are scanned and new ones are shipped to the storage. 0 means shipping is disabled. (default 1m0s) + -blocks-storage.tsdb.stripe-size int + The number of shards of series to use in TSDB (must be a power of 2). Reducing this will decrease memory footprint, but can negatively impact performance. (default 16384) + -blocks-storage.tsdb.wal-compression-enabled + True to enable TSDB WAL compression. + -blocks-storage.tsdb.wal-segment-size-bytes int + TSDB WAL segments files max size (bytes). (default 134217728) + -compactor.block-ranges value + List of compaction time ranges. (default 2h0m0s,12h0m0s,24h0m0s) + -compactor.block-sync-concurrency int + Number of Go routines to use when downloading blocks for compaction and uploading resulting blocks. (default 8) + -compactor.blocks-retention-period value + Delete blocks containing samples older than the specified retention period. 0 to disable. + -compactor.cleanup-concurrency int + Max number of tenants for which blocks cleanup and maintenance should run concurrently. (default 20) + -compactor.cleanup-interval duration + How frequently compactor should run blocks cleanup and maintenance, as well as update the bucket index. (default 15m0s) + -compactor.compaction-concurrency int + Max number of concurrent compactions running. (default 1) + -compactor.compaction-interval duration + The frequency at which the compaction runs (default 1h0m0s) + -compactor.compaction-jobs-order string + The sorting to use when deciding which compaction jobs should run first for a given tenant. Supported values are: smallest-range-oldest-blocks-first, newest-blocks-first. (default "smallest-range-oldest-blocks-first") + -compactor.compaction-retries int + How many times to retry a failed compaction within a single compaction run. (default 3) + -compactor.compactor-tenant-shard-size int + Max number of compactors that can compact blocks for single tenant. 0 to disable the limit and use all compactors. + -compactor.consistency-delay duration + Minimum age of fresh (non-compacted) blocks before they are being processed. Malformed blocks older than the maximum of consistency-delay and 48h0m0s will be removed. + -compactor.data-dir string + Directory to temporarily store blocks during compaction. This directory is not required to be persisted between restarts. (default "./data-compactor/") + -compactor.deletion-delay duration + Time before a block marked for deletion is deleted from bucket. If not 0, blocks will be marked for deletion and compactor component will permanently delete blocks marked for deletion from the bucket. If 0, blocks will be deleted straight away. Note that deleting blocks immediately can cause query failures. (default 12h0m0s) + -compactor.disabled-tenants value + Comma separated list of tenants that cannot be compacted by this compactor. If specified, and compactor would normally pick given tenant for compaction (via -compactor.enabled-tenants or sharding), it will be ignored instead. + -compactor.enabled-tenants value + Comma separated list of tenants that can be compacted. If specified, only these tenants will be compacted by compactor, otherwise all tenants can be compacted. Subject to sharding. + -compactor.max-closing-blocks-concurrency int + Max number of blocks that can be closed concurrently during split compaction. Note that closing of newly compacted block uses a lot of memory for writing index. (default 1) + -compactor.max-compaction-time duration + Max time for starting compactions for a single tenant. After this time no new compactions for the tenant are started before next compaction cycle. This can help in multi-tenant environments to avoid single tenant using all compaction time, but also in single-tenant environments to force new discovery of blocks more often. 0 = disabled. + -compactor.max-opening-blocks-concurrency int + Number of goroutines opening blocks before compaction. (default 1) + -compactor.meta-sync-concurrency int + Number of Go routines to use when syncing block meta files from the long term storage. (default 20) + -compactor.ring.consul.acl-token string + ACL Token used to interact with Consul. + -compactor.ring.consul.client-timeout duration + HTTP timeout when talking to Consul (default 20s) + -compactor.ring.consul.consistent-reads + Enable consistent reads to Consul. + -compactor.ring.consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -compactor.ring.consul.watch-burst-size int + Burst size used in rate limit. Values less than 1 are treated as 1. (default 1) + -compactor.ring.consul.watch-rate-limit float + Rate limit when watching key or prefix in Consul, in requests per second. 0 disables the rate limit. (default 1) + -compactor.ring.etcd.dial-timeout duration + The dial timeout for the etcd connection. (default 10s) + -compactor.ring.etcd.endpoints value + The etcd endpoints to connect to. + -compactor.ring.etcd.max-retries int + The maximum number of retries to do for failed ops. (default 10) + -compactor.ring.etcd.password string + Etcd password. + -compactor.ring.etcd.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -compactor.ring.etcd.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -compactor.ring.etcd.tls-enabled + Enable TLS. + -compactor.ring.etcd.tls-insecure-skip-verify + Skip validating server certificate. + -compactor.ring.etcd.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -compactor.ring.etcd.tls-server-name string + Override the expected name on the server certificate. + -compactor.ring.etcd.username string + Etcd username. + -compactor.ring.heartbeat-period duration + Period at which to heartbeat to the ring. 0 = disabled. (default 5s) + -compactor.ring.heartbeat-timeout duration + The heartbeat timeout after which compactors are considered unhealthy within the ring. 0 = never (timeout disabled). (default 1m0s) + -compactor.ring.instance-addr string + IP address to advertise in the ring. + -compactor.ring.instance-id string + Instance ID to register in the ring. (default "{{.Hostname}}") + -compactor.ring.instance-interface-names value + Name of network interface to read address from. (default [eth0 en0]) + -compactor.ring.instance-port int + Port to advertise in the ring (defaults to server.grpc-listen-port). + -compactor.ring.multi.mirror-enabled + Mirror writes to secondary store. + -compactor.ring.multi.mirror-timeout duration + Timeout for storing value to secondary store. (default 2s) + -compactor.ring.multi.primary string + Primary backend storage used by multi-client. + -compactor.ring.multi.secondary string + Secondary backend storage used by multi-client. + -compactor.ring.prefix string + The prefix for the keys in the store. Should end with a /. (default "collectors/") + -compactor.ring.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "memberlist") + -compactor.ring.wait-active-instance-timeout duration + Timeout for waiting on compactor to become ACTIVE in the ring. (default 10m0s) + -compactor.ring.wait-stability-max-duration duration + Maximum time to wait for ring stability at startup. If the compactor ring keeps changing after this period of time, the compactor will start anyway. (default 5m0s) + -compactor.ring.wait-stability-min-duration duration + Minimum time to wait for ring stability at startup. 0 to disable. + -compactor.split-and-merge-shards int + The number of shards to use when splitting blocks. 0 to disable splitting. + -compactor.split-groups int + Number of groups that blocks for splitting should be grouped into. Each group of blocks is then split separately. Number of output split shards is controlled by -compactor.split-and-merge-shards. (default 1) + -compactor.symbols-flushers-concurrency int + Number of symbols flushers used when doing split compaction. (default 1) + -compactor.tenant-cleanup-delay duration + For tenants marked for deletion, this is time between deleting of last block, and doing final cleanup (marker files, debug files) of the tenant. (default 6h0m0s) + -config.expand-env + Expands ${var} or $var in config according to the values of the environment variables. + -config.file value + Configuration file to load. + -consul.acl-token string + ACL Token used to interact with Consul. + -consul.client-timeout duration + HTTP timeout when talking to Consul (default 20s) + -consul.consistent-reads + Enable consistent reads to Consul. + -consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -consul.watch-burst-size int + Burst size used in rate limit. Values less than 1 are treated as 1. (default 1) + -consul.watch-rate-limit float + Rate limit when watching key or prefix in Consul, in requests per second. 0 disables the rate limit. (default 1) + -debug.block-profile-rate int + Fraction of goroutine blocking events that are reported in the blocking profile. 1 to include every blocking event in the profile, 0 to disable. + -debug.mutex-profile-fraction int + Fraction of mutex contention events that are reported in the mutex profile. On average 1/rate events are reported. 0 to disable. + -distributor.client-cleanup-period duration + How frequently to clean up clients for ingesters that have gone away. (default 15s) + -distributor.drop-label value + This flag can be used to specify label names that to drop during sample ingestion within the distributor and can be repeated in order to drop multiple labels. + -distributor.excluded-zones value + Comma-separated list of zones to exclude from the ring. Instances in excluded zones will be filtered out from the ring. + -distributor.extend-writes + Try writing to an additional ingester in the presence of an ingester not in the ACTIVE state. It is useful to disable this along with -ingester.unregister-on-shutdown=false in order to not spread samples to extra ingesters during rolling restarts with consistent naming. (default true) + -distributor.extra-query-delay duration + Time to wait before sending more than the minimum successful query requests. + -distributor.ha-tracker.cluster string + Prometheus label to look for in samples to identify a Prometheus HA cluster. (default "cluster") + -distributor.ha-tracker.consul.acl-token string + ACL Token used to interact with Consul. + -distributor.ha-tracker.consul.client-timeout duration + HTTP timeout when talking to Consul (default 20s) + -distributor.ha-tracker.consul.consistent-reads + Enable consistent reads to Consul. + -distributor.ha-tracker.consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -distributor.ha-tracker.consul.watch-burst-size int + Burst size used in rate limit. Values less than 1 are treated as 1. (default 1) + -distributor.ha-tracker.consul.watch-rate-limit float + Rate limit when watching key or prefix in Consul, in requests per second. 0 disables the rate limit. (default 1) + -distributor.ha-tracker.enable + Enable the distributors HA tracker so that it can accept samples from Prometheus HA replicas gracefully (requires labels). + -distributor.ha-tracker.enable-for-all-users + Flag to enable, for all users, handling of samples with external labels identifying replicas in an HA Prometheus setup. + -distributor.ha-tracker.etcd.dial-timeout duration + The dial timeout for the etcd connection. (default 10s) + -distributor.ha-tracker.etcd.endpoints value + The etcd endpoints to connect to. + -distributor.ha-tracker.etcd.max-retries int + The maximum number of retries to do for failed ops. (default 10) + -distributor.ha-tracker.etcd.password string + Etcd password. + -distributor.ha-tracker.etcd.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -distributor.ha-tracker.etcd.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -distributor.ha-tracker.etcd.tls-enabled + Enable TLS. + -distributor.ha-tracker.etcd.tls-insecure-skip-verify + Skip validating server certificate. + -distributor.ha-tracker.etcd.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -distributor.ha-tracker.etcd.tls-server-name string + Override the expected name on the server certificate. + -distributor.ha-tracker.etcd.username string + Etcd username. + -distributor.ha-tracker.failover-timeout duration + If we don't receive any samples from the accepted replica for a cluster in this amount of time we will failover to the next replica we receive a sample from. This value must be greater than the update timeout (default 30s) + -distributor.ha-tracker.max-clusters int + Maximum number of clusters that HA tracker will keep track of for single user. 0 to disable the limit. + -distributor.ha-tracker.multi.mirror-enabled + Mirror writes to secondary store. + -distributor.ha-tracker.multi.mirror-timeout duration + Timeout for storing value to secondary store. (default 2s) + -distributor.ha-tracker.multi.primary string + Primary backend storage used by multi-client. + -distributor.ha-tracker.multi.secondary string + Secondary backend storage used by multi-client. + -distributor.ha-tracker.prefix string + The prefix for the keys in the store. Should end with a /. (default "ha-tracker/") + -distributor.ha-tracker.replica string + Prometheus label to look for in samples to identify a Prometheus HA replica. (default "__replica__") + -distributor.ha-tracker.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "consul") + -distributor.ha-tracker.update-timeout duration + Update the timestamp in the KV store for a given cluster/replica only after this amount of time has passed since the current stored timestamp. (default 15s) + -distributor.ha-tracker.update-timeout-jitter-max duration + Maximum jitter applied to the update timeout, in order to spread the HA heartbeats over time. (default 5s) + -distributor.health-check-ingesters + Run a health check on each ingester client during periodic cleanup. (default true) + -distributor.ingestion-burst-size int + Per-user allowed ingestion burst size (in number of samples). (default 200000) + -distributor.ingestion-rate-limit float + Per-user ingestion rate limit in samples per second. (default 10000) + -distributor.ingestion-tenant-shard-size int + The tenant's shard size used by shuffle-sharding. Must be set both on ingesters and distributors. 0 disables shuffle sharding. + -distributor.instance-limits.max-inflight-push-requests int + Max inflight push requests that this distributor can handle. This limit is per-distributor, not per-tenant. Additional requests will be rejected. 0 = unlimited. (default 2000) + -distributor.instance-limits.max-ingestion-rate float + Max ingestion rate (samples/sec) that this distributor will accept. This limit is per-distributor, not per-tenant. Additional push requests will be rejected. Current ingestion rate is computed as exponentially weighted moving average, updated every second. 0 = unlimited. + -distributor.max-recv-msg-size int + remote_write API max receive message size (bytes). (default 104857600) + -distributor.remote-timeout duration + Timeout for downstream ingesters. (default 20s) + -distributor.replication-factor int + The number of ingesters to write to and read from. (default 3) + -distributor.ring.consul.acl-token string + ACL Token used to interact with Consul. + -distributor.ring.consul.client-timeout duration + HTTP timeout when talking to Consul (default 20s) + -distributor.ring.consul.consistent-reads + Enable consistent reads to Consul. + -distributor.ring.consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -distributor.ring.consul.watch-burst-size int + Burst size used in rate limit. Values less than 1 are treated as 1. (default 1) + -distributor.ring.consul.watch-rate-limit float + Rate limit when watching key or prefix in Consul, in requests per second. 0 disables the rate limit. (default 1) + -distributor.ring.etcd.dial-timeout duration + The dial timeout for the etcd connection. (default 10s) + -distributor.ring.etcd.endpoints value + The etcd endpoints to connect to. + -distributor.ring.etcd.max-retries int + The maximum number of retries to do for failed ops. (default 10) + -distributor.ring.etcd.password string + Etcd password. + -distributor.ring.etcd.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -distributor.ring.etcd.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -distributor.ring.etcd.tls-enabled + Enable TLS. + -distributor.ring.etcd.tls-insecure-skip-verify + Skip validating server certificate. + -distributor.ring.etcd.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -distributor.ring.etcd.tls-server-name string + Override the expected name on the server certificate. + -distributor.ring.etcd.username string + Etcd username. + -distributor.ring.heartbeat-period duration + Period at which to heartbeat to the ring. 0 = disabled. (default 5s) + -distributor.ring.heartbeat-timeout duration + The heartbeat timeout after which distributors are considered unhealthy within the ring. 0 = never (timeout disabled). (default 1m0s) + -distributor.ring.instance-addr string + IP address to advertise in the ring. + -distributor.ring.instance-id string + Instance ID to register in the ring. (default "{{.Hostname}}") + -distributor.ring.instance-interface-names value + Name of network interface to read address from. (default [eth0 en0]) + -distributor.ring.instance-port int + Port to advertise in the ring (defaults to server.grpc-listen-port). + -distributor.ring.multi.mirror-enabled + Mirror writes to secondary store. + -distributor.ring.multi.mirror-timeout duration + Timeout for storing value to secondary store. (default 2s) + -distributor.ring.multi.primary string + Primary backend storage used by multi-client. + -distributor.ring.multi.secondary string + Secondary backend storage used by multi-client. + -distributor.ring.prefix string + The prefix for the keys in the store. Should end with a /. (default "collectors/") + -distributor.ring.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "memberlist") + -distributor.zone-awareness-enabled + True to enable the zone-awareness and replicate ingested samples across different availability zones. + -etcd.dial-timeout duration + The dial timeout for the etcd connection. (default 10s) + -etcd.endpoints value + The etcd endpoints to connect to. + -etcd.max-retries int + The maximum number of retries to do for failed ops. (default 10) + -etcd.password string + Etcd password. + -etcd.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -etcd.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -etcd.tls-enabled + Enable TLS. + -etcd.tls-insecure-skip-verify + Skip validating server certificate. + -etcd.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -etcd.tls-server-name string + Override the expected name on the server certificate. + -etcd.username string + Etcd username. + -flusher.exit-after-flush + Stop after flush has finished. If false, process will keep running, doing nothing. (default true) + -frontend.align-querier-with-step + Mutate incoming queries to align their start and end with their step. + -frontend.cache-results + Cache query results. + -frontend.cache-unaligned-requests + Cache requests that are not step-aligned. + -frontend.downstream-url string + URL of downstream Prometheus. + -frontend.grpc-client-config.backoff-max-period duration + Maximum delay when backing off. (default 10s) + -frontend.grpc-client-config.backoff-min-period duration + Minimum delay when backing off. (default 100ms) + -frontend.grpc-client-config.backoff-on-ratelimits + Enable backoff and retry when we hit ratelimits. + -frontend.grpc-client-config.backoff-retries int + Number of times to backoff and retry before failing. (default 10) + -frontend.grpc-client-config.grpc-client-rate-limit float + Rate limit for gRPC client; 0 means disabled. + -frontend.grpc-client-config.grpc-client-rate-limit-burst int + Rate limit burst for gRPC client. + -frontend.grpc-client-config.grpc-compression string + Use compression when sending messages. Supported values are: 'gzip', 'snappy' and '' (disable compression) + -frontend.grpc-client-config.grpc-max-recv-msg-size int + gRPC client max receive message size (bytes). (default 104857600) + -frontend.grpc-client-config.grpc-max-send-msg-size int + gRPC client max send message size (bytes). (default 104857600) + -frontend.grpc-client-config.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -frontend.grpc-client-config.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -frontend.grpc-client-config.tls-enabled + Enable TLS in the GRPC client. This flag needs to be enabled when any other TLS flag is set. If set to false, insecure connection to gRPC server will be used. + -frontend.grpc-client-config.tls-insecure-skip-verify + Skip validating server certificate. + -frontend.grpc-client-config.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -frontend.grpc-client-config.tls-server-name string + Override the expected name on the server certificate. + -frontend.instance-addr string + IP address to advertise to querier (via scheduler) (resolved via interfaces by default). + -frontend.instance-interface-names value + Name of network interface to read address from. This address is sent to query-scheduler and querier, which uses it to send the query response back to query-frontend. (default [eth0 en0]) + -frontend.instance-port int + Port to advertise to querier (via scheduler) (defaults to server.grpc-listen-port). + -frontend.log-queries-longer-than duration + Log queries that are slower than the specified duration. Set to 0 to disable. Set to < 0 to enable on all queries. + -frontend.max-body-size int + Max body size for downstream prometheus. (default 10485760) + -frontend.max-cache-freshness value + Most recent allowed cacheable result per-tenant, to prevent caching very recent results that might still be in flux. (default 1m) + -frontend.max-queriers-per-tenant int + Maximum number of queriers that can handle requests for a single tenant. If set to 0 or value higher than number of available queriers, *all* queriers will handle requests for the tenant. Each frontend (or query-scheduler, if used) will select the same set of queriers for the same tenant (given that all queriers are connected to all frontends / query-schedulers). This option only works with queriers connecting to the query-frontend / query-scheduler, not when using downstream URL. + -frontend.max-retries-per-request int + Maximum number of retries for a single request; beyond this, the downstream error is returned. (default 5) + -frontend.parallelize-shardable-queries + True to enable query sharding. + -frontend.query-sharding-max-sharded-queries int + The max number of sharded queries that can be run for a given received query. 0 to disable limit. (default 128) + -frontend.query-sharding-total-shards int + The amount of shards to use when doing parallelisation via query sharding by tenant. 0 to disable query sharding for tenant. Query sharding implementation will adjust the number of query shards based on compactor shards. This allows querier to not search the blocks which cannot possibly have the series for given query shard. (default 16) + -frontend.query-stats-enabled + False to disable query statistics tracking. When enabled, a message with some statistics is logged for every query. (default true) + -frontend.results-cache.backend string + Backend for query-frontend results cache, if not empty. Supported values: [memcached]. + -frontend.results-cache.compression string + Enable cache compression, if not empty. Supported values are: snappy. + -frontend.results-cache.memcached.addresses string + Comma separated list of memcached addresses. Supported prefixes are: dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup made after that). + -frontend.results-cache.memcached.max-async-buffer-size int + The maximum number of enqueued asynchronous operations allowed. (default 25000) + -frontend.results-cache.memcached.max-async-concurrency int + The maximum number of concurrent asynchronous operations can occur. (default 50) + -frontend.results-cache.memcached.max-get-multi-batch-size int + The maximum number of keys a single underlying get operation should run. If more keys are specified, internally keys are split into multiple batches and fetched concurrently, honoring the max concurrency. If set to 0, the max batch size is unlimited. (default 100) + -frontend.results-cache.memcached.max-get-multi-concurrency int + The maximum number of concurrent connections running get operations. If set to 0, concurrency is unlimited. (default 100) + -frontend.results-cache.memcached.max-idle-connections int + The maximum number of idle connections that will be maintained per address. (default 100) + -frontend.results-cache.memcached.max-item-size int + The maximum size of an item stored in memcached. Bigger items are not stored. If set to 0, no maximum size is enforced. (default 1048576) + -frontend.results-cache.memcached.timeout duration + The socket read/write timeout. (default 200ms) + -frontend.scheduler-address string + DNS hostname used for finding query-schedulers. + -frontend.scheduler-dns-lookup-period duration + How often to resolve the scheduler-address, in order to look for new query-scheduler instances. (default 10s) + -frontend.scheduler-worker-concurrency int + Number of concurrent workers forwarding queries to single query-scheduler. (default 5) + -frontend.split-queries-by-interval duration + Split queries by an interval and execute in parallel. You should use a multiple of 24 hours to optimize querying blocks. 0 to disable it. (default 24h0m0s) + -h + Print basic help. + -help + Print basic help. + -help-all + Print help, also including advanced and experimental parameters. + -http.alertmanager-http-prefix string + HTTP URL path under which the Alertmanager ui and api will be served. (default "/alertmanager") + -http.prefix string + HTTP path prefix for API. (default "/api/prom") + -http.prometheus-http-prefix string + HTTP URL path under which the Prometheus api will be served. (default "/prometheus") + -ingester.active-series-custom-trackers value + Additional active series metrics, matching the provided matchers. Matchers should be in form :, like 'foobar:{foo="bar"}'. Multiple matchers can be provided either providing the flag multiple times or providing multiple semicolon-separated values to a single flag. + -ingester.active-series-metrics-enabled + Enable tracking of active series and export them as metrics. (default true) + -ingester.active-series-metrics-idle-timeout duration + After what time a series is considered to be inactive. (default 10m0s) + -ingester.active-series-metrics-update-period duration + How often to update active series metrics. (default 1m0s) + -ingester.availability-zone string + The availability zone where this instance is running. + -ingester.client.backoff-max-period duration + Maximum delay when backing off. (default 10s) + -ingester.client.backoff-min-period duration + Minimum delay when backing off. (default 100ms) + -ingester.client.backoff-on-ratelimits + Enable backoff and retry when we hit ratelimits. + -ingester.client.backoff-retries int + Number of times to backoff and retry before failing. (default 10) + -ingester.client.grpc-client-rate-limit float + Rate limit for gRPC client; 0 means disabled. + -ingester.client.grpc-client-rate-limit-burst int + Rate limit burst for gRPC client. + -ingester.client.grpc-compression string + Use compression when sending messages. Supported values are: 'gzip', 'snappy' and '' (disable compression) + -ingester.client.grpc-max-recv-msg-size int + gRPC client max receive message size (bytes). (default 104857600) + -ingester.client.grpc-max-send-msg-size int + gRPC client max send message size (bytes). (default 104857600) + -ingester.client.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -ingester.client.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -ingester.client.tls-enabled + Enable TLS in the GRPC client. This flag needs to be enabled when any other TLS flag is set. If set to false, insecure connection to gRPC server will be used. + -ingester.client.tls-insecure-skip-verify + Skip validating server certificate. + -ingester.client.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -ingester.client.tls-server-name string + Override the expected name on the server certificate. + -ingester.exemplars-update-period duration + [experimental] Period with which to update per-user max exemplars. (default 15s) + -ingester.final-sleep duration + Duration to sleep for before exiting, to ensure metrics are scraped. + -ingester.heartbeat-period duration + Period at which to heartbeat to consul. 0 = disabled. (default 5s) + -ingester.ignore-series-limit-for-metric-names string + Comma-separated list of metric names, for which the -ingester.max-global-series-per-metric limit will be ignored. Does not affect the -ingester.max-global-series-per-user limit. + -ingester.instance-limits.max-inflight-push-requests int + Max inflight push requests that this ingester can handle (across all tenants). Additional requests will be rejected. 0 = unlimited. (default 30000) + -ingester.instance-limits.max-ingestion-rate float + Max ingestion rate (samples/sec) that ingester will accept. This limit is per-ingester, not per-tenant. Additional push requests will be rejected. Current ingestion rate is computed as exponentially weighted moving average, updated every second. 0 = unlimited. + -ingester.instance-limits.max-series int + Max series that this ingester can hold (across all tenants). Requests to create additional series will be rejected. 0 = unlimited. + -ingester.instance-limits.max-tenants int + Max users (also referred to as 'tenants') that this ingester can hold. Requests from additional users will be rejected. 0 = unlimited. + -ingester.join-after duration + Period to wait for a claim from another member; will join automatically after this. + -ingester.lifecycler.ID string + ID to register in the ring. (default "{{.Hostname}}") + -ingester.lifecycler.addr string + IP address to advertise in the ring. + -ingester.lifecycler.interface value + Name of network interface to read address from. (default [eth0 en0]) + -ingester.lifecycler.port int + port to advertise in consul (defaults to server.grpc-listen-port). + -ingester.max-global-exemplars-per-user int + [experimental] The maximum number of exemplars in memory, across the cluster. 0 to disable exemplars ingestion. + -ingester.max-global-metadata-per-metric int + The maximum number of metadata per metric, across the cluster. 0 to disable. + -ingester.max-global-metadata-per-user int + The maximum number of active metrics with metadata per user, across the cluster. 0 to disable. + -ingester.max-global-series-per-metric int + The maximum number of active series per metric name, across the cluster before replication. 0 to disable. (default 20000) + -ingester.max-global-series-per-user int + The maximum number of active series per user, across the cluster before replication. 0 to disable. (default 150000) + -ingester.metadata-retain-period duration + Period at which metadata we have not seen will remain in memory before being deleted. (default 10m0s) + -ingester.min-ready-duration duration + Minimum duration to wait after the internal readiness checks have passed but before succeeding the readiness endpoint. This is used to slowdown deployment controllers (eg. Kubernetes) after an instance is ready and before they proceed with a rolling update, to give the rest of the cluster instances enough time to receive ring updates. (default 15s) + -ingester.num-tokens int + Number of tokens for each ingester. (default 128) + -ingester.observe-period duration + Observe tokens after generating to resolve collisions. Useful when using gossiping ring. + -ingester.rate-update-period duration + Period with which to update the per-user ingestion rates. (default 15s) + -ingester.readiness-check-ring-health + When enabled the readiness probe succeeds only after all instances are ACTIVE and healthy in the ring, otherwise only the instance itself is checked. This option should be disabled if in your cluster multiple instances can be rolled out simultaneously, otherwise rolling updates may be slowed down. (default true) + -ingester.stream-chunks-when-using-blocks + Stream chunks from ingesters to queriers. (default true) + -ingester.tokens-file-path string + File path where tokens are stored. If empty, tokens are not stored at shutdown and restored at startup. + -ingester.unregister-on-shutdown + Unregister from the ring upon clean shutdown. It can be useful to disable for rolling restarts with consistent naming in conjunction with -distributor.extend-writes=false. (default true) + -log.format value + Output log messages in the given format. Valid formats: [logfmt, json] (default logfmt) + -log.level value + Only log messages with the given severity or above. Valid levels: [debug, info, warn, error] (default info) + -mem-ballast-size-bytes int + Size of memory ballast to allocate. + -memberlist.abort-if-join-fails + If this node fails to join memberlist cluster, abort. (default true) + -memberlist.advertise-addr string + Gossip address to advertise to other members in the cluster. Used for NAT traversal. + -memberlist.advertise-port int + Gossip port to advertise to other members in the cluster. Used for NAT traversal. (default 7946) + -memberlist.bind-addr value + IP address to listen on for gossip messages. Multiple addresses may be specified. Defaults to 0.0.0.0 + -memberlist.bind-port int + Port to listen on for gossip messages. (default 7946) + -memberlist.compression-enabled + Enable message compression. This can be used to reduce bandwidth usage at the cost of slightly more CPU utilization. (default true) + -memberlist.dead-node-reclaim-time duration + How soon can dead node's name be reclaimed with new address. 0 to disable. + -memberlist.gossip-interval duration + How often to gossip. (default 200ms) + -memberlist.gossip-nodes int + How many nodes to gossip to. (default 3) + -memberlist.gossip-to-dead-nodes-time duration + How long to keep gossiping to dead nodes, to give them chance to refute their death. (default 30s) + -memberlist.join value + Other cluster members to join. Can be specified multiple times. It can be an IP, hostname or an entry specified in the DNS Service Discovery format. + -memberlist.leave-timeout duration + Timeout for leaving memberlist cluster. (default 5s) + -memberlist.left-ingesters-timeout duration + How long to keep LEFT ingesters in the ring. (default 5m0s) + -memberlist.max-join-backoff duration + Max backoff duration to join other cluster members. (default 1m0s) + -memberlist.max-join-retries int + Max number of retries to join other cluster members. (default 10) + -memberlist.message-history-buffer-bytes int + How much space to use for keeping received and sent messages in memory for troubleshooting (two buffers). 0 to disable. + -memberlist.min-join-backoff duration + Min backoff duration to join other cluster members. (default 1s) + -memberlist.nodename string + Name of the node in memberlist cluster. Defaults to hostname. + -memberlist.packet-dial-timeout duration + Timeout used when connecting to other nodes to send packet. (default 5s) + -memberlist.packet-write-timeout duration + Timeout for writing 'packet' data. (default 5s) + -memberlist.pullpush-interval duration + How often to use pull/push sync. (default 30s) + -memberlist.randomize-node-name + Add random suffix to the node name. (default true) + -memberlist.rejoin-interval duration + If not 0, how often to rejoin the cluster. Occasional rejoin can help to fix the cluster split issue, and is harmless otherwise. For example when using only few components as a seed nodes (via -memberlist.join), then it's recommended to use rejoin. If -memberlist.join points to dynamic service that resolves to all gossiping nodes (eg. Kubernetes headless service), then rejoin is not needed. + -memberlist.retransmit-factor int + Multiplication factor used when sending out messages (factor * log(N+1)). (default 4) + -memberlist.stream-timeout duration + The timeout for establishing a connection with a remote node, and for read/write operations. (default 10s) + -memberlist.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -memberlist.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -memberlist.tls-enabled + Enable TLS on the memberlist transport layer. + -memberlist.tls-insecure-skip-verify + Skip validating server certificate. + -memberlist.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -memberlist.tls-server-name string + Override the expected name on the server certificate. + -memberlist.transport-debug + Log debug transport messages. Note: global log.level must be at debug level as well. + -modules + List available values that can be used as target. + -multi.mirror-enabled + Mirror writes to secondary store. + -multi.mirror-timeout duration + Timeout for storing value to secondary store. (default 2s) + -multi.primary string + Primary backend storage used by multi-client. + -multi.secondary string + Secondary backend storage used by multi-client. + -print.config + Print the config and exit. + -querier.batch-iterators + Use batch iterators to execute query, as opposed to fully materialising the series in memory. Takes precedent over the -querier.iterators flag. (default true) + -querier.cardinality-analysis-enabled + Enables endpoints used for cardinality analysis. + -querier.default-evaluation-interval duration + The default evaluation interval or step size for subqueries. This config option should be set on query-frontend too when query sharding is enabled. (default 1m0s) + -querier.dns-lookup-period duration + How often to query DNS for query-frontend or query-scheduler address. (default 10s) + -querier.frontend-address string + Address of query frontend service, in host:port format. If -querier.scheduler-address is set as well, querier will use scheduler instead. Only one of -querier.frontend-address or -querier.scheduler-address can be set. If neither is set, queries are only received via HTTP endpoint. + -querier.frontend-client.backoff-max-period duration + Maximum delay when backing off. (default 10s) + -querier.frontend-client.backoff-min-period duration + Minimum delay when backing off. (default 100ms) + -querier.frontend-client.backoff-on-ratelimits + Enable backoff and retry when we hit ratelimits. + -querier.frontend-client.backoff-retries int + Number of times to backoff and retry before failing. (default 10) + -querier.frontend-client.grpc-client-rate-limit float + Rate limit for gRPC client; 0 means disabled. + -querier.frontend-client.grpc-client-rate-limit-burst int + Rate limit burst for gRPC client. + -querier.frontend-client.grpc-compression string + Use compression when sending messages. Supported values are: 'gzip', 'snappy' and '' (disable compression) + -querier.frontend-client.grpc-max-recv-msg-size int + gRPC client max receive message size (bytes). (default 104857600) + -querier.frontend-client.grpc-max-send-msg-size int + gRPC client max send message size (bytes). (default 104857600) + -querier.frontend-client.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -querier.frontend-client.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -querier.frontend-client.tls-enabled + Enable TLS in the GRPC client. This flag needs to be enabled when any other TLS flag is set. If set to false, insecure connection to gRPC server will be used. + -querier.frontend-client.tls-insecure-skip-verify + Skip validating server certificate. + -querier.frontend-client.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -querier.frontend-client.tls-server-name string + Override the expected name on the server certificate. + -querier.id string + Querier ID, sent to frontend service to identify requests from the same querier. Defaults to hostname. + -querier.iterators + Use iterators to execute query, as opposed to fully materialising the series in memory. + -querier.label-names-and-values-results-max-size-bytes int + Maximum size in bytes of distinct label names and values. When querier receives response from ingester, it merges the response with responses from other ingesters. This maximum size limit is applied to the merged(distinct) results. If the limit is reached, an error is returned. (default 419430400) + -querier.label-values-max-cardinality-label-names-per-request int + Maximum number of label names allowed to be queried in a single /api/v1/cardinality/label_values API call. (default 100) + -querier.lookback-delta duration + Time since the last sample after which a time series is considered stale and ignored by expression evaluations. This config option should be set on query-frontend too when query sharding is enabled. (default 5m0s) + -querier.max-concurrent int + The maximum number of concurrent queries. This config option should be set on query-frontend too when query sharding is enabled. (default 20) + -querier.max-fetched-chunk-bytes-per-query int + The maximum size of all chunks in bytes that a query can fetch from each ingester and storage. This limit is enforced in the querier and ruler. 0 to disable. + -querier.max-fetched-chunks-per-query int + Maximum number of chunks that can be fetched in a single query from ingesters and long-term storage. This limit is enforced in the querier, ruler and store-gateway. 0 to disable. (default 2000000) + -querier.max-fetched-series-per-query int + The maximum number of unique series for which a query can fetch samples from each ingesters and storage. This limit is enforced in the querier and ruler. 0 to disable + -querier.max-outstanding-requests-per-tenant int + Maximum number of outstanding requests per tenant per frontend; requests beyond this error with HTTP 429. (default 100) + -querier.max-query-into-future duration + Maximum duration into the future you can query. 0 to disable. (default 10m0s) + -querier.max-query-lookback value + Limit how long back data (series and metadata) can be queried, up until duration ago. This limit is enforced in the query-frontend, querier and ruler. If the requested time range is outside the allowed range, the request will not fail but will be manipulated to only query data within the allowed time range. 0 to disable. + -querier.max-query-parallelism int + Maximum number of split queries will be scheduled in parallel by the frontend. (default 14) + -querier.max-samples int + Maximum number of samples a single query can load into memory. This config option should be set on query-frontend too when query sharding is enabled. (default 50000000) + -querier.query-ingesters-within duration + Maximum lookback beyond which queries are not sent to ingester. 0 means all queries are sent to ingester. (default 13h0m0s) + -querier.query-label-names-with-matchers-enabled + True to enable queriers to use an optimized implementation which passes down to ingesters the label matchers when running the label names API. Can be enabled once all ingesters run a version >= the one where this option has been introduced. + -querier.query-store-after duration + The time after which a metric should be queried from storage and not just ingesters. 0 means all queries are sent to store. If this option is enabled, the time range of the query sent to the store-gateway will be manipulated to ensure the query end is not more recent than 'now - query-store-after'. + -querier.scheduler-address string + Hostname (and port) of scheduler that querier will periodically resolve, connect to and receive queries from. Only one of -querier.frontend-address or -querier.scheduler-address can be set. If neither is set, queries are only received via HTTP endpoint. + -querier.shuffle-sharding-ingesters-lookback-period duration + When distributor's sharding strategy is shuffle-sharding and this setting is > 0, queriers fetch in-memory series from the minimum set of required ingesters, selecting only ingesters which may have received series since 'now - lookback period'. The lookback period should be greater or equal than the configured -querier.query-store-after and -querier.query-ingesters-within. If this setting is 0, queriers always query all ingesters (ingesters shuffle sharding on read path is disabled). + -querier.store-gateway-client.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -querier.store-gateway-client.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -querier.store-gateway-client.tls-enabled + Enable TLS for gRPC client connecting to store-gateway. + -querier.store-gateway-client.tls-insecure-skip-verify + Skip validating server certificate. + -querier.store-gateway-client.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -querier.store-gateway-client.tls-server-name string + Override the expected name on the server certificate. + -querier.timeout duration + The timeout for a query. This config option should be set on query-frontend too when query sharding is enabled. (default 2m0s) + -query-frontend.querier-forget-delay duration + If a querier disconnects without sending notification about graceful shutdown, the query-frontend will keep the querier in the tenant's shard until the forget delay has passed. This feature is useful to reduce the blast radius when shuffle-sharding is enabled. + -query-scheduler.grpc-client-config.backoff-max-period duration + Maximum delay when backing off. (default 10s) + -query-scheduler.grpc-client-config.backoff-min-period duration + Minimum delay when backing off. (default 100ms) + -query-scheduler.grpc-client-config.backoff-on-ratelimits + Enable backoff and retry when we hit ratelimits. + -query-scheduler.grpc-client-config.backoff-retries int + Number of times to backoff and retry before failing. (default 10) + -query-scheduler.grpc-client-config.grpc-client-rate-limit float + Rate limit for gRPC client; 0 means disabled. + -query-scheduler.grpc-client-config.grpc-client-rate-limit-burst int + Rate limit burst for gRPC client. + -query-scheduler.grpc-client-config.grpc-compression string + Use compression when sending messages. Supported values are: 'gzip', 'snappy' and '' (disable compression) + -query-scheduler.grpc-client-config.grpc-max-recv-msg-size int + gRPC client max receive message size (bytes). (default 104857600) + -query-scheduler.grpc-client-config.grpc-max-send-msg-size int + gRPC client max send message size (bytes). (default 104857600) + -query-scheduler.grpc-client-config.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -query-scheduler.grpc-client-config.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -query-scheduler.grpc-client-config.tls-enabled + Enable TLS in the GRPC client. This flag needs to be enabled when any other TLS flag is set. If set to false, insecure connection to gRPC server will be used. + -query-scheduler.grpc-client-config.tls-insecure-skip-verify + Skip validating server certificate. + -query-scheduler.grpc-client-config.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -query-scheduler.grpc-client-config.tls-server-name string + Override the expected name on the server certificate. + -query-scheduler.max-outstanding-requests-per-tenant int + Maximum number of outstanding requests per tenant per query-scheduler. In-flight requests above this limit will fail with HTTP response status code 429. (default 100) + -query-scheduler.querier-forget-delay duration + If a querier disconnects without sending notification about graceful shutdown, the query-scheduler will keep the querier in the tenant's shard until the forget delay has passed. This feature is useful to reduce the blast radius when shuffle-sharding is enabled. + -ring.heartbeat-timeout duration + The heartbeat timeout after which ingesters are skipped for reads/writes. 0 = never (timeout disabled). (default 1m0s) + -ring.prefix string + The prefix for the keys in the store. Should end with a /. (default "collectors/") + -ring.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "memberlist") + -ruler-storage.azure.account-key value + Azure storage account key + -ruler-storage.azure.account-name string + Azure storage account name + -ruler-storage.azure.container-name string + Azure storage container name + -ruler-storage.azure.endpoint-suffix string + Azure storage endpoint suffix without schema. The account name will be prefixed to this value to create the FQDN. If set to empty string, default endpoint suffix is used. + -ruler-storage.azure.max-retries int + Number of retries for recoverable errors (default 20) + -ruler-storage.azure.msi-resource string + If set, this URL is used instead of https://. for obtaining ServicePrincipalToken from MSI. + -ruler-storage.azure.user-assigned-id string + User assigned identity. If empty, then System assigned identity is used. + -ruler-storage.backend string + Backend storage to use. Supported backends are: s3, gcs, azure, swift, filesystem, local. (default "filesystem") + -ruler-storage.filesystem.dir string + Local filesystem storage directory. (default "ruler") + -ruler-storage.gcs.bucket-name string + GCS bucket name + -ruler-storage.gcs.service-account value + JSON representing either a Google Developers Console client_credentials.json file or a Google Developers service account key file. If empty, fallback to Google default logic. + -ruler-storage.local.directory string + Directory to scan for rules + -ruler-storage.s3.access-key-id string + S3 access key ID + -ruler-storage.s3.bucket-name string + S3 bucket name + -ruler-storage.s3.endpoint string + The S3 bucket endpoint. It could be an AWS S3 endpoint listed at https://docs.aws.amazon.com/general/latest/gr/s3.html or the address of an S3-compatible service in hostname:port format. + -ruler-storage.s3.expect-continue-timeout duration + The time to wait for a server's first response headers after fully writing the request headers if the request has an Expect header. 0 to send the request body immediately. (default 1s) + -ruler-storage.s3.http.idle-conn-timeout duration + The time an idle connection will remain idle before closing. (default 1m30s) + -ruler-storage.s3.http.insecure-skip-verify + If the client connects to S3 via HTTPS and this option is enabled, the client will accept any certificate and hostname. + -ruler-storage.s3.http.response-header-timeout duration + The amount of time the client will wait for a servers response headers. (default 2m0s) + -ruler-storage.s3.insecure + If enabled, use http:// for the S3 endpoint instead of https://. This could be useful in local dev/test environments while using an S3-compatible backend storage, like Minio. + -ruler-storage.s3.max-connections-per-host int + Maximum number of connections per host. 0 means no limit. + -ruler-storage.s3.max-idle-connections int + Maximum number of idle (keep-alive) connections across all hosts. 0 means no limit. (default 100) + -ruler-storage.s3.max-idle-connections-per-host int + Maximum number of idle (keep-alive) connections to keep per-host. If 0, a built-in default value is used. (default 100) + -ruler-storage.s3.region string + S3 region. If unset, the client will issue a S3 GetBucketLocation API call to autodetect it. + -ruler-storage.s3.secret-access-key value + S3 secret access key + -ruler-storage.s3.signature-version string + The signature version to use for authenticating against S3. Supported values are: v4, v2. (default "v4") + -ruler-storage.s3.sse.kms-encryption-context string + KMS Encryption Context used for object encryption. It expects JSON formatted string. + -ruler-storage.s3.sse.kms-key-id string + KMS Key ID used to encrypt objects in S3 + -ruler-storage.s3.sse.type string + Enable AWS Server Side Encryption. Supported values: SSE-KMS, SSE-S3. + -ruler-storage.s3.tls-handshake-timeout duration + Maximum time to wait for a TLS handshake. 0 means no limit. (default 10s) + -ruler-storage.swift.auth-url string + OpenStack Swift authentication URL + -ruler-storage.swift.auth-version int + OpenStack Swift authentication API version. 0 to autodetect. + -ruler-storage.swift.connect-timeout duration + Time after which a connection attempt is aborted. (default 10s) + -ruler-storage.swift.container-name string + Name of the OpenStack Swift container to put chunks in. + -ruler-storage.swift.domain-id string + OpenStack Swift user's domain ID. + -ruler-storage.swift.domain-name string + OpenStack Swift user's domain name. + -ruler-storage.swift.max-retries int + Max retries on requests error. (default 3) + -ruler-storage.swift.password string + OpenStack Swift API key. + -ruler-storage.swift.project-domain-id string + ID of the OpenStack Swift project's domain (v3 auth only), only needed if it differs the from user domain. + -ruler-storage.swift.project-domain-name string + Name of the OpenStack Swift project's domain (v3 auth only), only needed if it differs from the user domain. + -ruler-storage.swift.project-id string + OpenStack Swift project ID (v2,v3 auth only). + -ruler-storage.swift.project-name string + OpenStack Swift project name (v2,v3 auth only). + -ruler-storage.swift.region-name string + OpenStack Swift Region to use (v2,v3 auth only). + -ruler-storage.swift.request-timeout duration + Time after which an idle request is aborted. The timeout watchdog is reset each time some data is received, so the timeout triggers after X time no data is received on a request. (default 5s) + -ruler-storage.swift.user-domain-id string + OpenStack Swift user's domain ID. + -ruler-storage.swift.user-domain-name string + OpenStack Swift user's domain name. + -ruler-storage.swift.user-id string + OpenStack Swift user ID. + -ruler-storage.swift.username string + OpenStack Swift username. + -ruler.alertmanager-client.basic-auth-password string + HTTP Basic authentication password. It overrides the password set in the URL (if any). + -ruler.alertmanager-client.basic-auth-username string + HTTP Basic authentication username. It overrides the username set in the URL (if any). + -ruler.alertmanager-client.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -ruler.alertmanager-client.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -ruler.alertmanager-client.tls-insecure-skip-verify + Skip validating server certificate. + -ruler.alertmanager-client.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -ruler.alertmanager-client.tls-server-name string + Override the expected name on the server certificate. + -ruler.alertmanager-refresh-interval duration + How long to wait between refreshing DNS resolutions of Alertmanager hosts. (default 1m0s) + -ruler.alertmanager-url string + Comma-separated list of URL(s) of the Alertmanager(s) to send notifications to. Each URL is treated as a separate group. Multiple Alertmanagers in HA per group can be supported by using DNS service discovery format. Basic auth is supported as part of the URL. + -ruler.client.backoff-max-period duration + Maximum delay when backing off. (default 10s) + -ruler.client.backoff-min-period duration + Minimum delay when backing off. (default 100ms) + -ruler.client.backoff-on-ratelimits + Enable backoff and retry when we hit ratelimits. + -ruler.client.backoff-retries int + Number of times to backoff and retry before failing. (default 10) + -ruler.client.grpc-client-rate-limit float + Rate limit for gRPC client; 0 means disabled. + -ruler.client.grpc-client-rate-limit-burst int + Rate limit burst for gRPC client. + -ruler.client.grpc-compression string + Use compression when sending messages. Supported values are: 'gzip', 'snappy' and '' (disable compression) + -ruler.client.grpc-max-recv-msg-size int + gRPC client max receive message size (bytes). (default 104857600) + -ruler.client.grpc-max-send-msg-size int + gRPC client max send message size (bytes). (default 104857600) + -ruler.client.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -ruler.client.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -ruler.client.tls-enabled + Enable TLS in the GRPC client. This flag needs to be enabled when any other TLS flag is set. If set to false, insecure connection to gRPC server will be used. + -ruler.client.tls-insecure-skip-verify + Skip validating server certificate. + -ruler.client.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -ruler.client.tls-server-name string + Override the expected name on the server certificate. + -ruler.disabled-tenants value + Comma separated list of tenants whose rules this ruler cannot evaluate. If specified, a ruler that would normally pick the specified tenant(s) for processing will ignore them instead. Subject to sharding. + -ruler.enable-api + Enable the ruler config API. (default true) + -ruler.enabled-tenants value + Comma separated list of tenants whose rules this ruler can evaluate. If specified, only these tenants will be handled by ruler, otherwise this ruler can process rules from all tenants. Subject to sharding. + -ruler.evaluation-delay-duration value + Duration to delay the evaluation of rules to ensure the underlying metrics have been pushed. + -ruler.evaluation-interval duration + How frequently to evaluate rules (default 1m0s) + -ruler.external.url value + URL of alerts return path. + -ruler.flush-period duration + Period with which to attempt to flush rule groups. (default 1m0s) + -ruler.for-grace-period duration + Minimum duration between alert and restored "for" state. This is maintained only for alerts with configured "for" time greater than grace period. (default 10m0s) + -ruler.for-outage-tolerance duration + Max time to tolerate outage for restoring "for" state of alert. (default 1h0m0s) + -ruler.max-rule-groups-per-tenant int + Maximum number of rule groups per-tenant. 0 to disable. (default 70) + -ruler.max-rules-per-rule-group int + Maximum number of rules per rule group per-tenant. 0 to disable. (default 20) + -ruler.notification-queue-capacity int + Capacity of the queue for notifications to be sent to the Alertmanager. (default 10000) + -ruler.notification-timeout duration + HTTP timeout duration when sending notifications to the Alertmanager. (default 10s) + -ruler.poll-interval duration + How frequently to poll for rule changes (default 1m0s) + -ruler.query-stats-enabled + Report the wall time for ruler queries to complete as a per user metric and as an info level log message. + -ruler.resend-delay duration + Minimum amount of time to wait before resending an alert to Alertmanager. (default 1m0s) + -ruler.ring.consul.acl-token string + ACL Token used to interact with Consul. + -ruler.ring.consul.client-timeout duration + HTTP timeout when talking to Consul (default 20s) + -ruler.ring.consul.consistent-reads + Enable consistent reads to Consul. + -ruler.ring.consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -ruler.ring.consul.watch-burst-size int + Burst size used in rate limit. Values less than 1 are treated as 1. (default 1) + -ruler.ring.consul.watch-rate-limit float + Rate limit when watching key or prefix in Consul, in requests per second. 0 disables the rate limit. (default 1) + -ruler.ring.etcd.dial-timeout duration + The dial timeout for the etcd connection. (default 10s) + -ruler.ring.etcd.endpoints value + The etcd endpoints to connect to. + -ruler.ring.etcd.max-retries int + The maximum number of retries to do for failed ops. (default 10) + -ruler.ring.etcd.password string + Etcd password. + -ruler.ring.etcd.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -ruler.ring.etcd.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -ruler.ring.etcd.tls-enabled + Enable TLS. + -ruler.ring.etcd.tls-insecure-skip-verify + Skip validating server certificate. + -ruler.ring.etcd.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -ruler.ring.etcd.tls-server-name string + Override the expected name on the server certificate. + -ruler.ring.etcd.username string + Etcd username. + -ruler.ring.heartbeat-period duration + Period at which to heartbeat to the ring. 0 = disabled. (default 5s) + -ruler.ring.heartbeat-timeout duration + The heartbeat timeout after which rulers are considered unhealthy within the ring. 0 = never (timeout disabled). (default 1m0s) + -ruler.ring.instance-addr string + IP address to advertise in the ring. + -ruler.ring.instance-id string + Instance ID to register in the ring. (default "{{.Hostname}}") + -ruler.ring.instance-interface-names value + Name of network interface to read address from. (default [eth0 en0]) + -ruler.ring.instance-port int + Port to advertise in the ring (defaults to server.grpc-listen-port). + -ruler.ring.multi.mirror-enabled + Mirror writes to secondary store. + -ruler.ring.multi.mirror-timeout duration + Timeout for storing value to secondary store. (default 2s) + -ruler.ring.multi.primary string + Primary backend storage used by multi-client. + -ruler.ring.multi.secondary string + Secondary backend storage used by multi-client. + -ruler.ring.num-tokens int + Number of tokens for each ruler. (default 128) + -ruler.ring.prefix string + The prefix for the keys in the store. Should end with a /. (default "rulers/") + -ruler.ring.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "memberlist") + -ruler.rule-path string + Directory to store temporary rule files loaded by the Prometheus rule managers. This directory is not required to be persisted between restarts. (default "./data-ruler/") + -ruler.search-pending-for duration + Time to spend searching for a pending ruler when shutting down. (default 5m0s) + -ruler.tenant-federation.enabled + Enable running rule groups against multiple tenants. The tenant IDs involved need to be in the rule group's 'source_tenants' field. If this flag is set to 'false' when there are already created federated rule groups, then these rules groups will be skipped during evaluations. + -ruler.tenant-shard-size int + The tenant's shard size when sharding is used by ruler. Value of 0 disables shuffle sharding for the tenant, and tenant rules will be sharded across all ruler replicas. + -runtime-config.file string + File with the configuration that can be updated in runtime. + -runtime-config.reload-period duration + How often to check runtime config file. (default 10s) + -server.graceful-shutdown-timeout duration + Timeout for graceful shutdowns (default 30s) + -server.grpc-conn-limit int + Maximum number of simultaneous grpc connections, <=0 to disable + -server.grpc-listen-address string + gRPC server listen address. + -server.grpc-listen-network string + gRPC server listen network (default "tcp") + -server.grpc-listen-port int + gRPC server listen port. (default 9095) + -server.grpc-max-concurrent-streams uint + Limit on the number of concurrent streams for gRPC calls (0 = unlimited) (default 100) + -server.grpc-max-recv-msg-size-bytes int + Limit on the size of a gRPC message this server can receive (bytes). (default 4194304) + -server.grpc-max-send-msg-size-bytes int + Limit on the size of a gRPC message this server can send (bytes). (default 4194304) + -server.grpc-tls-ca-path string + GRPC TLS Client CA path. + -server.grpc-tls-cert-path string + GRPC TLS server cert path. + -server.grpc-tls-client-auth string + GRPC TLS Client Auth type. + -server.grpc-tls-key-path string + GRPC TLS server key path. + -server.grpc.keepalive.max-connection-age duration + The duration for the maximum amount of time a connection may exist before it will be closed. Default: infinity (default 2562047h47m16.854775807s) + -server.grpc.keepalive.max-connection-age-grace duration + An additive period after max-connection-age after which the connection will be forcibly closed. Default: infinity (default 2562047h47m16.854775807s) + -server.grpc.keepalive.max-connection-idle duration + The duration after which an idle connection should be closed. Default: infinity (default 2562047h47m16.854775807s) + -server.grpc.keepalive.min-time-between-pings duration + Minimum amount of time a client should wait before sending a keepalive ping. If client sends keepalive ping more often, server will send GOAWAY and close the connection. (default 10s) + -server.grpc.keepalive.ping-without-stream-allowed + If true, server allows keepalive pings even when there are no active streams(RPCs). If false, and client sends ping when there are no active streams, server will send GOAWAY and close the connection. (default true) + -server.grpc.keepalive.time duration + Duration after which a keepalive probe is sent in case of no activity over the connection., Default: 2h (default 2h0m0s) + -server.grpc.keepalive.timeout duration + After having pinged for keepalive check, the duration after which an idle connection should be closed, Default: 20s (default 20s) + -server.http-conn-limit int + Maximum number of simultaneous http connections, <=0 to disable + -server.http-idle-timeout duration + Idle timeout for HTTP server (default 2m0s) + -server.http-listen-address string + HTTP server listen address. + -server.http-listen-network string + HTTP server listen network, default tcp (default "tcp") + -server.http-listen-port int + HTTP server listen port. (default 8080) + -server.http-read-timeout duration + Read timeout for HTTP server (default 30s) + -server.http-tls-ca-path string + HTTP TLS Client CA path. + -server.http-tls-cert-path string + HTTP server cert path. + -server.http-tls-client-auth string + HTTP TLS Client Auth type. + -server.http-tls-key-path string + HTTP server key path. + -server.http-write-timeout duration + Write timeout for HTTP server (default 30s) + -server.log-source-ips-enabled + Optionally log the source IPs. + -server.log-source-ips-header string + Header field storing the source IPs. Only used if server.log-source-ips-enabled is true. If not set the default Forwarded, X-Real-IP and X-Forwarded-For headers are used + -server.log-source-ips-regex string + Regex for matching the source IPs. Only used if server.log-source-ips-enabled is true. If not set the default Forwarded, X-Real-IP and X-Forwarded-For headers are used + -server.path-prefix string + Base path to serve all API routes from (e.g. /v1/) + -server.register-instrumentation + Register the intrumentation handlers (/metrics etc). (default true) + -store-gateway.sharding-ring.consul.acl-token string + ACL Token used to interact with Consul. + -store-gateway.sharding-ring.consul.client-timeout duration + HTTP timeout when talking to Consul (default 20s) + -store-gateway.sharding-ring.consul.consistent-reads + Enable consistent reads to Consul. + -store-gateway.sharding-ring.consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -store-gateway.sharding-ring.consul.watch-burst-size int + Burst size used in rate limit. Values less than 1 are treated as 1. (default 1) + -store-gateway.sharding-ring.consul.watch-rate-limit float + Rate limit when watching key or prefix in Consul, in requests per second. 0 disables the rate limit. (default 1) + -store-gateway.sharding-ring.etcd.dial-timeout duration + The dial timeout for the etcd connection. (default 10s) + -store-gateway.sharding-ring.etcd.endpoints value + The etcd endpoints to connect to. + -store-gateway.sharding-ring.etcd.max-retries int + The maximum number of retries to do for failed ops. (default 10) + -store-gateway.sharding-ring.etcd.password string + Etcd password. + -store-gateway.sharding-ring.etcd.tls-ca-path string + Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used. + -store-gateway.sharding-ring.etcd.tls-cert-path string + Path to the client certificate file, which will be used for authenticating with the server. Also requires the key path to be configured. + -store-gateway.sharding-ring.etcd.tls-enabled + Enable TLS. + -store-gateway.sharding-ring.etcd.tls-insecure-skip-verify + Skip validating server certificate. + -store-gateway.sharding-ring.etcd.tls-key-path string + Path to the key file for the client certificate. Also requires the client certificate to be configured. + -store-gateway.sharding-ring.etcd.tls-server-name string + Override the expected name on the server certificate. + -store-gateway.sharding-ring.etcd.username string + Etcd username. + -store-gateway.sharding-ring.heartbeat-period duration + Period at which to heartbeat to the ring. 0 = disabled. (default 15s) + -store-gateway.sharding-ring.heartbeat-timeout duration + The heartbeat timeout after which store gateways are considered unhealthy within the ring. 0 = never (timeout disabled). This option needs be set both on the store-gateway and querier when running in microservices mode. (default 1m0s) + -store-gateway.sharding-ring.instance-addr string + IP address to advertise in the ring. + -store-gateway.sharding-ring.instance-availability-zone string + The availability zone where this instance is running. Required if zone-awareness is enabled. + -store-gateway.sharding-ring.instance-id string + Instance ID to register in the ring. (default "{{.Hostname}}") + -store-gateway.sharding-ring.instance-interface-names value + Name of network interface to read address from. (default [eth0 en0]) + -store-gateway.sharding-ring.instance-port int + Port to advertise in the ring (defaults to server.grpc-listen-port). + -store-gateway.sharding-ring.multi.mirror-enabled + Mirror writes to secondary store. + -store-gateway.sharding-ring.multi.mirror-timeout duration + Timeout for storing value to secondary store. (default 2s) + -store-gateway.sharding-ring.multi.primary string + Primary backend storage used by multi-client. + -store-gateway.sharding-ring.multi.secondary string + Secondary backend storage used by multi-client. + -store-gateway.sharding-ring.prefix string + The prefix for the keys in the store. Should end with a /. (default "collectors/") + -store-gateway.sharding-ring.replication-factor int + The replication factor to use when sharding blocks. This option needs be set both on the store-gateway and querier when running in microservices mode. (default 3) + -store-gateway.sharding-ring.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "memberlist") + -store-gateway.sharding-ring.tokens-file-path string + File path where tokens are stored. If empty, tokens are not stored at shutdown and restored at startup. + -store-gateway.sharding-ring.unregister-on-shutdown + Unregister from the ring upon clean shutdown. (default true) + -store-gateway.sharding-ring.wait-stability-max-duration duration + Maximum time to wait for ring stability at startup. If the store-gateway ring keeps changing after this period of time, the store-gateway will start anyway. (default 5m0s) + -store-gateway.sharding-ring.wait-stability-min-duration duration + Minimum time to wait for ring stability at startup, if set to positive value. + -store-gateway.sharding-ring.zone-awareness-enabled + True to enable zone-awareness and replicate blocks across different availability zones. + -store-gateway.tenant-shard-size int + The tenant's shard size, used when store-gateway sharding is enabled. Value of 0 disables shuffle sharding for the tenant, that is all tenant blocks are sharded across all store-gateway replicas. + -store.max-labels-query-length value + Limit the time range (end - start time) of series, label names and values queries. This limit is enforced in the querier. If the requested time range is outside the allowed range, the request will not fail but will be manipulated to only query data within the allowed time range. 0 to disable. + -store.max-query-length value + Limit the query time range (end - start time). This limit is enforced in the query-frontend (on the received query), in the querier (on the query possibly split by the query-frontend) and ruler. 0 to disable. + -target value + Comma-separated list of modules to load. The alias 'all' can be used in the list to load a number of core modules and will enable single-binary mode. Use '-modules' command line flag to get a list of available modules, and to see which modules are included in 'all'. (default all) + -tenant-federation.enabled + If enabled on all services, queries can be federated across multiple tenants. The tenant IDs involved need to be specified separated by a '|' character in the 'X-Scope-OrgID' header. + -validation.create-grace-period value + Duration which table will be created/deleted before/after it's needed; we won't accept sample from before this time. (default 10m) + -validation.enforce-metadata-metric-name + Enforce every metadata has a metric name. (default true) + -validation.max-label-names-per-series int + Maximum number of label names per series. (default 30) + -validation.max-length-label-name int + Maximum length accepted for label names (default 1024) + -validation.max-length-label-value int + Maximum length accepted for label value. This setting also applies to the metric name (default 2048) + -validation.max-metadata-length int + Maximum length accepted for metric metadata. Metadata refers to Metric Name, HELP and UNIT. (default 1024) + -version + Print application version and exit. diff --git a/cmd/mimir/help.txt.tmpl b/cmd/mimir/help.txt.tmpl new file mode 100644 index 00000000000..3488cf80a7a --- /dev/null +++ b/cmd/mimir/help.txt.tmpl @@ -0,0 +1,627 @@ +Usage of ./cmd/mimir/mimir: + -activity-tracker.filepath string + File where ongoing activities are stored. If empty, activity tracking is disabled. (default "./metrics-activity.log") + -activity-tracker.max-entries int + Max number of concurrent activities that can be tracked. Used to size the file in advance. Additional activities are ignored. (default 1024) + -alertmanager-storage.azure.account-key value + Azure storage account key + -alertmanager-storage.azure.account-name string + Azure storage account name + -alertmanager-storage.azure.container-name string + Azure storage container name + -alertmanager-storage.azure.endpoint-suffix string + Azure storage endpoint suffix without schema. The account name will be prefixed to this value to create the FQDN. If set to empty string, default endpoint suffix is used. + -alertmanager-storage.backend string + Backend storage to use. Supported backends are: s3, gcs, azure, swift, filesystem, local. (default "filesystem") + -alertmanager-storage.filesystem.dir string + Local filesystem storage directory. (default "alertmanager") + -alertmanager-storage.gcs.bucket-name string + GCS bucket name + -alertmanager-storage.gcs.service-account value + JSON representing either a Google Developers Console client_credentials.json file or a Google Developers service account key file. If empty, fallback to Google default logic. + -alertmanager-storage.local.path string + Path at which alertmanager configurations are stored. + -alertmanager-storage.s3.access-key-id string + S3 access key ID + -alertmanager-storage.s3.bucket-name string + S3 bucket name + -alertmanager-storage.s3.endpoint string + The S3 bucket endpoint. It could be an AWS S3 endpoint listed at https://docs.aws.amazon.com/general/latest/gr/s3.html or the address of an S3-compatible service in hostname:port format. + -alertmanager-storage.s3.region string + S3 region. If unset, the client will issue a S3 GetBucketLocation API call to autodetect it. + -alertmanager-storage.s3.secret-access-key value + S3 secret access key + -alertmanager-storage.s3.sse.kms-encryption-context string + KMS Encryption Context used for object encryption. It expects JSON formatted string. + -alertmanager-storage.s3.sse.kms-key-id string + KMS Key ID used to encrypt objects in S3 + -alertmanager-storage.s3.sse.type string + Enable AWS Server Side Encryption. Supported values: SSE-KMS, SSE-S3. + -alertmanager-storage.swift.auth-url string + OpenStack Swift authentication URL + -alertmanager-storage.swift.auth-version int + OpenStack Swift authentication API version. 0 to autodetect. + -alertmanager-storage.swift.container-name string + Name of the OpenStack Swift container to put chunks in. + -alertmanager-storage.swift.domain-id string + OpenStack Swift user's domain ID. + -alertmanager-storage.swift.domain-name string + OpenStack Swift user's domain name. + -alertmanager-storage.swift.password string + OpenStack Swift API key. + -alertmanager-storage.swift.project-domain-id string + ID of the OpenStack Swift project's domain (v3 auth only), only needed if it differs the from user domain. + -alertmanager-storage.swift.project-domain-name string + Name of the OpenStack Swift project's domain (v3 auth only), only needed if it differs from the user domain. + -alertmanager-storage.swift.project-id string + OpenStack Swift project ID (v2,v3 auth only). + -alertmanager-storage.swift.project-name string + OpenStack Swift project name (v2,v3 auth only). + -alertmanager-storage.swift.region-name string + OpenStack Swift Region to use (v2,v3 auth only). + -alertmanager-storage.swift.user-domain-id string + OpenStack Swift user's domain ID. + -alertmanager-storage.swift.user-domain-name string + OpenStack Swift user's domain name. + -alertmanager-storage.swift.user-id string + OpenStack Swift user ID. + -alertmanager-storage.swift.username string + OpenStack Swift username. + -alertmanager.configs.fallback string + Filename of fallback config to use if none specified for instance. + -alertmanager.max-alerts-count int + Maximum number of alerts that a single user can have. Inserting more alerts will fail with a log message and metric increment. 0 = no limit. + -alertmanager.max-alerts-size-bytes int + Maximum total size of alerts that a single user can have, alert size is the sum of the bytes of its labels, annotations and generatorURL. Inserting more alerts will fail with a log message and metric increment. 0 = no limit. + -alertmanager.max-config-size-bytes int + Maximum size of configuration file for Alertmanager that tenant can upload via Alertmanager API. 0 = no limit. + -alertmanager.max-dispatcher-aggregation-groups int + Maximum number of aggregation groups in Alertmanager's dispatcher that a tenant can have. Each active aggregation group uses single goroutine. When the limit is reached, dispatcher will not dispatch alerts that belong to additional aggregation groups, but existing groups will keep working properly. 0 = no limit. + -alertmanager.max-template-size-bytes int + Maximum size of single template in tenant's Alertmanager configuration uploaded via Alertmanager API. 0 = no limit. + -alertmanager.max-templates-count int + Maximum number of templates in tenant's Alertmanager configuration uploaded via Alertmanager API. 0 = no limit. + -alertmanager.notification-rate-limit float + Per-user rate limit for sending notifications from Alertmanager in notifications/sec. 0 = rate limit disabled. Negative value = no notifications are allowed. + -alertmanager.notification-rate-limit-per-integration value + Per-integration notification rate limits. Value is a map, where each key is integration name and value is a rate-limit (float). On command line, this map is given in JSON format. Rate limit has the same meaning as -alertmanager.notification-rate-limit, but only applies for specific integration. Allowed integration names: webhook, email, pagerduty, opsgenie, wechat, slack, victorops, pushover, sns. (default {}) + -alertmanager.receivers-firewall-block-cidr-networks value + Comma-separated list of network CIDRs to block in Alertmanager receiver integrations. + -alertmanager.receivers-firewall-block-private-addresses + True to block private and local addresses in Alertmanager receiver integrations. It blocks private addresses defined by RFC 1918 (IPv4 addresses) and RFC 4193 (IPv6 addresses), as well as loopback, local unicast and local multicast addresses. + -alertmanager.sharding-ring.consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -alertmanager.sharding-ring.etcd.endpoints value + The etcd endpoints to connect to. + -alertmanager.sharding-ring.etcd.password string + Etcd password. + -alertmanager.sharding-ring.etcd.username string + Etcd username. + -alertmanager.sharding-ring.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "consul") + -alertmanager.storage.path string + Directory to store Alertmanager state and temporarily configuration files. The content of this directory is not required to be persisted between restarts unless Alertmanager replication has been disabled. (default "./data-alertmanager/") + -alertmanager.web.external-url value + The URL under which Alertmanager is externally reachable (eg. could be different than -http.alertmanager-http-prefix in case Alertmanager is served via a reverse proxy). This setting is used both to configure the internal requests router and to generate links in alert templates. If the external URL has a path portion, it will be used to prefix all HTTP endpoints served by Alertmanager, both the UI and API. (default http://localhost:8080/alertmanager) + -auth.multitenancy-enabled + When set to true, incoming HTTP requests must specify tenant ID in HTTP X-Scope-OrgId header. When set to false, tenant ID from -auth.no-auth-tenant is used instead. (default true) + -blocks-storage.azure.account-key value + Azure storage account key + -blocks-storage.azure.account-name string + Azure storage account name + -blocks-storage.azure.container-name string + Azure storage container name + -blocks-storage.azure.endpoint-suffix string + Azure storage endpoint suffix without schema. The account name will be prefixed to this value to create the FQDN. If set to empty string, default endpoint suffix is used. + -blocks-storage.backend string + Backend storage to use. Supported backends are: s3, gcs, azure, swift, filesystem. (default "filesystem") + -blocks-storage.bucket-store.bucket-index.enabled + If enabled, queriers and store-gateways discover blocks by reading a bucket index (created and updated by the compactor) instead of periodically scanning the bucket. (default true) + -blocks-storage.bucket-store.chunks-cache.backend string + Backend for chunks cache, if not empty. Supported values: memcached. + -blocks-storage.bucket-store.chunks-cache.memcached.addresses string + Comma separated list of memcached addresses. Supported prefixes are: dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup made after that). + -blocks-storage.bucket-store.chunks-cache.memcached.timeout duration + The socket read/write timeout. (default 200ms) + -blocks-storage.bucket-store.index-cache.backend string + The index cache backend type. Supported values: inmemory, memcached. (default "inmemory") + -blocks-storage.bucket-store.index-cache.inmemory.max-size-bytes uint + Maximum size in bytes of in-memory index cache used to speed up blocks index lookups (shared between all tenants). (default 1073741824) + -blocks-storage.bucket-store.index-cache.memcached.addresses string + Comma separated list of memcached addresses. Supported prefixes are: dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup made after that). + -blocks-storage.bucket-store.index-cache.memcached.timeout duration + The socket read/write timeout. (default 200ms) + -blocks-storage.bucket-store.metadata-cache.backend string + Backend for metadata cache, if not empty. Supported values: memcached. + -blocks-storage.bucket-store.metadata-cache.memcached.addresses string + Comma separated list of memcached addresses. Supported prefixes are: dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup made after that). + -blocks-storage.bucket-store.metadata-cache.memcached.timeout duration + The socket read/write timeout. (default 200ms) + -blocks-storage.bucket-store.sync-dir string + Directory to store synchronized TSDB index headers. This directory is not required to be persisted between restarts, but it's highly recommended in order to improve the store-gateway startup time. (default "./tsdb-sync/") + -blocks-storage.filesystem.dir string + Local filesystem storage directory. (default "blocks") + -blocks-storage.gcs.bucket-name string + GCS bucket name + -blocks-storage.gcs.service-account value + JSON representing either a Google Developers Console client_credentials.json file or a Google Developers service account key file. If empty, fallback to Google default logic. + -blocks-storage.s3.access-key-id string + S3 access key ID + -blocks-storage.s3.bucket-name string + S3 bucket name + -blocks-storage.s3.endpoint string + The S3 bucket endpoint. It could be an AWS S3 endpoint listed at https://docs.aws.amazon.com/general/latest/gr/s3.html or the address of an S3-compatible service in hostname:port format. + -blocks-storage.s3.region string + S3 region. If unset, the client will issue a S3 GetBucketLocation API call to autodetect it. + -blocks-storage.s3.secret-access-key value + S3 secret access key + -blocks-storage.s3.sse.kms-encryption-context string + KMS Encryption Context used for object encryption. It expects JSON formatted string. + -blocks-storage.s3.sse.kms-key-id string + KMS Key ID used to encrypt objects in S3 + -blocks-storage.s3.sse.type string + Enable AWS Server Side Encryption. Supported values: SSE-KMS, SSE-S3. + -blocks-storage.swift.auth-url string + OpenStack Swift authentication URL + -blocks-storage.swift.auth-version int + OpenStack Swift authentication API version. 0 to autodetect. + -blocks-storage.swift.container-name string + Name of the OpenStack Swift container to put chunks in. + -blocks-storage.swift.domain-id string + OpenStack Swift user's domain ID. + -blocks-storage.swift.domain-name string + OpenStack Swift user's domain name. + -blocks-storage.swift.password string + OpenStack Swift API key. + -blocks-storage.swift.project-domain-id string + ID of the OpenStack Swift project's domain (v3 auth only), only needed if it differs the from user domain. + -blocks-storage.swift.project-domain-name string + Name of the OpenStack Swift project's domain (v3 auth only), only needed if it differs from the user domain. + -blocks-storage.swift.project-id string + OpenStack Swift project ID (v2,v3 auth only). + -blocks-storage.swift.project-name string + OpenStack Swift project name (v2,v3 auth only). + -blocks-storage.swift.region-name string + OpenStack Swift Region to use (v2,v3 auth only). + -blocks-storage.swift.user-domain-id string + OpenStack Swift user's domain ID. + -blocks-storage.swift.user-domain-name string + OpenStack Swift user's domain name. + -blocks-storage.swift.user-id string + OpenStack Swift user ID. + -blocks-storage.swift.username string + OpenStack Swift username. + -blocks-storage.tsdb.dir string + Directory to store TSDBs (including WAL) in the ingesters. This directory is required to be persisted between restarts. (default "./tsdb/") + -blocks-storage.tsdb.retention-period duration + TSDB blocks retention in the ingester before a block is removed. This should be larger than the -blocks-storage.tsdb.block-ranges-period, -querier.query-store-after and large enough to give store-gateways and queriers enough time to discover newly uploaded blocks. (default 24h0m0s) + -compactor.blocks-retention-period value + Delete blocks containing samples older than the specified retention period. 0 to disable. + -compactor.compactor-tenant-shard-size int + Max number of compactors that can compact blocks for single tenant. 0 to disable the limit and use all compactors. + -compactor.data-dir string + Directory to temporarily store blocks during compaction. This directory is not required to be persisted between restarts. (default "./data-compactor/") + -compactor.ring.consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -compactor.ring.etcd.endpoints value + The etcd endpoints to connect to. + -compactor.ring.etcd.password string + Etcd password. + -compactor.ring.etcd.username string + Etcd username. + -compactor.ring.heartbeat-period duration + Period at which to heartbeat to the ring. 0 = disabled. (default 5s) + -compactor.ring.heartbeat-timeout duration + The heartbeat timeout after which compactors are considered unhealthy within the ring. 0 = never (timeout disabled). (default 1m0s) + -compactor.ring.instance-addr string + IP address to advertise in the ring. + -compactor.ring.instance-id string + Instance ID to register in the ring. (default "{{.Hostname}}") + -compactor.ring.instance-interface-names value + Name of network interface to read address from. (default [eth0 en0]) + -compactor.ring.instance-port int + Port to advertise in the ring (defaults to server.grpc-listen-port). + -compactor.ring.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "memberlist") + -compactor.ring.wait-active-instance-timeout duration + Timeout for waiting on compactor to become ACTIVE in the ring. (default 10m0s) + -compactor.ring.wait-stability-max-duration duration + Maximum time to wait for ring stability at startup. If the compactor ring keeps changing after this period of time, the compactor will start anyway. (default 5m0s) + -compactor.ring.wait-stability-min-duration duration + Minimum time to wait for ring stability at startup. 0 to disable. + -compactor.split-and-merge-shards int + The number of shards to use when splitting blocks. 0 to disable splitting. + -compactor.split-groups int + Number of groups that blocks for splitting should be grouped into. Each group of blocks is then split separately. Number of output split shards is controlled by -compactor.split-and-merge-shards. (default 1) + -config.expand-env + Expands ${var} or $var in config according to the values of the environment variables. + -config.file value + Configuration file to load. + -consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -debug.block-profile-rate int + Fraction of goroutine blocking events that are reported in the blocking profile. 1 to include every blocking event in the profile, 0 to disable. + -debug.mutex-profile-fraction int + Fraction of mutex contention events that are reported in the mutex profile. On average 1/rate events are reported. 0 to disable. + -distributor.drop-label value + This flag can be used to specify label names that to drop during sample ingestion within the distributor and can be repeated in order to drop multiple labels. + -distributor.excluded-zones value + Comma-separated list of zones to exclude from the ring. Instances in excluded zones will be filtered out from the ring. + -distributor.ha-tracker.cluster string + Prometheus label to look for in samples to identify a Prometheus HA cluster. (default "cluster") + -distributor.ha-tracker.consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -distributor.ha-tracker.enable + Enable the distributors HA tracker so that it can accept samples from Prometheus HA replicas gracefully (requires labels). + -distributor.ha-tracker.enable-for-all-users + Flag to enable, for all users, handling of samples with external labels identifying replicas in an HA Prometheus setup. + -distributor.ha-tracker.etcd.endpoints value + The etcd endpoints to connect to. + -distributor.ha-tracker.etcd.password string + Etcd password. + -distributor.ha-tracker.etcd.username string + Etcd username. + -distributor.ha-tracker.max-clusters int + Maximum number of clusters that HA tracker will keep track of for single user. 0 to disable the limit. + -distributor.ha-tracker.replica string + Prometheus label to look for in samples to identify a Prometheus HA replica. (default "__replica__") + -distributor.ha-tracker.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "consul") + -distributor.ingestion-burst-size int + Per-user allowed ingestion burst size (in number of samples). (default 200000) + -distributor.ingestion-rate-limit float + Per-user ingestion rate limit in samples per second. (default 10000) + -distributor.ingestion-tenant-shard-size int + The tenant's shard size used by shuffle-sharding. Must be set both on ingesters and distributors. 0 disables shuffle sharding. + -distributor.instance-limits.max-inflight-push-requests int + Max inflight push requests that this distributor can handle. This limit is per-distributor, not per-tenant. Additional requests will be rejected. 0 = unlimited. (default 2000) + -distributor.instance-limits.max-ingestion-rate float + Max ingestion rate (samples/sec) that this distributor will accept. This limit is per-distributor, not per-tenant. Additional push requests will be rejected. Current ingestion rate is computed as exponentially weighted moving average, updated every second. 0 = unlimited. + -distributor.replication-factor int + The number of ingesters to write to and read from. (default 3) + -distributor.ring.consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -distributor.ring.etcd.endpoints value + The etcd endpoints to connect to. + -distributor.ring.etcd.password string + Etcd password. + -distributor.ring.etcd.username string + Etcd username. + -distributor.ring.heartbeat-period duration + Period at which to heartbeat to the ring. 0 = disabled. (default 5s) + -distributor.ring.heartbeat-timeout duration + The heartbeat timeout after which distributors are considered unhealthy within the ring. 0 = never (timeout disabled). (default 1m0s) + -distributor.ring.instance-addr string + IP address to advertise in the ring. + -distributor.ring.instance-id string + Instance ID to register in the ring. (default "{{.Hostname}}") + -distributor.ring.instance-interface-names value + Name of network interface to read address from. (default [eth0 en0]) + -distributor.ring.instance-port int + Port to advertise in the ring (defaults to server.grpc-listen-port). + -distributor.ring.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "memberlist") + -distributor.zone-awareness-enabled + True to enable the zone-awareness and replicate ingested samples across different availability zones. + -etcd.endpoints value + The etcd endpoints to connect to. + -etcd.password string + Etcd password. + -etcd.username string + Etcd username. + -frontend.align-querier-with-step + Mutate incoming queries to align their start and end with their step. + -frontend.cache-results + Cache query results. + -frontend.cache-unaligned-requests + Cache requests that are not step-aligned. + -frontend.log-queries-longer-than duration + Log queries that are slower than the specified duration. Set to 0 to disable. Set to < 0 to enable on all queries. + -frontend.max-cache-freshness value + Most recent allowed cacheable result per-tenant, to prevent caching very recent results that might still be in flux. (default 1m) + -frontend.max-queriers-per-tenant int + Maximum number of queriers that can handle requests for a single tenant. If set to 0 or value higher than number of available queriers, *all* queriers will handle requests for the tenant. Each frontend (or query-scheduler, if used) will select the same set of queriers for the same tenant (given that all queriers are connected to all frontends / query-schedulers). This option only works with queriers connecting to the query-frontend / query-scheduler, not when using downstream URL. + -frontend.max-retries-per-request int + Maximum number of retries for a single request; beyond this, the downstream error is returned. (default 5) + -frontend.parallelize-shardable-queries + True to enable query sharding. + -frontend.query-sharding-max-sharded-queries int + The max number of sharded queries that can be run for a given received query. 0 to disable limit. (default 128) + -frontend.query-sharding-total-shards int + The amount of shards to use when doing parallelisation via query sharding by tenant. 0 to disable query sharding for tenant. Query sharding implementation will adjust the number of query shards based on compactor shards. This allows querier to not search the blocks which cannot possibly have the series for given query shard. (default 16) + -frontend.results-cache.backend string + Backend for query-frontend results cache, if not empty. Supported values: [memcached]. + -frontend.results-cache.compression string + Enable cache compression, if not empty. Supported values are: snappy. + -frontend.results-cache.memcached.addresses string + Comma separated list of memcached addresses. Supported prefixes are: dns+ (looked up as an A/AAAA query), dnssrv+ (looked up as a SRV query, dnssrvnoa+ (looked up as a SRV query, with no A/AAAA lookup made after that). + -frontend.results-cache.memcached.timeout duration + The socket read/write timeout. (default 200ms) + -frontend.scheduler-address string + DNS hostname used for finding query-schedulers. + -h + Print basic help. + -help + Print basic help. + -help-all + Print help, also including advanced and experimental parameters. + -http.prefix string + HTTP path prefix for API. (default "/api/prom") + -ingester.active-series-custom-trackers value + Additional active series metrics, matching the provided matchers. Matchers should be in form :, like 'foobar:{foo="bar"}'. Multiple matchers can be provided either providing the flag multiple times or providing multiple semicolon-separated values to a single flag. + -ingester.active-series-metrics-enabled + Enable tracking of active series and export them as metrics. (default true) + -ingester.availability-zone string + The availability zone where this instance is running. + -ingester.lifecycler.interface value + Name of network interface to read address from. (default [eth0 en0]) + -ingester.max-global-metadata-per-metric int + The maximum number of metadata per metric, across the cluster. 0 to disable. + -ingester.max-global-metadata-per-user int + The maximum number of active metrics with metadata per user, across the cluster. 0 to disable. + -ingester.max-global-series-per-metric int + The maximum number of active series per metric name, across the cluster before replication. 0 to disable. (default 20000) + -ingester.max-global-series-per-user int + The maximum number of active series per user, across the cluster before replication. 0 to disable. (default 150000) + -ingester.stream-chunks-when-using-blocks + Stream chunks from ingesters to queriers. (default true) + -ingester.tokens-file-path string + File path where tokens are stored. If empty, tokens are not stored at shutdown and restored at startup. + -log.format value + Output log messages in the given format. Valid formats: [logfmt, json] (default logfmt) + -log.level value + Only log messages with the given severity or above. Valid levels: [debug, info, warn, error] (default info) + -mem-ballast-size-bytes int + Size of memory ballast to allocate. + -memberlist.abort-if-join-fails + If this node fails to join memberlist cluster, abort. (default true) + -memberlist.advertise-addr string + Gossip address to advertise to other members in the cluster. Used for NAT traversal. + -memberlist.advertise-port int + Gossip port to advertise to other members in the cluster. Used for NAT traversal. (default 7946) + -memberlist.bind-addr value + IP address to listen on for gossip messages. Multiple addresses may be specified. Defaults to 0.0.0.0 + -memberlist.bind-port int + Port to listen on for gossip messages. (default 7946) + -memberlist.join value + Other cluster members to join. Can be specified multiple times. It can be an IP, hostname or an entry specified in the DNS Service Discovery format. + -memberlist.transport-debug + Log debug transport messages. Note: global log.level must be at debug level as well. + -modules + List available values that can be used as target. + -print.config + Print the config and exit. + -querier.batch-iterators + Use batch iterators to execute query, as opposed to fully materialising the series in memory. Takes precedent over the -querier.iterators flag. (default true) + -querier.cardinality-analysis-enabled + Enables endpoints used for cardinality analysis. + -querier.default-evaluation-interval duration + The default evaluation interval or step size for subqueries. This config option should be set on query-frontend too when query sharding is enabled. (default 1m0s) + -querier.frontend-address string + Address of query frontend service, in host:port format. If -querier.scheduler-address is set as well, querier will use scheduler instead. Only one of -querier.frontend-address or -querier.scheduler-address can be set. If neither is set, queries are only received via HTTP endpoint. + -querier.iterators + Use iterators to execute query, as opposed to fully materialising the series in memory. + -querier.label-names-and-values-results-max-size-bytes int + Maximum size in bytes of distinct label names and values. When querier receives response from ingester, it merges the response with responses from other ingesters. This maximum size limit is applied to the merged(distinct) results. If the limit is reached, an error is returned. (default 419430400) + -querier.label-values-max-cardinality-label-names-per-request int + Maximum number of label names allowed to be queried in a single /api/v1/cardinality/label_values API call. (default 100) + -querier.lookback-delta duration + Time since the last sample after which a time series is considered stale and ignored by expression evaluations. This config option should be set on query-frontend too when query sharding is enabled. (default 5m0s) + -querier.max-concurrent int + The maximum number of concurrent queries. This config option should be set on query-frontend too when query sharding is enabled. (default 20) + -querier.max-fetched-chunk-bytes-per-query int + The maximum size of all chunks in bytes that a query can fetch from each ingester and storage. This limit is enforced in the querier and ruler. 0 to disable. + -querier.max-fetched-chunks-per-query int + Maximum number of chunks that can be fetched in a single query from ingesters and long-term storage. This limit is enforced in the querier, ruler and store-gateway. 0 to disable. (default 2000000) + -querier.max-fetched-series-per-query int + The maximum number of unique series for which a query can fetch samples from each ingesters and storage. This limit is enforced in the querier and ruler. 0 to disable + -querier.max-query-into-future duration + Maximum duration into the future you can query. 0 to disable. (default 10m0s) + -querier.max-query-lookback value + Limit how long back data (series and metadata) can be queried, up until duration ago. This limit is enforced in the query-frontend, querier and ruler. If the requested time range is outside the allowed range, the request will not fail but will be manipulated to only query data within the allowed time range. 0 to disable. + -querier.max-query-parallelism int + Maximum number of split queries will be scheduled in parallel by the frontend. (default 14) + -querier.max-samples int + Maximum number of samples a single query can load into memory. This config option should be set on query-frontend too when query sharding is enabled. (default 50000000) + -querier.query-ingesters-within duration + Maximum lookback beyond which queries are not sent to ingester. 0 means all queries are sent to ingester. (default 13h0m0s) + -querier.query-label-names-with-matchers-enabled + True to enable queriers to use an optimized implementation which passes down to ingesters the label matchers when running the label names API. Can be enabled once all ingesters run a version >= the one where this option has been introduced. + -querier.query-store-after duration + The time after which a metric should be queried from storage and not just ingesters. 0 means all queries are sent to store. If this option is enabled, the time range of the query sent to the store-gateway will be manipulated to ensure the query end is not more recent than 'now - query-store-after'. + -querier.scheduler-address string + Hostname (and port) of scheduler that querier will periodically resolve, connect to and receive queries from. Only one of -querier.frontend-address or -querier.scheduler-address can be set. If neither is set, queries are only received via HTTP endpoint. + -querier.shuffle-sharding-ingesters-lookback-period duration + When distributor's sharding strategy is shuffle-sharding and this setting is > 0, queriers fetch in-memory series from the minimum set of required ingesters, selecting only ingesters which may have received series since 'now - lookback period'. The lookback period should be greater or equal than the configured -querier.query-store-after and -querier.query-ingesters-within. If this setting is 0, queriers always query all ingesters (ingesters shuffle sharding on read path is disabled). + -querier.timeout duration + The timeout for a query. This config option should be set on query-frontend too when query sharding is enabled. (default 2m0s) + -query-scheduler.max-outstanding-requests-per-tenant int + Maximum number of outstanding requests per tenant per query-scheduler. In-flight requests above this limit will fail with HTTP response status code 429. (default 100) + -query-scheduler.querier-forget-delay duration + If a querier disconnects without sending notification about graceful shutdown, the query-scheduler will keep the querier in the tenant's shard until the forget delay has passed. This feature is useful to reduce the blast radius when shuffle-sharding is enabled. + -ring.heartbeat-timeout duration + The heartbeat timeout after which ingesters are skipped for reads/writes. 0 = never (timeout disabled). (default 1m0s) + -ring.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "memberlist") + -ruler-storage.azure.account-key value + Azure storage account key + -ruler-storage.azure.account-name string + Azure storage account name + -ruler-storage.azure.container-name string + Azure storage container name + -ruler-storage.azure.endpoint-suffix string + Azure storage endpoint suffix without schema. The account name will be prefixed to this value to create the FQDN. If set to empty string, default endpoint suffix is used. + -ruler-storage.backend string + Backend storage to use. Supported backends are: s3, gcs, azure, swift, filesystem, local. (default "filesystem") + -ruler-storage.filesystem.dir string + Local filesystem storage directory. (default "ruler") + -ruler-storage.gcs.bucket-name string + GCS bucket name + -ruler-storage.gcs.service-account value + JSON representing either a Google Developers Console client_credentials.json file or a Google Developers service account key file. If empty, fallback to Google default logic. + -ruler-storage.local.directory string + Directory to scan for rules + -ruler-storage.s3.access-key-id string + S3 access key ID + -ruler-storage.s3.bucket-name string + S3 bucket name + -ruler-storage.s3.endpoint string + The S3 bucket endpoint. It could be an AWS S3 endpoint listed at https://docs.aws.amazon.com/general/latest/gr/s3.html or the address of an S3-compatible service in hostname:port format. + -ruler-storage.s3.region string + S3 region. If unset, the client will issue a S3 GetBucketLocation API call to autodetect it. + -ruler-storage.s3.secret-access-key value + S3 secret access key + -ruler-storage.s3.sse.kms-encryption-context string + KMS Encryption Context used for object encryption. It expects JSON formatted string. + -ruler-storage.s3.sse.kms-key-id string + KMS Key ID used to encrypt objects in S3 + -ruler-storage.s3.sse.type string + Enable AWS Server Side Encryption. Supported values: SSE-KMS, SSE-S3. + -ruler-storage.swift.auth-url string + OpenStack Swift authentication URL + -ruler-storage.swift.auth-version int + OpenStack Swift authentication API version. 0 to autodetect. + -ruler-storage.swift.container-name string + Name of the OpenStack Swift container to put chunks in. + -ruler-storage.swift.domain-id string + OpenStack Swift user's domain ID. + -ruler-storage.swift.domain-name string + OpenStack Swift user's domain name. + -ruler-storage.swift.password string + OpenStack Swift API key. + -ruler-storage.swift.project-domain-id string + ID of the OpenStack Swift project's domain (v3 auth only), only needed if it differs the from user domain. + -ruler-storage.swift.project-domain-name string + Name of the OpenStack Swift project's domain (v3 auth only), only needed if it differs from the user domain. + -ruler-storage.swift.project-id string + OpenStack Swift project ID (v2,v3 auth only). + -ruler-storage.swift.project-name string + OpenStack Swift project name (v2,v3 auth only). + -ruler-storage.swift.region-name string + OpenStack Swift Region to use (v2,v3 auth only). + -ruler-storage.swift.user-domain-id string + OpenStack Swift user's domain ID. + -ruler-storage.swift.user-domain-name string + OpenStack Swift user's domain name. + -ruler-storage.swift.user-id string + OpenStack Swift user ID. + -ruler-storage.swift.username string + OpenStack Swift username. + -ruler.alertmanager-client.basic-auth-password string + HTTP Basic authentication password. It overrides the password set in the URL (if any). + -ruler.alertmanager-client.basic-auth-username string + HTTP Basic authentication username. It overrides the username set in the URL (if any). + -ruler.alertmanager-url string + Comma-separated list of URL(s) of the Alertmanager(s) to send notifications to. Each URL is treated as a separate group. Multiple Alertmanagers in HA per group can be supported by using DNS service discovery format. Basic auth is supported as part of the URL. + -ruler.enable-api + Enable the ruler config API. (default true) + -ruler.evaluation-delay-duration value + Duration to delay the evaluation of rules to ensure the underlying metrics have been pushed. + -ruler.external.url value + URL of alerts return path. + -ruler.max-rule-groups-per-tenant int + Maximum number of rule groups per-tenant. 0 to disable. (default 70) + -ruler.max-rules-per-rule-group int + Maximum number of rules per rule group per-tenant. 0 to disable. (default 20) + -ruler.ring.consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -ruler.ring.etcd.endpoints value + The etcd endpoints to connect to. + -ruler.ring.etcd.password string + Etcd password. + -ruler.ring.etcd.username string + Etcd username. + -ruler.ring.heartbeat-period duration + Period at which to heartbeat to the ring. 0 = disabled. (default 5s) + -ruler.ring.heartbeat-timeout duration + The heartbeat timeout after which rulers are considered unhealthy within the ring. 0 = never (timeout disabled). (default 1m0s) + -ruler.ring.instance-addr string + IP address to advertise in the ring. + -ruler.ring.instance-id string + Instance ID to register in the ring. (default "{{.Hostname}}") + -ruler.ring.instance-interface-names value + Name of network interface to read address from. (default [eth0 en0]) + -ruler.ring.instance-port int + Port to advertise in the ring (defaults to server.grpc-listen-port). + -ruler.ring.num-tokens int + Number of tokens for each ruler. (default 128) + -ruler.ring.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "memberlist") + -ruler.rule-path string + Directory to store temporary rule files loaded by the Prometheus rule managers. This directory is not required to be persisted between restarts. (default "./data-ruler/") + -ruler.tenant-federation.enabled + Enable running rule groups against multiple tenants. The tenant IDs involved need to be in the rule group's 'source_tenants' field. If this flag is set to 'false' when there are already created federated rule groups, then these rules groups will be skipped during evaluations. + -ruler.tenant-shard-size int + The tenant's shard size when sharding is used by ruler. Value of 0 disables shuffle sharding for the tenant, and tenant rules will be sharded across all ruler replicas. + -runtime-config.file string + File with the configuration that can be updated in runtime. + -server.grpc-listen-address string + gRPC server listen address. + -server.grpc-listen-port int + gRPC server listen port. (default 9095) + -server.http-listen-address string + HTTP server listen address. + -server.http-listen-port int + HTTP server listen port. (default 8080) + -store-gateway.sharding-ring.consul.hostname string + Hostname and port of Consul. (default "localhost:8500") + -store-gateway.sharding-ring.etcd.endpoints value + The etcd endpoints to connect to. + -store-gateway.sharding-ring.etcd.password string + Etcd password. + -store-gateway.sharding-ring.etcd.username string + Etcd username. + -store-gateway.sharding-ring.heartbeat-period duration + Period at which to heartbeat to the ring. 0 = disabled. (default 15s) + -store-gateway.sharding-ring.heartbeat-timeout duration + The heartbeat timeout after which store gateways are considered unhealthy within the ring. 0 = never (timeout disabled). This option needs be set both on the store-gateway and querier when running in microservices mode. (default 1m0s) + -store-gateway.sharding-ring.instance-addr string + IP address to advertise in the ring. + -store-gateway.sharding-ring.instance-availability-zone string + The availability zone where this instance is running. Required if zone-awareness is enabled. + -store-gateway.sharding-ring.instance-id string + Instance ID to register in the ring. (default "{{.Hostname}}") + -store-gateway.sharding-ring.instance-interface-names value + Name of network interface to read address from. (default [eth0 en0]) + -store-gateway.sharding-ring.instance-port int + Port to advertise in the ring (defaults to server.grpc-listen-port). + -store-gateway.sharding-ring.replication-factor int + The replication factor to use when sharding blocks. This option needs be set both on the store-gateway and querier when running in microservices mode. (default 3) + -store-gateway.sharding-ring.store string + Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "memberlist") + -store-gateway.sharding-ring.tokens-file-path string + File path where tokens are stored. If empty, tokens are not stored at shutdown and restored at startup. + -store-gateway.sharding-ring.unregister-on-shutdown + Unregister from the ring upon clean shutdown. (default true) + -store-gateway.sharding-ring.wait-stability-max-duration duration + Maximum time to wait for ring stability at startup. If the store-gateway ring keeps changing after this period of time, the store-gateway will start anyway. (default 5m0s) + -store-gateway.sharding-ring.wait-stability-min-duration duration + Minimum time to wait for ring stability at startup, if set to positive value. + -store-gateway.sharding-ring.zone-awareness-enabled + True to enable zone-awareness and replicate blocks across different availability zones. + -store-gateway.tenant-shard-size int + The tenant's shard size, used when store-gateway sharding is enabled. Value of 0 disables shuffle sharding for the tenant, that is all tenant blocks are sharded across all store-gateway replicas. + -store.max-labels-query-length value + Limit the time range (end - start time) of series, label names and values queries. This limit is enforced in the querier. If the requested time range is outside the allowed range, the request will not fail but will be manipulated to only query data within the allowed time range. 0 to disable. + -store.max-query-length value + Limit the query time range (end - start time). This limit is enforced in the query-frontend (on the received query), in the querier (on the query possibly split by the query-frontend) and ruler. 0 to disable. + -target value + Comma-separated list of modules to load. The alias 'all' can be used in the list to load a number of core modules and will enable single-binary mode. Use '-modules' command line flag to get a list of available modules, and to see which modules are included in 'all'. (default all) + -tenant-federation.enabled + If enabled on all services, queries can be federated across multiple tenants. The tenant IDs involved need to be specified separated by a '|' character in the 'X-Scope-OrgID' header. + -validation.create-grace-period value + Duration which table will be created/deleted before/after it's needed; we won't accept sample from before this time. (default 10m) + -validation.enforce-metadata-metric-name + Enforce every metadata has a metric name. (default true) + -validation.max-label-names-per-series int + Maximum number of label names per series. (default 30) + -validation.max-length-label-name int + Maximum length accepted for label names (default 1024) + -validation.max-length-label-value int + Maximum length accepted for label value. This setting also applies to the metric name (default 2048) + -validation.max-metadata-length int + Maximum length accepted for metric metadata. Metadata refers to Metric Name, HELP and UNIT. (default 1024) + -version + Print application version and exit. + +To see all flags, use -help-all diff --git a/cmd/mimir/main.go b/cmd/mimir/main.go index 4465d3c194e..191a706b8a7 100644 --- a/cmd/mimir/main.go +++ b/cmd/mimir/main.go @@ -21,19 +21,12 @@ import ( "github.com/grafana/dskit/flagext" "github.com/pkg/errors" "github.com/prometheus/client_golang/prometheus" - "github.com/prometheus/common/version" "github.com/weaveworks/common/tracing" "gopkg.in/yaml.v2" "github.com/grafana/mimir/pkg/mimir" util_log "github.com/grafana/mimir/pkg/util/log" -) - -// Version is set via build flag -ldflags -X main.Version -var ( - Version string - Branch string - Revision string + "github.com/grafana/mimir/pkg/util/version" ) // configHash exposes information about the loaded config @@ -46,9 +39,7 @@ var configHash *prometheus.GaugeVec = prometheus.NewGaugeVec( ) func init() { - version.Version = Version - version.Branch = Branch - version.Revision = Revision + prometheus.MustRegister(version.NewCollector("mimir")) prometheus.MustRegister(version.NewCollector("cortex")) prometheus.MustRegister(configHash) } @@ -68,6 +59,8 @@ func main() { blockProfileRate int printVersion bool printModules bool + printHelp bool + printHelpAll bool ) configFile, expandENV := parseConfigFileParameter(os.Args[1:]) @@ -95,24 +88,32 @@ func main() { flag.IntVar(&blockProfileRate, "debug.block-profile-rate", 0, "Fraction of goroutine blocking events that are reported in the blocking profile. 1 to include every blocking event in the profile, 0 to disable.") flag.BoolVar(&printVersion, "version", false, "Print application version and exit.") flag.BoolVar(&printModules, "modules", false, "List available values that can be used as target.") + flag.BoolVar(&printHelp, "help", false, "Print basic help.") + flag.BoolVar(&printHelp, "h", false, "Print basic help.") + flag.BoolVar(&printHelpAll, "help-all", false, "Print help, also including advanced and experimental parameters.") - usage := flag.CommandLine.Usage flag.CommandLine.Usage = func() { /* don't do anything by default, we will print usage ourselves, but only when requested. */ } flag.CommandLine.Init(flag.CommandLine.Name(), flag.ContinueOnError) - err := flag.CommandLine.Parse(os.Args[1:]) - if err == flag.ErrHelp { - // Print available parameters to stdout, so that users can grep/less it easily. - flag.CommandLine.SetOutput(os.Stdout) - usage() + if err := flag.CommandLine.Parse(os.Args[1:]); err != nil { + fmt.Fprintln(flag.CommandLine.Output(), "Run with -help to get a list of available parameters") if !testMode { os.Exit(2) } - } else if err != nil { - fmt.Fprintln(flag.CommandLine.Output(), "Run with -help to get list of available parameters") + } + + if printHelp || printHelpAll { + // Print available parameters to stdout, so that users can grep/less them easily. + flag.CommandLine.SetOutput(os.Stdout) + if err := usage(&cfg, printHelpAll); err != nil { + fmt.Fprintf(os.Stderr, "error printing usage: %s\n", err) + os.Exit(1) + } + if !testMode { os.Exit(2) } + return } if printVersion { @@ -122,8 +123,7 @@ func main() { // Validate the config once both the config file has been loaded // and CLI flags parsed. - err = cfg.Validate(util_log.Logger) - if err != nil { + if err := cfg.Validate(util_log.Logger); err != nil { fmt.Fprintf(os.Stderr, "error validating config: %v\n", err) if !testMode { os.Exit(1) diff --git a/cmd/mimir/main_test.go b/cmd/mimir/main_test.go index a14068fd52f..0f14f95f87d 100644 --- a/cmd/mimir/main_test.go +++ b/cmd/mimir/main_test.go @@ -9,11 +9,12 @@ import ( "bytes" "flag" "io" - "io/ioutil" "os" + "path/filepath" "strings" "sync" "testing" + "text/template" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -28,15 +29,27 @@ func TestFlagParsing(t *testing.T) { stdoutExcluded string // string that must NOT be included in stdout stderrExcluded string // string that must NOT be included in stderr }{ - "help": { + "help-short": { arguments: []string{"-h"}, stdoutMessage: "Usage of", // Usage must be on stdout, not stderr. stderrExcluded: "Usage of", }, + "help": { + arguments: []string{"-help"}, + stdoutMessage: "Usage of", // Usage must be on stdout, not stderr. + stderrExcluded: "Usage of", + }, + + "help-all": { + arguments: []string{"-help-all"}, + stdoutMessage: "Usage of", // Usage must be on stdout, not stderr. + stderrExcluded: "Usage of", + }, + "unknown flag": { arguments: []string{"-unknown.flag"}, - stderrMessage: "Run with -help to get list of available parameters", + stderrMessage: "Run with -help to get a list of available parameters", stdoutExcluded: "Usage of", // No usage description on unknown flag. stderrExcluded: "Usage of", }, @@ -99,6 +112,74 @@ func TestFlagParsing(t *testing.T) { } } +func TestHelp(t *testing.T) { + for _, tc := range []struct { + name string + arg string + filename string + }{ + { + name: "basic", + arg: "-h", + filename: "help.txt.tmpl", + }, + { + name: "all", + arg: "-help-all", + filename: "help-all.txt.tmpl", + }, + } { + t.Run(tc.name, func(t *testing.T) { + oldArgs, oldStdout, oldStderr, oldTestMode, oldCmdLine := os.Args, os.Stdout, os.Stderr, testMode, flag.CommandLine + restored := false + restoreIfNeeded := func() { + if restored { + return + } + + os.Stdout = oldStdout + os.Stderr = oldStderr + os.Args = oldArgs + testMode = oldTestMode + flag.CommandLine = oldCmdLine + restored = true + } + t.Cleanup(restoreIfNeeded) + + testMode = true + co := captureOutput(t) + + const cmd = "./cmd/mimir/mimir" + os.Args = []string{cmd, tc.arg} + + // reset default flags + flag.CommandLine = flag.NewFlagSet(os.Args[0], flag.ExitOnError) + + main() + + stdout, stderr := co.Done() + + // Restore stdout and stderr before reporting errors to make them visible. + restoreIfNeeded() + + hostname, err := os.Hostname() + require.NoError(t, err) + c := struct { + Hostname string + }{ + Hostname: hostname, + } + + tmpl, err := template.ParseFiles(tc.filename) + require.NoError(t, err) + var b strings.Builder + require.NoError(t, tmpl.Execute(&b, c)) + assert.Equalf(t, b.String(), string(stdout), "%s %s output changed; try `make reference-help`", cmd, tc.arg) + assert.Empty(t, stderr) + }) + } +} + func testSingle(t *testing.T, arguments []string, yaml string, stdoutMessage, stderrMessage, stdoutExcluded, stderrExcluded []byte) { t.Helper() oldArgs, oldStdout, oldStderr, oldTestMode := os.Args, os.Stdout, os.Stderr, testMode @@ -116,18 +197,12 @@ func testSingle(t *testing.T, arguments []string, yaml string, stdoutMessage, st defer restoreIfNeeded() if yaml != "" { - tempFile, err := ioutil.TempFile("", "test") - require.NoError(t, err) - - defer func() { - require.NoError(t, tempFile.Close()) - require.NoError(t, os.Remove(tempFile.Name())) - }() - - _, err = tempFile.WriteString(yaml) + tempDir := t.TempDir() + fpath := filepath.Join(tempDir, "test") + err := os.WriteFile(fpath, []byte(yaml), 0600) require.NoError(t, err) - arguments = append(arguments, "-"+configFileOption, tempFile.Name()) + arguments = append(arguments, "-"+configFileOption, fpath) } arguments = append([]string{"./mimir"}, arguments...) diff --git a/cmd/mimir/usage.go b/cmd/mimir/usage.go new file mode 100644 index 00000000000..8ee16c6f8ec --- /dev/null +++ b/cmd/mimir/usage.go @@ -0,0 +1,168 @@ +// SPDX-License-Identifier: AGPL-3.0-only + +package main + +import ( + "flag" + "fmt" + "os" + "reflect" + "strings" + + "github.com/grafana/mimir/pkg/mimir" + "github.com/grafana/mimir/pkg/util/fieldcategory" +) + +// usage prints command-line usage, the printAll argument controls whether also non-basic flags will be included. +func usage(cfg *mimir.Config, printAll bool) error { + fields := map[uintptr]reflect.StructField{} + if err := parseConfig(cfg, fields); err != nil { + return err + } + + fs := flag.CommandLine + fmt.Fprintf(fs.Output(), "Usage of %s:\n", os.Args[0]) + fs.VisitAll(func(fl *flag.Flag) { + v := reflect.ValueOf(fl.Value) + fieldCat := fieldcategory.Basic + + if override, ok := fieldcategory.GetOverride(fl.Name); ok { + fieldCat = override + } else if v.Kind() == reflect.Ptr { + ptr := v.Pointer() + field, ok := fields[ptr] + if ok { + catStr := field.Tag.Get("category") + switch catStr { + case "advanced": + fieldCat = fieldcategory.Advanced + case "experimental": + fieldCat = fieldcategory.Experimental + } + } + } + + if fieldCat != fieldcategory.Basic && !printAll { + // Don't print help for this flag since we're supposed to print only basic flags + return + } + + var b strings.Builder + // Two spaces before -; see next two comments. + fmt.Fprintf(&b, " -%s", fl.Name) + name := getFlagName(fl) + if len(name) > 0 { + b.WriteString(" ") + b.WriteString(name) + } + // Four spaces before the tab triggers good alignment + // for both 4- and 8-space tab stops. + b.WriteString("\n \t") + if fieldCat == fieldcategory.Experimental { + b.WriteString("[experimental] ") + } + b.WriteString(strings.ReplaceAll(fl.Usage, "\n", "\n \t")) + + if !isZeroValue(fl, fl.DefValue) { + v := reflect.ValueOf(fl.Value) + if v.Kind() == reflect.Ptr { + v = v.Elem() + } + if v.Kind() == reflect.String { + // put quotes on the value + fmt.Fprintf(&b, " (default %q)", fl.DefValue) + } else { + fmt.Fprintf(&b, " (default %v)", fl.DefValue) + } + } + fmt.Fprint(fs.Output(), b.String(), "\n") + }) + + if !printAll { + fmt.Fprintf(fs.Output(), "\nTo see all flags, use -help-all\n") + } + + return nil +} + +// isZeroValue determines whether the string represents the zero +// value for a flag. +func isZeroValue(fl *flag.Flag, value string) bool { + // Build a zero value of the flag's Value type, and see if the + // result of calling its String method equals the value passed in. + // This works unless the Value type is itself an interface type. + typ := reflect.TypeOf(fl.Value) + var z reflect.Value + if typ.Kind() == reflect.Ptr { + z = reflect.New(typ.Elem()) + } else { + z = reflect.Zero(typ) + } + return value == z.Interface().(flag.Value).String() +} + +// parseConfig parses a mimir.Config and populates fields. +func parseConfig(cfg interface{}, fields map[uintptr]reflect.StructField) error { + // The input config is expected to be a pointer to struct. + if reflect.TypeOf(cfg).Kind() != reflect.Ptr { + t := reflect.TypeOf(cfg) + return fmt.Errorf("%s is a %s while a %s is expected", t, t.Kind(), reflect.Ptr) + } + v := reflect.ValueOf(cfg).Elem() + if v.Kind() != reflect.Struct { + return fmt.Errorf("%s is a %s while a %s is expected", v, v.Kind(), reflect.Struct) + } + + t := v.Type() + + for i := 0; i < t.NumField(); i++ { + field := t.Field(i) + if field.Type.Kind() == reflect.Func { + continue + } + + fieldValue := v.FieldByIndex(field.Index) + + // Take address of field value and map it to field + fields[fieldValue.Addr().Pointer()] = field + + // Recurse if a struct + if field.Type.Kind() != reflect.Struct { + continue + } + + if err := parseConfig(fieldValue.Addr().Interface(), fields); err != nil { + return err + } + } + + return nil +} + +func getFlagName(fl *flag.Flag) string { + getter, ok := fl.Value.(flag.Getter) + if !ok { + return "value" + } + + name := "value" + + v := reflect.ValueOf(getter.Get()) + t := v.Type() + switch t.Name() { + case "bool": + name = "" + case "Duration": + name = "duration" + case "float64": + name = "float" + case "int", "int64": + name = "int" + case "string": + name = "string" + case "uint", "uint64": + name = "uint" + } + + return name +} diff --git a/cmd/mimirtool/CHANGELOG.md b/cmd/mimirtool/CHANGELOG.md index 791ec008252..587c2a5f1b8 100644 --- a/cmd/mimirtool/CHANGELOG.md +++ b/cmd/mimirtool/CHANGELOG.md @@ -7,6 +7,15 @@ Order should be `CHANGE`, `FEATURE`, `ENHANCEMENT`, and `BUGFIX` * [FEATURE] Support Arm64 on Darwin for all binaries (benchtool etc). https://github.com/grafana/cortex-tools/pull/215 * [ENHANCEMENT] Correctly support federated rules. #823 * [BUGFIX] Fix `cortextool rules` legends displaying wrong symbols for updates and deletions. https://github.com/grafana/cortex-tools/pull/226 +* [CHANGE] The following environment variables have been renamed: #883 + * `CORTEX_ADDRESS` to `MIMIR_ADDRESS` + * `CORTEX_API_USER` to `MIMIR_API_USER` + * `CORTEX_API_KEY` to `MIMIR_API_KEY` + * `CORTEX_TENANT_ID` to `MIMIR_TENANT_ID` + * `CORTEX_TLS_CA_PATH` to `MIMIR_TLS_CA_PATH` + * `CORTEX_TLS_CERT_PATH` to `MIMIR_TLS_CERT_PATH` + * `CORTEX_TLS_KEY_PATH` to `MIMIR_TLS_KEY_PATH` +* [CHANGE] Change `cortex` backend to `mimir`. #883 ## v0.10.7 diff --git a/cmd/mimirtool/README.md b/cmd/mimirtool/README.md deleted file mode 100644 index a34d092d772..00000000000 --- a/cmd/mimirtool/README.md +++ /dev/null @@ -1,476 +0,0 @@ -# Cortex Tools - -This repo contains tools used for interacting with [Cortex](https://github.com/cortexproject/cortex). - -- [benchtool](docs/benchtool.md): A powerful YAML driven tool for benchmarking - Cortex write and query API. -- [cortextool](#cortextool): Interacts with user-facing Cortex APIs and backend storage components -- [chunktool](#chunktool): Interacts with chunks stored and indexed in Cortex storage backends. -- [logtool](#logtool): Tool which parses Cortex query-frontend logs and formats them for easy analysis. -- [e2ealerting](docs/e2ealerting.md): Tool that helps measure how long an alerts takes from scrape of sample to Alertmanager notifcation delivery. - -# Installation - -The various binaries are available for macOS, Windows, and Linux. - -## macOS - -`cortextool` is available on macOS via [Homebrew](https://brew.sh/): - -```bash -$ brew install grafana/grafana/cortextool -``` - -## Linux, Docker and Windows - -Refer to the [latest release](https://github.com/grafana/cortex-tools/releases) for installation intructions on these. - -## cortextool - -This tool is designed to interact with the various user-facing APIs provided by Cortex, as well as, interact with various backend storage components containing Cortex data. - -### Config Commands - -Config commands interact with the Cortex api and read/create/update/delete user configs from Cortex. Specifically a users alertmanager and rule configs can be composed and updated using these commands. - -#### Configuration - -| Env Variables | Flag | Description | -| ---------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| CORTEX_ADDRESS | `address` | Address of the API of the desired Cortex cluster. | -| CORTEX_API_USER | `user` | In cases where the Cortex API is set behind a basic auth gateway, a user can be set as a basic auth user. If empty and CORTEX_API_KEY is set, CORTEX_TENANT_ID will be used instead. | -| CORTEX_API_KEY | `key` | In cases where the Cortex API is set behind a basic auth gateway, a key can be set as a basic auth password. | -| CORTEX_TENANT_ID | `id` | The tenant ID of the Cortex instance to interact with. | - -#### Alertmanager - -The following commands are used by users to interact with their Cortex alertmanager configuration, as well as their alert template files. - -##### Alertmanager Get - - cortextool alertmanager get - -##### Alertmanager Load - - cortextool alertmanager load ./example_alertmanager_config.yaml - - cortextool alertmanager load ./example_alertmanager_config.yaml template_file1.tmpl template_file2.tmpl - -#### Rules - -The following commands are used by users to interact with their Cortex ruler configuration. They can load prometheus rule files, as well as interact with individual rule groups. - -##### Rules List - -This command will retrieve all of the rule groups stored in the specified Cortex instance and print each one by rule group name and namespace to the terminal. - - cortextool rules list - -##### Rules Print - -This command will retrieve all of the rule groups stored in the specified Cortex instance and print them to the terminal. - - cortextool rules print - -##### Rules Get - -This command will retrieve the specified rule group from Cortex and print it to the terminal. - - cortextool rules get example_namespace example_rule_group - -##### Rules Delete - -This command will delete the specified rule group from the specified namespace. - - cortextool rules delete example_namespace example_rule_group - -##### Rules Load - -This command will load each rule group in the specified files and load them into Cortex. If a rule already exists in Cortex it will be overwritten if a diff is found. - - cortextool rules load ./example_rules_one.yaml ./example_rules_two.yaml ... - -Example file: - -```yaml -namespace: my_namespace -groups: - - name: example - interval: 5m - source_tenants: [team-engineering, team-finance] - rules: - - record: job:http_inprogress_requests:sum - expr: sum by (job) (http_inprogress_requests) -``` - -##### Rules Lint - -This command lints a rules file. The linter's aim is not to verify correctness but just YAML and PromQL expression formatting within the rule file. This command always edits in place, you can use the dry run flag (`-n`) if you'd like to perform a trial run that does not make any changes. This command does not interact with your Cortex cluster. - - cortextool rules lint -n ./example_rules_one.yaml ./example_rules_two.yaml ... - -The format of the file is the same as in [Rules Diff](#rules-diff). - -##### Rules Prepare - -This command prepares a rules file for upload to Cortex. It lints all your PromQL expressions and adds an specific label to your PromQL query aggregations in the file. This command does not interact with your Cortex cluster. - - cortextool rules prepare -i ./example_rules_one.yaml ./example_rules_two.yaml ... - -The format of the file is the same as in [Rules Diff](#rules-diff). - -There are two flags of note for this command: - -- `-i` which allows you to edit in place, otherwise a a new file with a `.output` extension is created with the results of the run. -- `-l` which allows you specify the label you want you add for your aggregations, it is `cluster` by default. - -At the end of the run, the command tells you whenever the operation was a success in the form of - - INFO[0000] SUCESS: 194 rules found, 0 modified expressions - -It is important to note that a modification can be a PromQL expression lint or a label add to your aggregation. - -##### Rules Check - -This command checks rules against the recommended [best practices](https://prometheus.io/docs/practices/rules/) for rules. This command does not interact with your Mimir cluster. - - mimirtool rules check ./example_rules_one.yaml - -The format of the file is the same as in [Rules Diff](#rules-diff). - -##### Rules Diff - -This command compares rules against the rules in your Mimir cluster. - - mimirtool rules diff ./example_rules_one.yaml - -The format of the file is the same as in [Rules Diff](#rules-diff). - -##### Rules Sync - -This command compares rules against the rules in your Mimir cluster. It applies any differences to your Mimir cluster. - - mimirtool rules sync ./example_rules_one.yaml - -The format of the file is the same as in [Rules Diff](#rules-diff). - -#### Remote Read - -Cortex exposes a [Remote Read API] which allows access to the stored series. The `remote-read` subcommand of `cortextool` allows to interact with its API, to find out which series are stored. - -[remote read api]: https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations - -##### Remote Read show statistics - -The `remote-read stats` command summarizes statistics of the stored series matching the selector. - -```sh -cortextool remote-read stats --selector '{job="node"}' --address http://demo.robustperception.io:9090 --remote-read-path /api/v1/read -INFO[0000] Create remote read client using endpoint 'http://demo.robustperception.io:9090/api/v1/read' -INFO[0000] Querying time from=2020-12-30T14:00:00Z to=2020-12-30T15:00:00Z with selector={job="node"} -INFO[0000] MIN TIME MAX TIME DURATION NUM SAMPLES NUM SERIES NUM STALE NAN VALUES NUM NAN VALUES -INFO[0000] 2020-12-30 14:00:00.629 +0000 UTC 2020-12-30 14:59:59.629 +0000 UTC 59m59s 159480 425 0 0 -``` - -##### Remote Read dump series - -The `remote-read dump` command prints all series and samples matching the selector. - -```sh -cortextool remote-read dump --selector 'up{job="node"}' --address http://demo.robustperception.io:9090 --remote-read-path /api/v1/read -{__name__="up", instance="demo.robustperception.io:9100", job="node"} 1 1609336914711 -{__name__="up", instance="demo.robustperception.io:9100", job="node"} NaN 1609336924709 # StaleNaN -[...] -``` - -##### Remote Read export series into local TSDB - -The `remote-read export` command exports all series and samples matching the selector into a local TSDB. This TSDB can then be further analysed with local tooling like `prometheus` and `promtool`. - -```sh -# Use Remote Read API to download all metrics with label job=name into local tsdb -cortextool remote-read export --selector '{job="node"}' --address http://demo.robustperception.io:9090 --remote-read-path /api/v1/read --tsdb-path ./local-tsdb -INFO[0000] Create remote read client using endpoint 'http://demo.robustperception.io:9090/api/v1/read' -INFO[0000] Created TSDB in path './local-tsdb' -INFO[0000] Using existing TSDB in path './local-tsdb' -INFO[0000] Querying time from=2020-12-30T13:53:59Z to=2020-12-30T14:53:59Z with selector={job="node"} -INFO[0001] Store TSDB blocks in './local-tsdb' -INFO[0001] BLOCK ULID MIN TIME MAX TIME DURATION NUM SAMPLES NUM CHUNKS NUM SERIES SIZE -INFO[0001] 01ETT28D6B8948J87NZXY8VYD9 2020-12-30 13:53:59 +0000 UTC 2020-12-30 13:59:59 +0000 UTC 6m0.001s 15950 429 425 105KiB867B -INFO[0001] 01ETT28D91Z9SVRYF3DY0KNV41 2020-12-30 14:00:00 +0000 UTC 2020-12-30 14:53:58 +0000 UTC 53m58.001s 143530 1325 425 509KiB679B - -# Examples for using local TSDB -## Analyzing contents using promtool -promtool tsdb analyze ./local-tsdb - -## Dump all values of the TSDB -promtool tsdb dump ./local-tsdb - -## Run a local prometheus -prometheus --storage.tsdb.path ./local-tsdb --config.file=<(echo "") -``` - -#### Overrides Exporter - -The Overrides Exporter allows to continuously export [per tenant configuration overrides][runtime-config] as metrics. Optionally it can also export a presets file (cf. example [override config file] and [presets file]). - - cortextool overrides-exporter --overrides-file overrides.yaml --presets-file presets.yaml - -[override config file]: ./pkg/commands/testdata/overrides.yaml -[presets file]: ./pkg/commands/testdata/presets.yaml -[runtime-config]: https://cortexmetrics.io/docs/configuration/arguments/#runtime-configuration-file - -#### Generate ACL Headers - -This lets you generate the header which can then be used to enforce access control rules in GME / GrafanaCloud. - -``` -./cortextool acl generate-header --id=1234 --rule='{namespace="A"}' -``` - -#### Analyse - -Run analysis against your Prometheus, Grafana and Cortex to see which metrics being used and exported. Can also extract metrics -from dashboard JSON and rules YAML files. - -##### `analyse grafana` - -This command will run against your Grafana instance and will download its dashboards and then extract the Prometheus metrics used in its queries. The output is a JSON file. - -###### Configuration - -| Env Variables | Flag | Description | -| --------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------- | -| GRAFANA_ADDRESS | `address` | Address of the Grafana instance. | -| GRAFANA_API_KEY | `key` | The API Key for the Grafana instance. Create a key using the following instructions: https://grafana.com/docs/grafana/latest/http_api/auth/ | -| \_\_ | `output` | The output file path. metrics-in-grafana.json by default. | - -###### Running the command - -```shell -cortextool analyse grafana --address= --key= -``` - -###### Sample output - -```json -{ - "metricsUsed": [ - "apiserver_request:availability30d", - "workqueue_depth", - "workqueue_queue_duration_seconds_bucket", - ... - ], - "dashboards": [ - { - "slug": "", - "uid": "09ec8aa1e996d6ffcd6817bbaff4db1b", - "title": "Kubernetes / API server", - "metrics": [ - "apiserver_request:availability30d", - "apiserver_request_total", - "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile", - "workqueue_depth", - "workqueue_queue_duration_seconds_bucket", - ... - ], - "parse_errors": null - } - ] -} -``` - -##### `analyse ruler` - -This command will run against your Grafana Cloud Prometheus instance and will fetch its rule groups. It will then extract the Prometheus metrics used in the rule queries. The output is a JSON file. - -###### Configuration - -| Env Variables | Flag | Description | -| ---------------- | --------- | ------------------------------------------------------- | -| CORTEX_ADDRESS | `address` | Address of the Prometheus instance. | -| CORTEX_TENANT_ID | `id` | If you're using Grafana Cloud this is your instance ID. | -| CORTEX_API_KEY | `key` | If you're using Grafana Cloud this is your API Key. | -| \_\_ | `output` | The output file path. metrics-in-ruler.json by default. | - -###### Running the command - -```shell -cortextool analyse ruler --address=https://prometheus-blocks-prod-us-central1.grafana.net --id=<1234> --key= -``` - -###### Sample output - -```json -{ - "metricsUsed": [ - "apiserver_request_duration_seconds_bucket", - "container_cpu_usage_seconds_total", - "scheduler_scheduling_algorithm_duration_seconds_bucket" - ... - ], - "ruleGroups": [ - { - "namspace": "prometheus_rules", - "name": "kube-apiserver.rules", - "metrics": [ - "apiserver_request_duration_seconds_bucket", - "apiserver_request_duration_seconds_count", - "apiserver_request_total" - ], - "parse_errors": null - }, - ... -} -``` - -##### `analyse prometheus` - -This command will run against your Prometheus / Cloud Prometheus instance. It will then use the output from `analyse grafana` and `analyse ruler` to show you how many series in the Prometheus server are actually being used in dashboards and rules. Also, it'll show which metrics exist in Grafana Cloud that are **not** in dashboards or rules. The output is a JSON file. - -###### Configuration - -| Env Variables | Flag | Description | -| ---------------- | ---------------------- | ---------------------------------------------------------------------------- | -| CORTEX_ADDRESS | `address` | Address of the Prometheus instance. | -| CORTEX_TENANT_ID | `id` | If you're using Grafana Cloud this is your instance ID. | -| CORTEX_API_KEY | `key` | If you're using Grafana Cloud this is your API Key. | -| \_\_ | `grafana-metrics-file` | The dashboard metrics input file path. `metrics-in-grafana.json` by default. | -| \_\_ | `ruler-metrics-file` | The rules metrics input file path. `metrics-in-ruler.json` by default. | -| \_\_ | `output` | The output file path. `prometheus-metrics.json` by default. | - -###### Running the command - -```shell -cortextool analyse prometheus --address=https://prometheus-blocks-prod-us-central1.grafana.net --id=<1234> --key= --log.level=debug -``` - -###### Sample output - -```json -{ - "total_active_series": 38184, - "in_use_active_series": 14047, - "additional_active_series": 24137, - "in_use_metric_counts": [ - { - "metric": "apiserver_request_duration_seconds_bucket", - "count": 11400, - "job_counts": [ - { - "job": "apiserver", - "count": 11400 - } - ] - }, - { - "metric": "apiserver_request_total", - "count": 684, - "job_counts": [ - { - "job": "apiserver", - "count": 684 - } - ] - }, - ... - ], - "additional_metric_counts": [ - { - "metric": "etcd_request_duration_seconds_bucket", - "count": 2688, - "job_counts": [ - { - "job": "apiserver", - "count": 2688 - } - ] - }, - ... -``` - -##### `analyse dashboard` - -This command accepts Grafana dashboard JSON files as input and extracts Prometheus metrics used in the queries. The output is a JSON file compatible with `analyse prometheus`. - -###### Running the command - -```shell -cortextool analyse dashboard ./dashboard_one.json ./dashboard_two.json ... -``` - -##### `analyse rule-file` - -This command accepts Prometheus rule YAML files as input and extracts Prometheus metrics used in the queries. The output is a JSON file compatible with `analyse prometheus`. - -###### Running the command - -```shell -cortextool analyse rule-file ./rule_file_one.yaml ./rule_file_two.yaml ... -``` - -## chunktool - -This repo also contains the `chunktool`. A client meant to interact with chunks stored and indexed in cortex backends. - -##### Chunk Delete - -The delete command currently cleans all index entries pointing to chunks in the specified index. Only bigtable and the v10 schema are currently fully supported. This will not delete the entire index entry, only the corresponding chunk entries within the index row. - -##### Chunk Migrate - -The migrate command helps with migrating chunks across cortex clusters. It also takes care of setting right index in the new cluster as per the specified schema config. - -As of now it only supports `Bigtable` or `GCS` as a source to read chunks from for migration while for writing it supports all the storages that Cortex supports. -More details about it [here](./pkg/chunk/migrate/README.md) - -##### Chunk Validate/Clean-Index - -The `chunk validate-index` and `chunk clean-index` command allows users to scan their index and chunk backends for invalid entries. The `validate-index` command will find invalid entries and ouput them to a CSV file. The `clean-index` command will take that CSV file as input and delete the invalid entries. - -## logtool - -A CLI tool to parse Cortex query-frontend logs and formats them for easy analysis. - -``` -Options: - -dur duration - only show queries which took longer than this duration, e.g. -dur 10s - -query - show the query - -utc - show timestamp in UTC time -``` - -Feed logs into it using [`logcli`](https://github.com/grafana/loki/blob/master/docs/getting-started/logcli.md) from Loki, [`kubectl`](https://kubernetes.io/docs/reference/kubectl/overview/) for Kubernetes, `cat` from a file, or any other way to get raw logs: - -Loki `logcli` example: - -``` -$ logcli query '{cluster="us-central1", name="query-frontend", namespace="dev"}' --limit=5000 --since=3h --forward -o raw | ./logtool -dur 5s -https://logs-dev-ops-tools1.grafana.net/loki/api/v1/query_range?direction=FORWARD&end=1591119479093405000&limit=5000&query=%7Bcluster%3D%22us-central1%22%2C+name%3D%22query-frontend%22%2C+namespace%3D%22dev%22%7D&start=1591108679093405000 -Common labels: {cluster="us-central1", container_name="query-frontend", job="dev/query-frontend", level="debug", name="query-frontend", namespace="dev", pod_template_hash="7cd4bf469d", stream="stderr"} - -Timestamp TraceID Length Duration Status Path -2020-06-02 10:38:40.34205349 -0400 EDT 1f2533b40f7711d3 12h0m0s 21.92465802s (200) /api/prom/api/v1/query_range -2020-06-02 10:40:25.171649132 -0400 EDT 2ac59421db0000d8 168h0m0s 16.378698276s (200) /api/prom/api/v1/query_range -2020-06-02 10:40:29.698167258 -0400 EDT 3fd088d900160ba8 168h0m0s 20.912864541s (200) /api/prom/api/v1/query_range -``` - -``` -$ cat query-frontend-logs.log | ./logtool -dur 5s -Timestamp TraceID Length Duration Status Path -2020-05-26 13:51:15.0577354 -0400 EDT 76b9939fd5c78b8f 6h0m0s 10.249149614s (200) /api/prom/api/v1/query_range -2020-05-26 13:52:15.771988849 -0400 EDT 2e7473ab10160630 10h33m0s 7.472855362s (200) /api/prom/api/v1/query_range -2020-05-26 13:53:46.712563497 -0400 EDT 761f3221dcdd85de 10h33m0s 11.874296689s (200) /api/prom/api/v1/query_range -``` - -## benchtool - -A tool for benchmarking a Prometheus remote-write backend and PromQL compatible -API. It allows for metrics to be generated using a [workload -file](docs/benchtool.md). - -### License - -Licensed Apache 2.0, see [LICENSE](LICENSE). diff --git a/cmd/mimirtool/main.go b/cmd/mimirtool/main.go index 1cf579075ff..0942c45dc9a 100644 --- a/cmd/mimirtool/main.go +++ b/cmd/mimirtool/main.go @@ -29,17 +29,19 @@ var ( ) func main() { - app := kingpin.New("mimirtool", "A command-line tool to manage cortex.") - logConfig.Register(app) - alertCommand.Register(app) - alertmanagerCommand.Register(app) - ruleCommand.Register(app) - pushGateway.Register(app) - loadgenCommand.Register(app) - remoteReadCommand.Register(app) - aclCommand.Register(app) - analyseCommand.Register(app) - bucketValidateCommand.Register(app) + app := kingpin.New("mimirtool", "A command-line tool to manage mimir and GEM.") + + envVars := commands.NewEnvVarsWithPrefix("MIMIR") + logConfig.Register(app, envVars) + alertCommand.Register(app, envVars) + alertmanagerCommand.Register(app, envVars) + ruleCommand.Register(app, envVars) + pushGateway.Register(app, envVars) + loadgenCommand.Register(app, envVars) + remoteReadCommand.Register(app, envVars) + aclCommand.Register(app, envVars) + analyseCommand.Register(app, envVars) + bucketValidateCommand.Register(app, envVars) app.Command("version", "Get the version of the mimirtool CLI").Action(func(k *kingpin.ParseContext) error { fmt.Print(version.Template) diff --git a/cmd/test-exporter/Dockerfile b/cmd/test-exporter/Dockerfile deleted file mode 100644 index 35f3a5b146d..00000000000 --- a/cmd/test-exporter/Dockerfile +++ /dev/null @@ -1,14 +0,0 @@ -# SPDX-License-Identifier: AGPL-3.0-only -# Provenance-includes-location: https://github.com/cortexproject/cortex/cmd/test-exporter/Dockerfile -# Provenance-includes-license: Apache-2.0 -# Provenance-includes-copyright: The Cortex Authors. - -FROM alpine:3.13 -RUN apk add --no-cache ca-certificates -COPY test-exporter / -ENTRYPOINT ["/test-exporter"] - -ARG revision -LABEL org.opencontainers.image.title="test-exporter" \ - org.opencontainers.image.source="https://github.com/grafana/mimir/tree/main/cmd/test-exporter" \ - org.opencontainers.image.revision="${revision}" diff --git a/cmd/test-exporter/main.go b/cmd/test-exporter/main.go deleted file mode 100644 index fb7b851fbb4..00000000000 --- a/cmd/test-exporter/main.go +++ /dev/null @@ -1,75 +0,0 @@ -// SPDX-License-Identifier: AGPL-3.0-only -// Provenance-includes-location: https://github.com/cortexproject/cortex/blob/master/cmd/test-exporter/main.go -// Provenance-includes-license: Apache-2.0 -// Provenance-includes-copyright: The Cortex Authors. - -package main - -import ( - "flag" - "math" - "time" - - "github.com/go-kit/log/level" - "github.com/grafana/dskit/flagext" - "github.com/prometheus/client_golang/prometheus" - "github.com/weaveworks/common/server" - "github.com/weaveworks/common/tracing" - - "github.com/grafana/mimir/pkg/testexporter/correctness" - "github.com/grafana/mimir/pkg/util/log" -) - -var ( - unixStart = time.Unix(0, 0) -) - -func main() { - var ( - serverConfig server.Config - runnerConfig correctness.RunnerConfig - ) - flagext.RegisterFlags(&serverConfig, &runnerConfig) - flag.Parse() - - log.InitLogger(&serverConfig) - - // Setting the environment variable JAEGER_AGENT_HOST enables tracing - if trace, err := tracing.NewFromEnv("test-exporter"); err != nil { - level.Error(log.Logger).Log("msg", "Failed to setup tracing", "err", err.Error()) - } else { - defer trace.Close() - } - - server, err := server.New(serverConfig) - log.CheckFatal("initializing server", err) - defer server.Shutdown() - - runner, err := correctness.NewRunner(runnerConfig) - log.CheckFatal("initializing runner", err) - defer runner.Stop() - - runner.Add(correctness.NewSimpleTestCase("now_seconds", func(t time.Time) float64 { - return t.Sub(unixStart).Seconds() - }, runnerConfig.CommonTestConfig)) - - runner.Add(correctness.NewSimpleTestCase("sine_wave", func(t time.Time) float64 { - // With a 15-second scrape interval this gives a ten-minute period - period := float64(40 * runnerConfig.CommonTestConfig.ScrapeInterval.Nanoseconds()) - radians := float64(t.UnixNano()) / period * 2 * math.Pi - return math.Sin(radians) - }, runnerConfig.CommonTestConfig)) - - if runnerConfig.EnableDeleteSeriesTest { - runnerConfig.DeleteSeriesTestConfig.ExtraSelectors = runnerConfig.ExtraSelectors - runnerConfig.DeleteSeriesTestConfig.PrometheusAddr = runnerConfig.PrometheusAddr - runnerConfig.DeleteSeriesTestConfig.UserID = runnerConfig.UserID - runner.Add(correctness.NewDeleteSeriesTest("delete_series", func(t time.Time) float64 { - return t.Sub(unixStart).Seconds() - }, runnerConfig.DeleteSeriesTestConfig, runnerConfig.CommonTestConfig)) - } - - prometheus.MustRegister(runner) - err = server.Run() - log.CheckFatal("running server", err) -} diff --git a/cmd/thanosconvert/main.go b/cmd/thanosconvert/main.go deleted file mode 100644 index 1d6af744c4a..00000000000 --- a/cmd/thanosconvert/main.go +++ /dev/null @@ -1,88 +0,0 @@ -// SPDX-License-Identifier: AGPL-3.0-only -// Provenance-includes-location: https://github.com/cortexproject/cortex/blob/master/cmd/thanosconvert/main.go -// Provenance-includes-license: Apache-2.0 -// Provenance-includes-copyright: The Cortex Authors. - -package main - -import ( - "context" - "flag" - "fmt" - "io/ioutil" - "os" - "strings" - - "github.com/weaveworks/common/logging" - "gopkg.in/yaml.v2" - - "github.com/grafana/mimir/pkg/storage/bucket" - "github.com/grafana/mimir/pkg/util/log" - "github.com/grafana/mimir/tools/thanosconvert" -) - -func main() { - var ( - configFilename string - dryRun bool - cfg bucket.Config - ) - - logfmt, loglvl := logging.Format{}, logging.Level{} - logfmt.RegisterFlags(flag.CommandLine) - loglvl.RegisterFlags(flag.CommandLine) - cfg.RegisterFlags(flag.CommandLine) - flag.StringVar(&configFilename, "config", "", "Path to bucket config YAML") - flag.BoolVar(&dryRun, "dry-run", false, "Don't make changes; only report what needs to be done") - flag.Usage = func() { - fmt.Fprintf(flag.CommandLine.Output(), "%s is a tool to convert block metadata from Thanos.\nPlease see %s for instructions on how to run it.\n\n", os.Args[0], "https://cortexmetrics.io/docs/blocks-storage/migrate-storage-from-thanos-and-prometheus/") - fmt.Fprintf(flag.CommandLine.Output(), "Flags:\n") - flag.PrintDefaults() - } - flag.Parse() - - logger := log.NewDefaultLogger(loglvl, logfmt) - - if configFilename != "" { - buf, err := ioutil.ReadFile(configFilename) - if err != nil { - fatal("failed to load config file from %s: %v", configFilename, err) - } - err = yaml.UnmarshalStrict(buf, &cfg) - if err != nil { - fatal("failed to parse config file: %v", err) - } - } - - if err := cfg.Validate(); err != nil { - fatal("bucket config is invalid: %v", err) - } - - ctx := context.Background() - - converter, err := thanosconvert.NewThanosBlockConverter(ctx, cfg, dryRun, logger) - if err != nil { - fatal("couldn't initilize converter: %v", err) - } - - iterCtx := context.Background() - results, err := converter.Run(iterCtx) - - fmt.Println("Results:") - for user, res := range results { - fmt.Printf("User %s:\n", user) - fmt.Printf(" Converted %d:\n %s", len(res.ConvertedBlocks), strings.Join(res.ConvertedBlocks, ",")) - fmt.Printf(" Unchanged %d:\n %s", len(res.UnchangedBlocks), strings.Join(res.UnchangedBlocks, ",")) - fmt.Printf(" Failed %d:\n %s", len(res.FailedBlocks), strings.Join(res.FailedBlocks, ",")) - } - - if err != nil { - fatal("converter failed: %v", err) - } - -} - -func fatal(msg string, args ...interface{}) { - fmt.Fprintf(os.Stderr, msg+"\n", args...) - os.Exit(1) -} diff --git a/code-of-conduct.md b/code-of-conduct.md deleted file mode 100644 index fb32cfa04ee..00000000000 --- a/code-of-conduct.md +++ /dev/null @@ -1,3 +0,0 @@ -## Cortex Community Code of Conduct - -Cortex follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md). diff --git a/development/tsdb-blocks-storage-s3-single-binary/config/mimir.yaml b/development/tsdb-blocks-storage-s3-single-binary/config/mimir.yaml index 44ecd5161c5..3fa92293029 100644 --- a/development/tsdb-blocks-storage-s3-single-binary/config/mimir.yaml +++ b/development/tsdb-blocks-storage-s3-single-binary/config/mimir.yaml @@ -1,4 +1,4 @@ -auth_enabled: false +multitenancy_enabled: false distributor: pool: @@ -29,7 +29,6 @@ querier: query_ingesters_within: 3h store_gateway: - sharding_enabled: true sharding_ring: replication_factor: 1 kvstore: @@ -57,8 +56,6 @@ blocks_storage: insecure: true ruler: - enable_api: true - enable_sharding: true poll_interval: 2s ring: kvstore: diff --git a/development/tsdb-blocks-storage-s3/config/mimir.yaml b/development/tsdb-blocks-storage-s3/config/mimir.yaml index 314c7c27dd4..b63a312c6e4 100644 --- a/development/tsdb-blocks-storage-s3/config/mimir.yaml +++ b/development/tsdb-blocks-storage-s3/config/mimir.yaml @@ -1,4 +1,4 @@ -auth_enabled: false +multitenancy_enabled: false distributor: pool: @@ -34,9 +34,6 @@ memberlist: querier: query_ingesters_within: 3h - # Used when the blocks sharding is disabled. - store_gateway_addresses: store-gateway-1:9008,store-gateway-2:9009 - blocks_storage: backend: s3 @@ -74,8 +71,6 @@ blocks_storage: insecure: true ruler: - enable_api: true - enable_sharding: true ring: heartbeat_period: 5s heartbeat_timeout: 15s @@ -85,7 +80,6 @@ ruler: host: consul:8500 alertmanager_url: http://alertmanager-1:8031/alertmanager,http://alertmanager-2:8032/alertmanager,http://alertmanager-3:8033/alertmanager - enable_alertmanager_v2: false ruler_storage: backend: s3 @@ -97,7 +91,6 @@ ruler_storage: insecure: true alertmanager: - enable_api: true sharding_enabled: true sharding_ring: replication_factor: 3 @@ -121,7 +114,6 @@ compactor: compaction_interval: 30s data_dir: /tmp/mimir-compactor consistency_delay: 1m - sharding_enabled: true cleanup_interval: 1m tenant_cleanup_delay: 1m sharding_ring: @@ -131,7 +123,6 @@ compactor: host: consul:8500 store_gateway: - sharding_enabled: true sharding_ring: replication_factor: 1 heartbeat_period: 5s @@ -144,26 +135,22 @@ store_gateway: frontend: query_stats_enabled: true - -frontend_worker: - frontend_address: "query-frontend:9007" - match_max_concurrent: true - - # By setting scheduler_address, querier worker would use scheduler instead of frontend. - # scheduler_address: "query-scheduler:9012" - -query_range: - split_queries_by_interval: 24h parallelize_shardable_queries: true align_queries_with_step: true cache_results: true results_cache: - backend: "memcached" + backend: "memcached" memcached: addresses: "dns+memcached:11211" compression: snappy +frontend_worker: + frontend_address: "query-frontend:9007" + + # By setting scheduler_address, querier worker would use scheduler instead of frontend. + # scheduler_address: "query-scheduler:9012" + limits: # Limit max query time range to 31d max_query_length: 744h diff --git a/development/tsdb-blocks-storage-s3/ruler.sh b/development/tsdb-blocks-storage-s3/ruler.sh index 7a2e6c26fdc..064ea703b27 100755 --- a/development/tsdb-blocks-storage-s3/ruler.sh +++ b/development/tsdb-blocks-storage-s3/ruler.sh @@ -1,7 +1,7 @@ #!/bin/sh # SPDX-License-Identifier: AGPL-3.0-only -export CORTEX_ADDRESS=http://localhost:8021/ -export CORTEX_TENANT_ID=fake +export MIMIR_ADDRESS=http://localhost:8021/ +export MIMIR_TENANT_ID=anonymous -cortextool rules $@ +mimirtool rules $@ diff --git a/development/tsdb-blocks-storage-swift-single-binary/config/mimir.yaml b/development/tsdb-blocks-storage-swift-single-binary/config/mimir.yaml index 25cbd242f95..4e0abf6f5e7 100644 --- a/development/tsdb-blocks-storage-swift-single-binary/config/mimir.yaml +++ b/development/tsdb-blocks-storage-swift-single-binary/config/mimir.yaml @@ -1,4 +1,4 @@ -auth_enabled: false +multitenancy_enabled: false distributor: pool: @@ -29,7 +29,6 @@ querier: query_ingesters_within: 3h store_gateway: - sharding_enabled: true sharding_ring: replication_factor: 1 kvstore: @@ -58,7 +57,6 @@ blocks_storage: container_name: mimir-tsdb ruler: - enable_api: true enable_sharding: true poll_interval: 2s ring: diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 00000000000..6aa45aad7ca --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,10 @@ +.DEFAULT_GOAL: help + +# Adapted from https://www.thapaliya.com/en/writings/well-documented-makefiles/ +.PHONY: help +help: ## Display this help. +help: + @awk 'BEGIN {FS = ": ##"; printf "Usage:\n make \n\nTargets:\n"} /^[a-zA-Z0-9_\.\-\/%]+: ##/ { printf " %-45s %s\n", $$1, $$2 }' $(MAKEFILE_LIST) + +include docs.mk + diff --git a/docs/configurations/demo.yaml b/docs/configurations/demo.yaml new file mode 100644 index 00000000000..4aedfd05df3 --- /dev/null +++ b/docs/configurations/demo.yaml @@ -0,0 +1,45 @@ +# Do not use this configuration in production. +# It is for demonstration purposes only. +multitenancy_enabled: false + +blocks_storage: + backend: filesystem + bucket_store: + sync_dir: /tmp/mimir/tsdb-sync + filesystem: + dir: /tmp/mimir/data/tsdb + tsdb: + dir: /tmp/mimir/tsdb + +compactor: + data_dir: /tmp/mimir/compactor + sharding_ring: + kvstore: + store: memberlist + +distributor: + ring: + instance_addr: 127.0.0.1 + kvstore: + store: memberlist + +ingester: + lifecycler: + address: 127.0.0.1 + ring: + kvstore: + store: memberlist + replication_factor: 1 + +ruler_storage: + backend: local + local: + directory: /tmp/mimir/rules + +server: + http_listen_port: 9009 + log_level: error + +store_gateway: + sharding_ring: + replication_factor: 1 diff --git a/docs/sources/guides/overrides-exporter-runtime.yaml b/docs/configurations/overrides-exporter-runtime.yaml similarity index 88% rename from docs/sources/guides/overrides-exporter-runtime.yaml rename to docs/configurations/overrides-exporter-runtime.yaml index 46ea90fd7ee..4702a5c1c91 100644 --- a/docs/sources/guides/overrides-exporter-runtime.yaml +++ b/docs/configurations/overrides-exporter-runtime.yaml @@ -6,4 +6,3 @@ overrides: ingestion_rate: 350000 max_global_series_per_metric: 300000 max_global_series_per_user: 300000 - max_samples_per_query: 100000 diff --git a/docs/sources/configuration/prometheus-frontend.yml b/docs/configurations/prometheus-frontend.yml similarity index 91% rename from docs/sources/configuration/prometheus-frontend.yml rename to docs/configurations/prometheus-frontend.yml index 9cd09c500c4..895010091ea 100644 --- a/docs/sources/configuration/prometheus-frontend.yml +++ b/docs/configurations/prometheus-frontend.yml @@ -3,8 +3,8 @@ # the benefits of query parallelisation and caching. # Disable the requirement that every request to Cortex has a -# X-Scope-OrgID header. `fake` will be substituted in instead. -auth_enabled: false +# X-Scope-OrgID header. `anonymous` will be substituted in instead. +multitenancy_enabled: false # We only want to run the query-frontend module. target: query-frontend diff --git a/docs/sources/operations/requests-mirroring-envoy.yaml b/docs/configurations/requests-mirroring-envoy.yaml similarity index 80% rename from docs/sources/operations/requests-mirroring-envoy.yaml rename to docs/configurations/requests-mirroring-envoy.yaml index 377a5e08b74..9161553f7ad 100644 --- a/docs/sources/operations/requests-mirroring-envoy.yaml +++ b/docs/configurations/requests-mirroring-envoy.yaml @@ -6,14 +6,14 @@ admin: static_resources: listeners: - - name: cortex_listener + - name: mimir_listener address: socket_address: { address: 0.0.0.0, port_value: 9900 } filter_chains: - filters: - name: envoy.http_connection_manager config: - stat_prefix: cortex_ingress + stat_prefix: mimir_ingress route_config: name: all_routes virtual_hosts: @@ -22,7 +22,7 @@ static_resources: routes: - match: { prefix: "/" } route: - cluster: cortex_primary + cluster: mimir_primary # Specifies the upstream timeout. This spans between the point at which the entire downstream # request has been processed and when the upstream response has been completely processed. @@ -33,17 +33,17 @@ static_resources: # client will always be the one from the primary one. The requests from Envoy to the secondary # cluster are "fire and forget". request_mirror_policies: - - cluster: cortex_secondary + - cluster: mimir_secondary http_filters: - name: envoy.router clusters: - - name: cortex_primary + - name: mimir_primary type: STRICT_DNS connect_timeout: 1s - hosts: [{ socket_address: { address: cortex-primary, port_value: 80 }}] + hosts: [{ socket_address: { address: mimir-primary, port_value: 8080 }}] dns_refresh_rate: 5s - - name: cortex_secondary + - name: mimir_secondary type: STRICT_DNS connect_timeout: 1s - hosts: [{ socket_address: { address: cortex-secondary, port_value: 80 }}] + hosts: [{ socket_address: { address: mimir-secondary, port_value: 8080 }}] dns_refresh_rate: 5s diff --git a/docs/sources/configuration/single-process-config-blocks-gossip-1.yaml b/docs/configurations/single-process-config-blocks-gossip-1.yaml similarity index 80% rename from docs/sources/configuration/single-process-config-blocks-gossip-1.yaml rename to docs/configurations/single-process-config-blocks-gossip-1.yaml index 805062cc1d0..6930b8150f1 100644 --- a/docs/sources/configuration/single-process-config-blocks-gossip-1.yaml +++ b/docs/configurations/single-process-config-blocks-gossip-1.yaml @@ -4,8 +4,8 @@ # and development. # Disable the requirement that every request to Mimir has a -# X-Scope-OrgID header. `fake` will be substituted in instead. -auth_enabled: false +# X-Scope-OrgID header. `anonymous` will be substituted in instead. +multitenancy_enabled: false server: http_listen_port: 9109 @@ -17,9 +17,6 @@ server: grpc_server_max_concurrent_streams: 1000 distributor: - ring: - kvstore: - store: memberlist pool: health_check_ingesters: true @@ -50,11 +47,7 @@ ingester: final_sleep: 5s num_tokens: 512 - # Use an in memory ring store, so we don't need to launch a Consul. ring: - kvstore: - store: memberlist - replication_factor: 1 memberlist: @@ -77,23 +70,6 @@ blocks_storage: filesystem: dir: /tmp/mimir/storage -store_gateway: - sharding_enabled: true - sharding_ring: - kvstore: - store: memberlist - -frontend_worker: - match_max_concurrent: true - -ruler: - enable_api: true - enable_sharding: true - ring: - num_tokens: 512 - kvstore: - store: memberlist - ruler_storage: backend: local local: @@ -101,6 +77,3 @@ ruler_storage: compactor: data_dir: /tmp/mimir/compactor - sharding_ring: - kvstore: - store: memberlist diff --git a/docs/sources/configuration/single-process-config-blocks-gossip-2.yaml b/docs/configurations/single-process-config-blocks-gossip-2.yaml similarity index 79% rename from docs/sources/configuration/single-process-config-blocks-gossip-2.yaml rename to docs/configurations/single-process-config-blocks-gossip-2.yaml index 1291a41433e..d1e36f54018 100644 --- a/docs/sources/configuration/single-process-config-blocks-gossip-2.yaml +++ b/docs/configurations/single-process-config-blocks-gossip-2.yaml @@ -4,8 +4,8 @@ # and development. # Disable the requirement that every request to Mimir has a -# X-Scope-OrgID header. `fake` will be substituted in instead. -auth_enabled: false +# X-Scope-OrgID header. `anonymous` will be substituted in instead. +multitenancy_enabled: false server: http_listen_port: 9209 @@ -17,9 +17,6 @@ server: grpc_server_max_concurrent_streams: 1000 distributor: - ring: - kvstore: - store: memberlist pool: health_check_ingesters: true @@ -35,7 +32,8 @@ ingester: # The address to advertise for this ingester. Will be autodiscovered by # looking up address on eth0 or en0; can be specified if this fails. address: 127.0.0.1 - id: "Ingester 2" # Since + # Defaults to hostname, but we run both ingesters in this demonstration on the same machine. + id: "Ingester 2" # We don't want to join immediately, but wait a bit to see other ingesters and their tokens first. # It can take a while to have the full picture when using gossip @@ -49,11 +47,7 @@ ingester: final_sleep: 5s num_tokens: 512 - # Use an in memory ring store, so we don't need to launch a Consul. ring: - kvstore: - store: memberlist - replication_factor: 1 memberlist: @@ -76,23 +70,6 @@ blocks_storage: filesystem: dir: /tmp/mimir/storage -store_gateway: - sharding_enabled: true - sharding_ring: - kvstore: - store: memberlist - -frontend_worker: - match_max_concurrent: true - -ruler: - enable_api: true - enable_sharding: true - ring: - num_tokens: 512 - kvstore: - store: memberlist - ruler_storage: backend: local local: @@ -100,6 +77,3 @@ ruler_storage: compactor: data_dir: /tmp/mimir/compactor - sharding_ring: - kvstore: - store: memberlist diff --git a/docs/sources/configuration/single-process-config-blocks-tls.yaml b/docs/configurations/single-process-config-blocks-tls.yaml similarity index 94% rename from docs/sources/configuration/single-process-config-blocks-tls.yaml rename to docs/configurations/single-process-config-blocks-tls.yaml index 884f8fa36a9..cf7f60ada51 100644 --- a/docs/sources/configuration/single-process-config-blocks-tls.yaml +++ b/docs/configurations/single-process-config-blocks-tls.yaml @@ -4,8 +4,8 @@ # and development. # Disable the requirement that every request to Mimir has a -# X-Scope-OrgID header. `fake` will be substituted in instead. -auth_enabled: false +# X-Scope-OrgID header. `anonymous` will be substituted in instead. +multitenancy_enabled: false server: http_listen_port: 9009 @@ -89,16 +89,11 @@ compactor: store: inmemory frontend_worker: - match_max_concurrent: true grpc_client_config: tls_cert_path: "client.crt" tls_key_path: "client.key" tls_ca_path: "root.crt" -ruler: - enable_api: true - enable_sharding: false - ruler_storage: backend: local local: diff --git a/docs/sources/configuration/single-process-config-blocks.yaml b/docs/configurations/single-process-config-blocks.yaml similarity index 90% rename from docs/sources/configuration/single-process-config-blocks.yaml rename to docs/configurations/single-process-config-blocks.yaml index dee71846357..85bc34083d3 100644 --- a/docs/sources/configuration/single-process-config-blocks.yaml +++ b/docs/configurations/single-process-config-blocks.yaml @@ -4,8 +4,8 @@ # and development. # Disable the requirement that every request to Mimir has a -# X-Scope-OrgID header. `fake` will be substituted in instead. -auth_enabled: false +# X-Scope-OrgID header. `anonymous` will be substituted in instead. +multitenancy_enabled: false server: http_listen_port: 9009 @@ -77,16 +77,6 @@ blocks_storage: compactor: data_dir: /tmp/mimir/compactor - sharding_ring: - kvstore: - store: inmemory - -frontend_worker: - match_max_concurrent: true - -ruler: - enable_api: true - enable_sharding: false ruler_storage: backend: local diff --git a/docs.mk b/docs/docs.mk similarity index 50% rename from docs.mk rename to docs/docs.mk index b38f1c4b4f2..94a1aa07a88 100644 --- a/docs.mk +++ b/docs/docs.mk @@ -1,6 +1,8 @@ +SHELL = /usr/bin/env bash + DOCS_IMAGE = grafana/docs-base:latest DOCS_PROJECT = mimir -DOCS_DIR = docs/sources +DOCS_DIR = sources # This allows ports and base URL to be overridden, so services like ngrok.io can # be used to share a local running docs instances. @@ -13,23 +15,14 @@ DOCS_VERSION = next DOCS_DOCKER_RUN_FLAGS = -ti -v $(CURDIR)/$(DOCS_DIR):/hugo/content/docs/$(DOCS_PROJECT)/$(DOCS_VERSION):ro,z -p $(DOCS_HOST_PORT):$(DOCS_LISTEN_PORT) --rm $(DOCS_IMAGE) DOCS_DOCKER_CONTAINER = $(DOCS_PROJECT)-docs -# This wrapper will delete the pre-existing Grafana and Loki docs, which -# significantly slows down the build process, due to the duplication of pages -# through versioning. +# This wrapper will serve documentation on a local webserver. define docs_docker_run - @docker run --name $(DOCS_DOCKER_CONTAINER) -d $(DOCS_DOCKER_RUN_FLAGS) /bin/bash -c 'find content/docs/ -mindepth 1 -maxdepth 1 -type d -a ! -name "$(DOCS_PROJECT)" -exec rm -rf {} \; && exec $(1)'; \ - until curl -sLw '%{http_code}' http://localhost:$(DOCS_HOST_PORT)/docs/$(DOCS_PROJECT)/ | grep -q 200; do \ - echo "Waiting for docs container to be ready..."; \ - sleep 1; \ - done; \ - docker logs $(DOCS_DOCKER_CONTAINER); \ - echo ; \ - echo ------------------------- ; \ - echo Serving documentation at: ; \ - echo http://$(DOCS_BASE_URL)/docs/$(DOCS_PROJECT)/$(DOCS_VERSION)/; \ - echo ------------------------- ; \ - echo ; \ - docker attach $(DOCS_DOCKER_CONTAINER) + @echo "Documentation will be served at:" + @echo "http://$(DOCS_BASE_URL)/docs/$(DOCS_PROJECT)/$(DOCS_VERSION)/" + @echo "" + @read -p "Press a key to continue" + + @docker run --name $(DOCS_DOCKER_CONTAINER) $(DOCS_DOCKER_RUN_FLAGS) /bin/bash -c 'find content/docs/ -mindepth 1 -maxdepth 1 -type d -a ! -name "$(DOCS_PROJECT)" -exec rm -rf {} \; && touch content/docs/mimir/_index.md && exec $(1)' endef .PHONY: docs-docker-rm diff --git a/docs/sources/guides/authentication-and-authorisation.md b/docs/sources/about-authentication-and-authorization.md similarity index 59% rename from docs/sources/guides/authentication-and-authorisation.md rename to docs/sources/about-authentication-and-authorization.md index 423a7db5a4f..16fac7c26ac 100644 --- a/docs/sources/guides/authentication-and-authorisation.md +++ b/docs/sources/about-authentication-and-authorization.md @@ -1,18 +1,17 @@ --- -title: "Authentication and Authorisation" -linkTitle: "Authentication and Authorisation" -weight: 10 -slug: auth +title: "About authentication and authorization" +description: "" +weight: 100 --- -All Cortex components take the tenant ID from a header `X-Scope-OrgID` +All Grafana Mimir components take the tenant ID from a header `X-Scope-OrgID` on each request. A tenant (also called "user" or "org") is the owner of -a set of series written to and queried from Cortex. All Cortex components -trust this value completely: if you need to protect your Cortex installation +a set of series written to and queried from Grafana Mimir. All Grafana Mimir components +trust this value completely: if you need to protect your Grafana Mimir installation from accidental or malicious calls then you must add an additional layer of protection. -Typically this means you run Cortex behind a reverse proxy, and you must +Typically this means you run Grafana Mimir behind a reverse proxy, and you must ensure that all callers, both machines sending data over the `remote_write` interface and humans sending queries from GUIs, supply credentials which identify them and confirm they are authorised. @@ -20,11 +19,11 @@ which identify them and confirm they are authorised. When configuring the `remote_write` API in Prometheus there is no way to add extra headers. The user and password fields of http Basic auth, or Bearer token, can be used to convey the tenant ID and/or credentials. -See the **Cortex-Tenant** section below for one way to solve this. +See the **Grafana Mimir-Tenant** section below for one way to solve this. To disable the multi-tenant functionality, you can pass the argument -`-auth.enabled=false` to every Cortex component, which will set the OrgID -to the string `fake` for every request. +`-auth.multitenancy-enabled=false` to every Grafana Mimir component, which will set the OrgID +to the string `anonymous` for every request (configurable with `-auth.no-auth-tenant` option). Note that the tenant ID that is used to write the series to the datastore should be the same as the one you use to query the data. If they don't match @@ -32,15 +31,15 @@ you won't see any data. As of now, you can't see series from other tenants. For more information regarding the tenant ID limits, refer to: [Tenant ID limitations](./limitations.md#tenant-id-naming) -### Cortex-Tenant +### Cortex-tenant One way to add `X-Scope-OrgID` to Prometheus requests is to use a [cortex-tenant](https://github.com/blind-oracle/cortex-tenant) proxy which is able to extract the tenant ID from Prometheus labels. -It can be placed between Prometheus and Cortex and will search for a predefined -label and use its value as `X-Scope-OrgID` header when proxying the timeseries to Cortex. +It can be placed between Prometheus and Grafana Mimir and will search for a predefined +label and use its value as `X-Scope-OrgID` header when proxying the timeseries to Grafana Mimir. -This can help to run Cortex in a trusted environment where you want to separate your metrics +This can help to run Grafana Mimir in a trusted environment where you want to separate your metrics into distinct namespaces by some criteria (e.g. teams, applications, etc). -Be advised that **cortex-tenant** is a third-party community project and it's not maintained by Cortex team. +Be advised that **cortex-tenant** is a third-party community project and it's not maintained by Grafana Mimir team. diff --git a/docs/sources/guides/limitations.md b/docs/sources/about-tenant-ids.md similarity index 60% rename from docs/sources/guides/limitations.md rename to docs/sources/about-tenant-ids.md index 5951371f740..58477646529 100644 --- a/docs/sources/guides/limitations.md +++ b/docs/sources/about-tenant-ids.md @@ -1,18 +1,17 @@ --- -title: "Limitations" -linkTitle: "Limitations" -weight: 998 -slug: limitations +title: "About tenant IDs" +description: "" +weight: 100 --- -## Tenant ID naming +# About tenant IDs -The tenant ID (also called "user ID" or "org ID") is the unique identifier of a tenant within a Cortex cluster. The tenant ID is an opaque information to Cortex, which doesn't make any assumption on its format/content, but its naming has two limitations: +The tenant ID (also called "user ID" or "org ID") is the unique identifier of a tenant within a Grafana Mimir cluster. The tenant ID is an opaque information to Grafana Mimir, which doesn't make any assumption on its format/content, but its naming has two limitations: 1. Supported characters 2. Length -### Supported characters +## Supported characters The following character sets are generally **safe for use in the tenant ID**: @@ -24,7 +23,7 @@ The following character sets are generally **safe for use in the tenant ID**: - Exclamation point (`!`) - Hyphen (`-`) - Underscore (`_`) - - Single Period (`.`), but the tenant IDs `.` and `..` is considered invalid + - Single Period (`.`), but the tenant IDs `.` and `..` are considered invalid - Asterisk (`*`) - Single quote (`'`) - Open parenthesis (`(`) @@ -32,6 +31,6 @@ The following character sets are generally **safe for use in the tenant ID**: All other characters are not safe to use. In particular, slashes `/` and whitespaces (` `) are **not supported**. -### Length +## Length The tenant ID length should not exceed 150 bytes/characters. diff --git a/docs/sources/architecture.md b/docs/sources/architecture.md deleted file mode 100644 index 2903af8b365..00000000000 --- a/docs/sources/architecture.md +++ /dev/null @@ -1,238 +0,0 @@ ---- -title: "Mimir architecture" -linkTitle: "Architecture" -weight: 2 -slug: architecture ---- - -Mimir consists of multiple horizontally scalable microservices. Each microservice uses the most appropriate technique for horizontal scaling; most are stateless and can handle requests for any users while some (namely the [ingesters](#ingester)) are semi-stateful and depend on consistent hashing. This document provides a basic overview of Mimir’s architecture. - -

Mimir architecture

- -## The role of Prometheus - -Prometheus instances scrape samples from various targets and then push them to Mimir (using Prometheus' [remote write API](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations)). That remote write API emits batched [Snappy](https://google.github.io/snappy/)-compressed [Protocol Buffer](https://developers.google.com/protocol-buffers/) messages inside the body of an HTTP `PUT` request. - -Mimir requires that each HTTP request bear a header specifying a tenant ID for the request. Request authentication and authorization are handled by an external reverse proxy. - -Incoming samples (writes from Prometheus) are handled by the [distributor](#distributor) while incoming reads (PromQL queries) are handled by the [querier](#querier) or optionally by the [query frontend](#query-frontend). - -## Storage - -The Mimir storage format is based on [Prometheus TSDB](https://prometheus.io/docs/prometheus/latest/storage/): it stores each tenant's time series into their own TSDB which write out their series to a on-disk block (defaults to 2h block range periods). Each block is composed by a few files storing the blocks and the block index. - -The TSDB block files contain the samples for multiple series. The series inside the blocks are then indexed by a per-block index, which indexes metric names and labels to time series in the block files. - -Mimir requires an object store for the block files, which can be: - -- [Amazon S3](https://aws.amazon.com/s3) -- [Google Cloud Storage](https://cloud.google.com/storage/) -- [Microsoft Azure Storage](https://azure.microsoft.com/en-us/services/storage/) -- [OpenStack Swift](https://wiki.openstack.org/wiki/Swift) (experimental) -- [Local Filesystem](https://thanos.io/storage.md/#filesystem) (single node only) - -For more information, see [Blocks storage](./blocks-storage/_index.md). - -## Services - -Mimir has a service-based architecture, in which the overall system is split up into a variety of components that perform a specific task. These components run separately and in parallel. Mimir can alternatively run in a single process mode, where all components are executed within a single process. The single process mode is particularly handy for local testing and development. - -Mimir is, for the most part, a shared-nothing system. Each layer of the system can run multiple instances of each component and they don't coordinate or communicate with each other within that layer. - -The Mimir services are: - -- [Distributor](#distributor) -- [Ingester](#ingester) -- [Querier](#querier) -- [Query frontend](#query-frontend) (optional) -- [Ruler](#ruler) (optional) -- [Alertmanager](#alertmanager) (optional) -- [Configs API](#configs-api) (optional) - -### Distributor - -The **distributor** service is responsible for handling incoming samples from Prometheus. It's the first stop in the write path for series samples. Once the distributor receives samples from Prometheus, each sample is validated for correctness and to ensure that it is within the configured tenant limits, falling back to default ones in case limits have not been overridden for the specific tenant. Valid samples are then split into batches and sent to multiple [ingesters](#ingester) in parallel. - -The validation done by the distributor includes: - -- The metric labels name are formally correct -- The configured max number of labels per metric is respected -- The configured max length of a label name and value is respected -- The timestamp is not older/newer than the configured min/max time range - -Distributors are **stateless** and can be scaled up and down as needed. - -#### High Availability Tracker - -The distributor features a **High Availability (HA) Tracker**. When enabled, the distributor deduplicates incoming samples from redundant Prometheus servers. This allows you to have multiple HA replicas of the same Prometheus servers, writing the same series to Mimir and then deduplicate these series in the Mimir distributor. - -The HA Tracker deduplicates incoming samples based on a cluster and replica label. The cluster label uniquely identifies the cluster of redundant Prometheus servers for a given tenant, while the replica label uniquely identifies the replica within the Prometheus cluster. Incoming samples are considered duplicated (and thus dropped) if received by any replica which is not the current primary within a cluster. - -The HA Tracker requires a key-value (KV) store to coordinate which replica is currently elected. The distributor will only accept samples from the current leader. Samples with one or no labels (of the replica and cluster) are accepted by default and never deduplicated. - -The supported KV stores for the HA tracker are: - -- [Consul](https://www.consul.io) -- [Etcd](https://etcd.io) - -Note: Memberlist is not supported. Memberlist-based KV store propagates updates using gossip, which is very slow for HA purposes: result is that different distributors may see different Prometheus server as elected HA replica, which is definitely not desirable. - -For more information, please refer to [config for sending HA pairs data to Mimir](guides/ha-pair-handling.md) in the documentation. - -#### Hashing - -Distributors use consistent hashing, in conjunction with a configurable replication factor, to determine which ingester instance(s) should receive a given series. - -The hash is calculated using the metric name, labels and tenant ID. - -There is a trade-off associated with including labels in the hash. Writes are more balanced across ingesters, but each query needs to talk to all ingesters since a metric could be spread across multiple ingesters given different label sets. - -#### The hash ring - -A hash ring (stored in a key-value store) is used to achieve consistent hashing for the series sharding and replication across the ingesters. All [ingesters](#ingester) register themselves into the hash ring with a set of tokens they own; each token is a random unsigned 32-bit number. Each incoming series is [hashed](#hashing) in the distributor and then pushed to the ingester owning the tokens range for the series hash number plus N-1 subsequent ingesters in the ring, where N is the replication factor. - -To do the hash lookup, distributors find the smallest appropriate token whose value is larger than the [hash of the series](#hashing). When the replication factor is larger than 1, the next subsequent tokens (clockwise in the ring) that belong to different ingesters will also be included in the result. - -The effect of this hash set up is that each token that an ingester owns is responsible for a range of hashes. If there are three tokens with values 0, 25, and 50, then a hash of 3 would be given to the ingester that owns the token 25; the ingester owning token 25 is responsible for the hash range of 1-25. - -The supported KV stores for the hash ring are: - -- [Consul](https://www.consul.io) -- [Etcd](https://etcd.io) -- Gossip [memberlist](https://github.com/hashicorp/memberlist) - -#### Quorum consistency - -Since all distributors share access to the same hash ring, write requests can be sent to any distributor and you can setup a stateless load balancer in front of it. - -To ensure consistent query results, Mimir uses [Dynamo-style](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) quorum consistency on reads and writes. This means that the distributor will wait for a positive response of at least one half plus one of the ingesters to send the sample to before successfully responding to the Prometheus write request. - -#### Load balancing across distributors - -We recommend randomly load balancing write requests across distributor instances. For example, if you're running Mimir in a Kubernetes cluster, you could run the distributors as a Kubernetes [Service](https://kubernetes.io/docs/concepts/services-networking/service/). - -### Ingester - -The **ingester** service is responsible for writing incoming series to a [long-term storage backend](#storage) on the write path and returning in-memory series samples for queries on the read path. - -Incoming series are not immediately written to the storage but kept in memory and periodically flushed to the storage (2 hours by default). For this reason, the [queriers](#querier) may need to fetch samples both from ingesters and long-term storage while executing a query on the read path. - -Ingesters contain a **lifecycler** which manages the lifecycle of an ingester and stores the **ingester state** in the [hash ring](#the-hash-ring). Each ingester could be in one of the following states: - -- **`PENDING`**
- The ingester has just started. While in this state, the ingester doesn't receive neither write and read requests. -- **`JOINING`**
- The ingester is starting up and joining the ring. While in this state the ingester doesn't receive neither write and read requests. The ingester will load tokens from disk (if `-ingester.tokens-file-path` is configured) or generate a set of new random ones. Finally, the ingester optionally observes the ring for tokens conflicts and then, once any conflict is resolved, will move to `ACTIVE` state. -- **`ACTIVE`**
- The ingester is up and running. While in this state the ingester can receive both write and read requests. -- **`LEAVING`**
- The ingester is shutting down and leaving the ring. While in this state the ingester doesn't receive write requests, while it could receive read requests. -- **`UNHEALTHY`**
- The ingester has failed to heartbeat to the ring's KV Store. While in this state, distributors skip the ingester while building the replication set for incoming series and the ingester does not receive write or read requests. - -Ingesters are **semi-stateful**. - -#### Ingesters failure and data loss - -If an ingester process crashes or exits abruptly, all the in-memory series that have not yet been flushed to the long-term storage will be lost. There are two main ways to mitigate this failure mode: - -1. Replication -2. Write-ahead log (WAL) - -The **replication** is used to hold multiple (typically 3) replicas of each time series in the ingesters. If the Mimir cluster looses an ingester, the in-memory series hold by the lost ingester are also replicated at least to another ingester. In the event of a single ingester failure, no time series samples will be lost while, in the event of multiple ingesters failure, time series may be potentially lost if failure affects all the ingesters holding the replicas of a specific time series. - -The **write-ahead log** (WAL) is used to write to a persistent disk all incoming series samples until they're flushed to the long-term storage. In the event of an ingester failure, a subsequent process restart will replay the WAL and recover the in-memory series samples. - -Contrary to the sole replication and given the persistent disk data is not lost, in the event of multiple ingesters failure each ingester will recover the in-memory series samples from WAL upon subsequent restart. The replication is still recommended in order to ensure no temporary failures on the read path in the event of a single ingester failure. - -#### Ingesters write de-amplification - -Ingesters store recently received samples in-memory in order to perform write de-amplification. If the ingesters would immediately write received samples to the long-term storage, the system would be very difficult to scale due to the very high pressure on the storage. For this reason, the ingesters batch and compress samples in-memory and periodically flush them out to the storage. - -Write de-amplification is the main source of Mimir's low total cost of ownership (TCO). - -### Querier - -The **querier** service handles queries using the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language. - -Queriers fetch series samples both from the ingesters and long-term storage: the ingesters hold the in-memory series which have not yet been flushed to the long-term storage. Because of the replication factor, it is possible that the querier may receive duplicated samples; to resolve this, for a given time series the querier internally **deduplicates** samples with the same exact timestamp. - -Queriers are **stateless** and can be scaled up and down as needed. - -### Query frontend - -The **query frontend** is an **optional service** providing the querier's API endpoints and can be used to accelerate the read path. When the query frontend is in place, incoming query requests should be directed to the query frontend instead of the queriers. The querier service will be still required within the cluster, in order to execute the actual queries. - -The query frontend internally performs some query adjustments and holds queries in an internal queue. In this setup, queriers act as workers which pull jobs from the queue, execute them, and return them to the query-frontend for aggregation. Queriers need to be configured with the query frontend address (via the `-querier.frontend-address` CLI flag) in order to allow them to connect to the query frontends. - -Query frontends are **stateless**. However, due to how the internal queue works, it's recommended to run a few query frontend replicas to reap the benefit of fair scheduling. Two replicas should suffice in most cases. - -Flow of the query in the system when using query-frontend: - -1. Query is received by query frontend, which can optionally split it or serve from the cache. -2. Query frontend stores the query into in-memory queue, where it waits for some querier to pick it up. -3. Querier picks up the query, and executes it. -4. Querier sends result back to query-frontend, which then forwards it to the client. - -#### Queueing - -The query frontend queuing mechanism is used to: - -- Ensure that large queries, that could cause an out-of-memory (OOM) error in the querier, will be retried on failure. This allows administrators to under-provision memory for queries, or optimistically run more small queries in parallel, which helps to reduce the TCO. -- Prevent multiple large requests from being convoyed on a single querier by distributing them across all queriers using a first-in/first-out queue (FIFO). -- Prevent a single tenant from denial-of-service-ing (DOSing) other tenants by fairly scheduling queries between tenants. - -#### Splitting - -The query frontend splits multi-day queries into multiple single-day queries, executing these queries in parallel on downstream queriers and stitching the results back together again. This prevents large (multi-day) queries from causing out of memory issues in a single querier and helps to execute them faster. - -#### Caching - -The query frontend supports caching query results and reuses them on subsequent queries. If the cached results are incomplete, the query frontend calculates the required subqueries and executes them in parallel on downstream queriers. The query frontend can optionally align queries with their step parameter to improve the cacheability of the query results. The result cache is compatible with any Mimir caching backend (currently memcached, redis, and an in-memory cache). - -### Query Scheduler - -Query Scheduler is an **optional** service that moves the internal queue from query frontend into separate component. -This enables independent scaling of query frontends and number of queues (query scheduler). - -In order to use query scheduler, both query frontend and queriers must be configured with query scheduler address -(using `-frontend.scheduler-address` and `-querier.scheduler-address` options respectively). - -Flow of the query in the system changes when using query scheduler: - -1. Query is received by query frontend, which can optionally split it or serve from the cache. -2. Query frontend forwards the query to random query scheduler process. -3. Query scheduler stores the query into in-memory queue, where it waits for some querier to pick it up. -4. Querier picks up the query, and executes it. -5. Querier sends result back to query-frontend, which then forwards it to the client. - -Query schedulers are **stateless**. It is recommended to run two replicas to make sure queries can still be serviced while one replica is restarting. - -### Ruler - -The **ruler** is an **optional service** executing PromQL queries for recording rules and alerts. The ruler requires a database storing the recording rules and alerts for each tenant. - -Ruler is **semi-stateful** and can be scaled horizontally. -Running rules internally have state, as well as the ring the rulers initiate. -However, if the rulers all fail and restart, -Prometheus alert rules have a feature where an alert is restored and returned to a firing state -if it would have been active in its for period. -However, there would be gaps in the series generated by the recording rules. - -### Alertmanager - -The **alertmanager** is an **optional service** responsible for accepting alert notifications from the [ruler](#ruler), deduplicating and grouping them, and routing them to the correct notification channel, such as email, PagerDuty or OpsGenie. - -The Mimir alertmanager is built on top of the [Prometheus Alertmanager](https://prometheus.io/docs/alerting/alertmanager/), adding multi-tenancy support. Like the [ruler](#ruler), the alertmanager requires a database storing the per-tenant configuration. - -Alertmanager is **semi-stateful**. -The Alertmanager persists information about silences and active alerts to its disk. -If all of the alertmanager nodes failed simultaneously there would be a loss of data. - -### Configs API - -The **configs API** is an **optional service** managing the configuration of Rulers and Alertmanagers. -It provides APIs to get/set/update the ruler and alertmanager configurations and store them into backend. -Current supported backend are PostgreSQL and in-memory. - -Configs API is **stateless**. diff --git a/docs/sources/architecture/_index.md b/docs/sources/architecture/_index.md new file mode 100644 index 00000000000..b36a3faea83 --- /dev/null +++ b/docs/sources/architecture/_index.md @@ -0,0 +1,80 @@ +--- +title: "About the architecture" +description: "Overview of the architecture of Grafana Mimir." +weight: 1000 +--- + +# About the architecture + +Grafana Mimir has a service-based architecture. +The system has multiple horizontally scalable microservices that run separately and in parallel. + + + +![Architecture of Grafana Mimir](../images/architecture.png) + +## Microservices + +Most microservices are stateless and don't require any data persisted between process restarts. + +Some microservices are stateful and rely on non-volatile storage to prevent data loss between process restarts. + +A dedicate page describes each microservice in detail. + +{{< section >}} + + + +### The write path + +**Ingesters** receive incoming samples from the distributors. Each push request belongs to a tenant, and the ingester appends the received samples to the specific per-tenant TSDB stored on the local disk. The received samples are both kept in-memory and written to a write-ahead log (WAL) and used to recover the in-memory series in case the ingester abruptly terminates. The per-tenant TSDB is lazily created in each ingester as soon as the first samples are received for that tenant. + +The in-memory samples are periodically flushed to disk - and the WAL truncated - when a new TSDB block is created, which by default occurs every 2 hours. Each newly created block is then uploaded to the long-term storage and kept in the ingester until the configured `-blocks-storage.tsdb.retention-period` expires, in order to give [queriers](./querier.md) and [store-gateways](./store-gateway.md) enough time to discover the new block on the storage and download its index-header. + +In order to effectively use the **WAL** and being able to recover the in-memory series upon ingester abruptly termination, the WAL needs to be stored to a persistent disk which can survive in the event of an ingester failure (ie. AWS EBS volume or GCP persistent disk when running in the cloud). For example, if you're running the Mimir cluster in Kubernetes, you may use a StatefulSet with a persistent volume claim for the ingesters. The location on the filesystem where the WAL is stored is the same where local TSDB blocks (compacted from head) are stored and cannot be decoupled. See also the [timeline of block uploads](production-tips/#how-to-estimate--querierquery-store-after) and [disk space estimate](production-tips/#ingester-disk-space). + +#### Distributor series sharding and replication + +Due to the replication factor N (typically 3), each time series is stored by N ingesters, and each ingester writes its own block to the long-term storage. [Compactor](./compactor.md) merges blocks from multiple ingesters into a single block, and removes duplicate samples. After blocks compaction, the storage utilization is significantly reduced. + +For more information, see [Compactor](./compactor.md) and [Production tips](./production-tips.md). + +### The read path + +[Queriers](./querier.md) and [store-gateways](./store-gateway.md) periodically iterate over the storage bucket to discover blocks recently uploaded by ingesters. + +For each discovered block, queriers only download the block's `meta.json` file (containing some metadata including min and max timestamp of samples within the block), while store-gateways download the `meta.json` as well as the index-header, which is a small subset of the block's index used by the store-gateway to lookup series at query time. + +Queriers use the blocks metadata to compute the list of blocks that need to be queried at query time and fetch matching series from the store-gateway instances holding the required blocks. + +For more information, please refer to the following dedicated sections: + + + + + +## The role of Prometheus + +Prometheus instances scrape samples from various targets and then push them to Mimir (using Prometheus' [remote write API](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations)). That remote write API emits batched [Snappy](https://google.github.io/snappy/)-compressed [Protocol Buffer](https://developers.google.com/protocol-buffers/) messages inside the body of an HTTP `PUT` request. + +Mimir requires that each HTTP request bear a header specifying a tenant ID for the request. Request authentication and authorization are handled by an external reverse proxy. + +Incoming samples (writes from Prometheus) are handled by the [distributor](#distributor) while incoming reads (PromQL queries) are handled by the [querier](#querier) or optionally by the [query frontend](#query-frontend). + +## Storage + +The Mimir storage format is based on [Prometheus TSDB](https://prometheus.io/docs/prometheus/latest/storage/): it stores each tenant's time series into their own TSDB which write their series to an on-disk block (defaults to 2h block range periods). Each block is composed oy a few files storing the blocks and the block index. + +The TSDB block files contain samples for multiple series. The series inside the blocks are then indexed by a per-block index, which indexes metric names and labels to time series in the block files. + +Mimir requires an object store for the block files, which can be: + +- [Amazon S3](https://aws.amazon.com/s3) +- [Google Cloud Storage](https://cloud.google.com/storage/) +- [Microsoft Azure Storage](https://azure.microsoft.com/en-us/services/storage/) +- [OpenStack Swift](https://wiki.openstack.org/wiki/Swift) +- [Local Filesystem](https://thanos.io/storage.md/#filesystem) (single node only) + +For more information, see [Blocks storage](./blocks-storage/_index.md). + + diff --git a/docs/sources/architecture/about-the-hash-ring.md b/docs/sources/architecture/about-the-hash-ring.md new file mode 100644 index 00000000000..dd7c1af6d37 --- /dev/null +++ b/docs/sources/architecture/about-the-hash-ring.md @@ -0,0 +1,7 @@ +--- +title: "About the hash ring" +description: "" +weight: 10 +--- + +# About the hash ring diff --git a/docs/sources/architecture/about-the-key-value-store.md b/docs/sources/architecture/about-the-key-value-store.md new file mode 100644 index 00000000000..c086cc1ba43 --- /dev/null +++ b/docs/sources/architecture/about-the-key-value-store.md @@ -0,0 +1,7 @@ +--- +title: "About the key-value store" +description: "" +weight: 10 +--- + +# About the key-value store diff --git a/docs/sources/architecture/alertmanager.md b/docs/sources/architecture/alertmanager.md new file mode 100644 index 00000000000..8623186fc0e --- /dev/null +++ b/docs/sources/architecture/alertmanager.md @@ -0,0 +1,15 @@ +--- +title: "(Optional) Alertmanager" +description: "Overview of the alertmanager microservice." +weight: 20 +--- + +# (Optional) Alertmanager + +The **alertmanager** is an **optional service** responsible for accepting alert notifications from the [ruler]({{}}), deduplicating and grouping them, and routing them to the correct notification channel, such as email, PagerDuty or OpsGenie. + +The Mimir alertmanager is built on top of the [Prometheus Alertmanager](https://prometheus.io/docs/alerting/alertmanager/), adding multi-tenancy support. Like the [ruler]({{}}), the alertmanager requires a database to store the per-tenant configuration. + +Alertmanager is **semi-stateful**. +The Alertmanager persists information about silences and active alerts to its disk. +If all of the alertmanager nodes failed simultaneously there would be a loss of data. diff --git a/docs/sources/blocks-storage/compactor.md b/docs/sources/architecture/compactor.md similarity index 77% rename from docs/sources/blocks-storage/compactor.md rename to docs/sources/architecture/compactor.md index 06d0ca562fc..7f4dae0628c 100644 --- a/docs/sources/blocks-storage/compactor.md +++ b/docs/sources/architecture/compactor.md @@ -1,12 +1,19 @@ --- title: "Compactor" -linkTitle: "Compactor" -weight: 4 -slug: compactor +description: "Overview of the compactor microservice." +weight: 10 --- + + +The **[compactor](./compactor.md)** is responsible to merge and deduplicate smaller blocks into larger ones, in order to reduce the number of blocks stored in the long-term storage for a given tenant and query them more efficiently. It also keeps the [bucket index](./bucket-index.md) updated and, for this reason, it's a required component. + +The `alertmanager` and `ruler` components can also use object storage to store its configurations and rules uploaded by users. In that case a separate bucket should be created to store alertmanager configurations and rules: using the same bucket between ruler/alertmanager and blocks will cause issue with the **[compactor](./compactor.md)**. + + + The **compactor** is an service which is responsible to: - Compact multiple blocks of a given tenant into a single optimized larger block. This helps to reduce storage costs (deduplication, index size reduction), and increase query speed (querying fewer blocks is faster). @@ -25,7 +32,7 @@ The **vertical compaction** merges all the blocks of a tenant uploaded by ingest The **horizontal compaction** triggers after the vertical compaction and compacts several blocks with adjacent 2-hour range periods into a single larger block. Even though the total size of block chunks doesn't change after this compaction, it may still significantly reduce the size of the index and the index-header kept in memory by store-gateways. -![Compactor - horizontal and vertical compaction](/images/blocks-storage/compactor-horizontal-and-vertical-compaction.png) +![Compactor - horizontal and vertical compaction](../images/compactor-horizontal-and-vertical-compaction.png) @@ -48,7 +55,7 @@ Given the split blocks, the compactor then runs the **merge** stage for each sha The merge stage is then run for subsequent compaction time ranges (eg. 12h, 24h), compacting together blocks belonging to the same shard (_not shown in the picture below_). -![Compactor - split-and-merge compaction strategy](/images/blocks-storage/compactor-split-and-merge.png) +![Compactor - split-and-merge compaction strategy](../images/compactor-split-and-merge.png) @@ -70,7 +77,7 @@ Whenever the pool of compactors increase or decrease (ie. following up a scale u The compactor sharding is based on the Mimir [hash ring](../architecture.md#the-hash-ring). At startup, a compactor generates random tokens and registers itself to the ring. While running, it periodically scans the storage bucket at every interval defined by `-compactor.compaction-interval` to discover the list of tenants in the storage and compacts blocks for each tenant whose hash matches the token ranges that are assigned to the instance itself within the ring. -This feature can be enabled via `-compactor.sharding-enabled=true` and requires the backend [hash ring](../architecture.md#the-hash-ring) to be configured via `-compactor.ring.*` flags (or their respective YAML config options). +This feature requires the backend [hash ring](../architecture.md#the-hash-ring) to be configured via `-compactor.ring.*` flags (or their respective YAML config options). ### Waiting for stable ring at startup @@ -134,120 +141,117 @@ Alternatively, assuming the largest `-compactor.block-ranges` is `24h` (default) This section described the compactor configuration. For the general Cortex configuration and references to common config blocks, please refer to the [configuration documentation](../configuration/config-file-reference.md). -### `compactor_config` +### compactor_config The `compactor_config` configures the compactor service. ```yaml compactor: - # List of compaction time ranges. + # (advanced) List of compaction time ranges. # CLI flag: -compactor.block-ranges [block_ranges: | default = 2h0m0s,12h0m0s,24h0m0s] - # Number of Go routines to use when downloading blocks for compaction and - # uploading resulting blocks. + # (advanced) Number of Go routines to use when downloading blocks for + # compaction and uploading resulting blocks. # CLI flag: -compactor.block-sync-concurrency [block_sync_concurrency: | default = 8] - # Number of Go routines to use when syncing block meta files from the long - # term storage. + # (advanced) Number of Go routines to use when syncing block meta files from + # the long term storage. # CLI flag: -compactor.meta-sync-concurrency [meta_sync_concurrency: | default = 20] - # Minimum age of fresh (non-compacted) blocks before they are being processed. - # Malformed blocks older than the maximum of consistency-delay and 48h0m0s - # will be removed. + # (advanced) Minimum age of fresh (non-compacted) blocks before they are being + # processed. Malformed blocks older than the maximum of consistency-delay and + # 48h0m0s will be removed. # CLI flag: -compactor.consistency-delay [consistency_delay: | default = 0s] - # Data directory in which to cache blocks and process compactions + # Directory to temporarily store blocks during compaction. This directory is + # not required to be persisted between restarts. # CLI flag: -compactor.data-dir - [data_dir: | default = "./data"] + [data_dir: | default = "./data-compactor/"] - # The frequency at which the compaction runs + # (advanced) The frequency at which the compaction runs # CLI flag: -compactor.compaction-interval [compaction_interval: | default = 1h] - # How many times to retry a failed compaction within a single compaction run. + # (advanced) How many times to retry a failed compaction within a single + # compaction run. # CLI flag: -compactor.compaction-retries [compaction_retries: | default = 3] - # Max number of concurrent compactions running. + # (advanced) Max number of concurrent compactions running. # CLI flag: -compactor.compaction-concurrency [compaction_concurrency: | default = 1] - # How frequently compactor should run blocks cleanup and maintenance, as well - # as update the bucket index. + # (advanced) How frequently compactor should run blocks cleanup and + # maintenance, as well as update the bucket index. # CLI flag: -compactor.cleanup-interval [cleanup_interval: | default = 15m] - # Max number of tenants for which blocks cleanup and maintenance should run - # concurrently. + # (advanced) Max number of tenants for which blocks cleanup and maintenance + # should run concurrently. # CLI flag: -compactor.cleanup-concurrency [cleanup_concurrency: | default = 20] - # Time before a block marked for deletion is deleted from bucket. If not 0, - # blocks will be marked for deletion and compactor component will permanently - # delete blocks marked for deletion from the bucket. If 0, blocks will be - # deleted straight away. Note that deleting blocks immediately can cause query - # failures. + # (advanced) Time before a block marked for deletion is deleted from bucket. + # If not 0, blocks will be marked for deletion and compactor component will + # permanently delete blocks marked for deletion from the bucket. If 0, blocks + # will be deleted straight away. Note that deleting blocks immediately can + # cause query failures. # CLI flag: -compactor.deletion-delay [deletion_delay: | default = 12h] - # For tenants marked for deletion, this is time between deleting of last - # block, and doing final cleanup (marker files, debug files) of the tenant. + # (advanced) For tenants marked for deletion, this is time between deleting of + # last block, and doing final cleanup (marker files, debug files) of the + # tenant. # CLI flag: -compactor.tenant-cleanup-delay [tenant_cleanup_delay: | default = 6h] - # Max time for starting compactions for a single tenant. After this time no - # new compactions for the tenant are started before next compaction cycle. - # This can help in multi-tenant environments to avoid single tenant using all - # compaction time, but also in single-tenant environments to force new - # discovery of blocks more often. 0 = disabled. + # (advanced) Max time for starting compactions for a single tenant. After this + # time no new compactions for the tenant are started before next compaction + # cycle. This can help in multi-tenant environments to avoid single tenant + # using all compaction time, but also in single-tenant environments to force + # new discovery of blocks more often. 0 = disabled. # CLI flag: -compactor.max-compaction-time [max_compaction_time: | default = 0s] - # Number of goroutines opening blocks before compaction. + # (advanced) Number of goroutines opening blocks before compaction. # CLI flag: -compactor.max-opening-blocks-concurrency [max_opening_blocks_concurrency: | default = 1] - # Max number of blocks that can be closed concurrently during split + # (advanced) Max number of blocks that can be closed concurrently during split # compaction. Note that closing of newly compacted block uses a lot of memory # for writing index. # CLI flag: -compactor.max-closing-blocks-concurrency [max_closing_blocks_concurrency: | default = 1] - # Number of symbols flushers used when doing split compaction. + # (advanced) Number of symbols flushers used when doing split compaction. # CLI flag: -compactor.symbols-flushers-concurrency [symbols_flushers_concurrency: | default = 1] - # Comma separated list of tenants that can be compacted. If specified, only - # these tenants will be compacted by compactor, otherwise all tenants can be - # compacted. Subject to sharding. + # (advanced) Comma separated list of tenants that can be compacted. If + # specified, only these tenants will be compacted by compactor, otherwise all + # tenants can be compacted. Subject to sharding. # CLI flag: -compactor.enabled-tenants [enabled_tenants: | default = ""] - # Comma separated list of tenants that cannot be compacted by this compactor. - # If specified, and compactor would normally pick given tenant for compaction - # (via -compactor.enabled-tenants or sharding), it will be ignored instead. + # (advanced) Comma separated list of tenants that cannot be compacted by this + # compactor. If specified, and compactor would normally pick given tenant for + # compaction (via -compactor.enabled-tenants or sharding), it will be ignored + # instead. # CLI flag: -compactor.disabled-tenants [disabled_tenants: | default = ""] - # Shard workload across multiple compactor instances. Sharding is required if - # you run multiple compactor instances, in order to coordinate compactions and - # avoid race conditions leading to the same tenant blocks simultaneously - # compacted by different instances. - # CLI flag: -compactor.sharding-enabled - [sharding_enabled: | default = false] - sharding_ring: kvstore: # Backend storage to use for the ring. Supported values are: consul, etcd, # inmemory, memberlist, multi. # CLI flag: -compactor.ring.store - [store: | default = "consul"] + [store: | default = "memberlist"] - # The prefix for the keys in the store. Should end with a /. + # (advanced) The prefix for the keys in the store. Should end with a /. # CLI flag: -compactor.ring.prefix [prefix: | default = "collectors/"] @@ -260,19 +264,19 @@ compactor: [etcd: ] multi: - # Primary backend storage used by multi-client. + # (advanced) Primary backend storage used by multi-client. # CLI flag: -compactor.ring.multi.primary [primary: | default = ""] - # Secondary backend storage used by multi-client. + # (advanced) Secondary backend storage used by multi-client. # CLI flag: -compactor.ring.multi.secondary [secondary: | default = ""] - # Mirror writes to secondary store. + # (advanced) Mirror writes to secondary store. # CLI flag: -compactor.ring.multi.mirror-enabled [mirror_enabled: | default = false] - # Timeout for storing value to secondary store. + # (advanced) Timeout for storing value to secondary store. # CLI flag: -compactor.ring.multi.mirror-timeout [mirror_timeout: | default = 2s] @@ -287,7 +291,7 @@ compactor: # Minimum time to wait for ring stability at startup. 0 to disable. # CLI flag: -compactor.ring.wait-stability-min-duration - [wait_stability_min_duration: | default = 1m] + [wait_stability_min_duration: | default = 0s] # Maximum time to wait for ring stability at startup. If the compactor ring # keeps changing after this period of time, the compactor will start anyway. @@ -302,9 +306,9 @@ compactor: # CLI flag: -compactor.ring.wait-active-instance-timeout [wait_active_instance_timeout: | default = 10m] - # The sorting to use when deciding which compaction jobs should run first for - # a given tenant. Supported values are: smallest-range-oldest-blocks-first, - # newest-blocks-first. + # (advanced) The sorting to use when deciding which compaction jobs should run + # first for a given tenant. Supported values are: + # smallest-range-oldest-blocks-first, newest-blocks-first. # CLI flag: -compactor.compaction-jobs-order [compaction_jobs_order: | default = "smallest-range-oldest-blocks-first"] ``` diff --git a/docs/sources/blocks-storage/compactor.template b/docs/sources/architecture/compactor.template similarity index 90% rename from docs/sources/blocks-storage/compactor.template rename to docs/sources/architecture/compactor.template index b054bac3db4..7d9bf062486 100644 --- a/docs/sources/blocks-storage/compactor.template +++ b/docs/sources/architecture/compactor.template @@ -1,12 +1,19 @@ --- title: "Compactor" -linkTitle: "Compactor" -weight: 4 -slug: compactor +description: "Overview of the compactor microservice." +weight: 10 --- {{ .GeneratedFileWarning }} + + +The **[compactor](./compactor.md)** is responsible to merge and deduplicate smaller blocks into larger ones, in order to reduce the number of blocks stored in the long-term storage for a given tenant and query them more efficiently. It also keeps the [bucket index](./bucket-index.md) updated and, for this reason, it's a required component. + +The `alertmanager` and `ruler` components can also use object storage to store its configurations and rules uploaded by users. In that case a separate bucket should be created to store alertmanager configurations and rules: using the same bucket between ruler/alertmanager and blocks will cause issue with the **[compactor](./compactor.md)**. + + + The **compactor** is an service which is responsible to: - Compact multiple blocks of a given tenant into a single optimized larger block. This helps to reduce storage costs (deduplication, index size reduction), and increase query speed (querying fewer blocks is faster). @@ -25,7 +32,7 @@ The **vertical compaction** merges all the blocks of a tenant uploaded by ingest The **horizontal compaction** triggers after the vertical compaction and compacts several blocks with adjacent 2-hour range periods into a single larger block. Even though the total size of block chunks doesn't change after this compaction, it may still significantly reduce the size of the index and the index-header kept in memory by store-gateways. -![Compactor - horizontal and vertical compaction](/images/blocks-storage/compactor-horizontal-and-vertical-compaction.png) +![Compactor - horizontal and vertical compaction](../images/compactor-horizontal-and-vertical-compaction.png) @@ -48,7 +55,7 @@ Given the split blocks, the compactor then runs the **merge** stage for each sha The merge stage is then run for subsequent compaction time ranges (eg. 12h, 24h), compacting together blocks belonging to the same shard (_not shown in the picture below_). -![Compactor - split-and-merge compaction strategy](/images/blocks-storage/compactor-split-and-merge.png) +![Compactor - split-and-merge compaction strategy](../images/compactor-split-and-merge.png) @@ -70,7 +77,7 @@ Whenever the pool of compactors increase or decrease (ie. following up a scale u The compactor sharding is based on the Mimir [hash ring](../architecture.md#the-hash-ring). At startup, a compactor generates random tokens and registers itself to the ring. While running, it periodically scans the storage bucket at every interval defined by `-compactor.compaction-interval` to discover the list of tenants in the storage and compacts blocks for each tenant whose hash matches the token ranges that are assigned to the instance itself within the ring. -This feature can be enabled via `-compactor.sharding-enabled=true` and requires the backend [hash ring](../architecture.md#the-hash-ring) to be configured via `-compactor.ring.*` flags (or their respective YAML config options). +This feature requires the backend [hash ring](../architecture.md#the-hash-ring) to be configured via `-compactor.ring.*` flags (or their respective YAML config options). ### Waiting for stable ring at startup diff --git a/docs/sources/architecture/distributor.md b/docs/sources/architecture/distributor.md new file mode 100644 index 00000000000..1870ab1432e --- /dev/null +++ b/docs/sources/architecture/distributor.md @@ -0,0 +1,67 @@ +--- +title: "Distributor" +description: "Overview of the distributor microservice." +weight: 10 +--- + +# Distributor + +The **distributor** service is responsible for handling incoming samples from Prometheus. It's the first stop in the write path for series samples. Once the distributor receives samples from Prometheus, each sample is validated for correctness and to ensure that it is within the configured tenant limits, falling back to defaults in case limits have not been overridden for the specific tenant. Valid samples are then split into batches and sent to multiple [ingesters]({{}}) in parallel. + +The validation done by the distributor includes: + +- The metric labels name are formally correct +- The configured max number of labels per metric is respected +- The configured max length of a label name and value is respected +- The timestamp is not older/newer than the configured min/max time range + +Distributors are **stateless** and can be scaled up and down as needed. + +## High Availability Tracker + +The distributor features a **High Availability (HA) Tracker**. When enabled, the distributor deduplicates incoming samples from redundant Prometheus servers. This allows you to have multiple HA replicas of the same Prometheus servers, writing the same series to Mimir and then deduplicate these series in the Mimir distributor. + +The HA Tracker deduplicates incoming samples based on a cluster and replica label. The cluster label uniquely identifies the cluster of redundant Prometheus servers for a given tenant, while the replica label uniquely identifies the replica within the Prometheus cluster. Incoming samples are considered duplicated (and thus dropped) if received by any replica which is not the current primary within a cluster. + +The HA Tracker requires a key-value (KV) store to coordinate which replica is currently elected. The distributor will only accept samples from the current leader. Samples with one or no labels (of the replica and cluster) are accepted by default and never deduplicated. + +The supported KV stores for the HA tracker are: + +- [Consul](https://www.consul.io) +- [Etcd](https://etcd.io) + +Note: Memberlist is not supported. Memberlist-based KV stores propagate updates using the gossip protocol, which is very slow for HA purposes: the result is that different distributors may see a different Prometheus server elected as an HA replica, which is definitely not desirable. + +For more information, please refer to [config for sending HA pairs data to Mimir](guides/ha-pair-handling.md) in the documentation. + +## Hashing + +Distributors use consistent hashing, in conjunction with a configurable replication factor, to determine which ingester instance(s) should receive a given series. + +The hash is calculated using the metric name, labels and tenant ID. + +There is a trade-off associated with including labels in the hash. Writes are more balanced across ingesters, but each query needs to talk to all ingesters since a metric could be spread across multiple ingesters given different label sets. + +### The hash ring + +A hash ring (stored in a key-value store) is used to achieve consistent hashing for the series sharding and replication across the ingesters. All [ingesters]({{}}) register themselves into the hash ring with a set of tokens they own; each token is a random unsigned 32-bit integer. Each incoming series is [hashed](#hashing) in the distributor and then pushed to the ingester which owns the token range for the series hash number plus N-1 subsequent ingesters in the ring, where N is the replication factor. + +To do the hash lookup, distributors find the smallest appropriate token whose value is larger than the [hash of the series](#hashing). When the replication factor is larger than 1, the subsequent tokens (clockwise in the ring) that belong to different ingesters will also be included in the result. + +The effect of this hash set up is that each token that an ingester owns is responsible for a range of hashes. If there are three tokens with values 0, 25, and 50, then a hash of 3 would be given to the ingester that owns the token 25; the ingester owning token 25 is responsible for the hash range of 1-25. + +The supported KV stores for the hash ring are: + +- [Consul](https://www.consul.io) +- [Etcd](https://etcd.io) +- Gossip [memberlist](https://github.com/hashicorp/memberlist) + +#### Quorum consistency + +Since all distributors share access to the same hash ring, write requests can be sent to any distributor and you can setup a stateless load balancer in front of it. + +To ensure consistent query results, Mimir uses [Dynamo-style](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) quorum consistency on reads and writes. This means that the distributor will wait for a positive response of at least one half plus one of the ingesters to send the sample to before successfully responding to the Prometheus write request. + +## Load balancing across distributors + +We recommend randomly load balancing write requests across distributor instances. For example, if you're running Mimir in a Kubernetes cluster, you could run the distributors as a Kubernetes [Service](https://kubernetes.io/docs/concepts/services-networking/service/). diff --git a/docs/sources/architecture/ingester.md b/docs/sources/architecture/ingester.md new file mode 100644 index 00000000000..2b494ac016f --- /dev/null +++ b/docs/sources/architecture/ingester.md @@ -0,0 +1,45 @@ +--- +title: "Ingester" +description: "Overview of the ingester microservice." +weight: 10 +--- + +# Ingester + +The **ingester** service is responsible for writing incoming series to a [long-term storage backend](#storage) on the write path and returning in-memory series samples for queries on the read path. + +Incoming series are not immediately written to the storage but kept in memory and periodically flushed to the storage (2 hours by default). For this reason, the [queriers](#querier) may need to fetch samples both from ingesters and long-term storage while executing a query on the read path. + +Ingesters contain a **lifecycler** which manages the lifecycle of an ingester and stores the **ingester state** in the [hash ring](#the-hash-ring). Each ingester could be in one of the following states: + +- **`PENDING`**
+ The ingester has just started. While in this state, the ingester receives neither write nor read requests. +- **`JOINING`**
+ The ingester is starting up and joining the ring. While in this state the ingester receives neither write nor read requests. The ingester will load tokens from disk (if `-ingester.tokens-file-path` is configured) or generate a set of new random tokens. Finally, the ingester optionally observes the ring for token conflicts, and once resolved, will move to the `ACTIVE` state. +- **`ACTIVE`**
+ The ingester is up and running. While in this state the ingester can receive both write and read requests. +- **`LEAVING`**
+ The ingester is shutting down and leaving the ring. While in this state the ingester doesn't receive write requests, but could still receive read requests. +- **`UNHEALTHY`**
+ The ingester has failed to heartbeat to the ring's KV Store. While in this state, distributors skip the ingester while building the replication set for incoming series and the ingester does not receive write or read requests. + +Ingesters are **semi-stateful**. + +## Ingesters failure and data loss + +If an ingester process crashes or exits abruptly, all the in-memory series that have not yet been flushed to the long-term storage will be lost. There are two main ways to mitigate this failure mode: + +1. Replication +2. Write-ahead log (WAL) + +**Replication** is used to hold multiple (typically 3) replicas of each time series in the ingesters. If the Mimir cluster loses an ingester, the in-memory series held by the lost ingester are replicated to at least one other ingester. In the event of a single ingester failure, no time series samples will be lost, while in the event that multiple ingesters fail, time series may potentially be lost if the failure affects all the ingesters holding the replicas of a specific time series. + +The **write-ahead log** (WAL) is used to write to a persistent disk all incoming series samples until they're flushed to the long-term storage. In the event of an ingester failure, a subsequent process restart will replay the WAL and recover the in-memory series samples. + +Contrary to the sole replication and given the persistent disk data is not lost, in the event of the failure of multiple ingesters, each ingester will recover the in-memory series samples from WAL upon subsequent restart. Replication is still recommended in order to ensure no temporary failures on the read path in the event of a single ingester failure. + +## Ingesters write de-amplification + +Ingesters store recently received samples in-memory in order to perform write de-amplification. If the ingesters would immediately write received samples to the long-term storage, the system would be difficult to scale due to the high pressure on the storage. For this reason, the ingesters batch and compress samples in-memory and periodically flush them out to the storage. + +Write de-amplification is the main source of Mimir's low total cost of ownership (TCO). diff --git a/docs/sources/blocks-storage/querier.md b/docs/sources/architecture/querier.md similarity index 68% rename from docs/sources/blocks-storage/querier.md rename to docs/sources/architecture/querier.md index 9ce24e5e7e8..8534663f069 100644 --- a/docs/sources/blocks-storage/querier.md +++ b/docs/sources/architecture/querier.md @@ -1,8 +1,7 @@ --- title: "Querier" -linkTitle: "Querier" -weight: 2 -slug: querier +description: "Overview of the querier microservice." +weight: 10 --- @@ -90,7 +89,7 @@ _The same memcached backend cluster should be shared between store-gateways and This section described the querier configuration. For the general Cortex configuration and references to common config blocks, please refer to the [configuration documentation](../configuration/config-file-reference.md). -### `querier_config` +### querier_config The `querier_config` configures the querier. @@ -109,7 +108,7 @@ querier: # Maximum lookback beyond which queries are not sent to ingester. 0 means all # queries are sent to ingester. # CLI flag: -querier.query-ingesters-within - [query_ingesters_within: | default = 0s] + [query_ingesters_within: | default = 13h] # True to enable queriers to use an optimized implementation which passes down # to ingesters the label matchers when running the label names API. Can be @@ -129,38 +128,32 @@ querier: # CLI flag: -querier.max-query-into-future [max_query_into_future: | default = 10m] - # Comma separated list of store-gateway addresses in DNS Service Discovery - # format. This option should be set when the store-gateway sharding is - # disabled (when enabled, the store-gateway instances form a ring and - # addresses are picked from the ring). - # CLI flag: -querier.store-gateway-addresses - [store_gateway_addresses: | default = ""] - store_gateway_client: - # Enable TLS for gRPC client connecting to store-gateway. + # (advanced) Enable TLS for gRPC client connecting to store-gateway. # CLI flag: -querier.store-gateway-client.tls-enabled [tls_enabled: | default = false] - # Path to the client certificate file, which will be used for authenticating - # with the server. Also requires the key path to be configured. + # (advanced) Path to the client certificate file, which will be used for + # authenticating with the server. Also requires the key path to be + # configured. # CLI flag: -querier.store-gateway-client.tls-cert-path [tls_cert_path: | default = ""] - # Path to the key file for the client certificate. Also requires the client - # certificate to be configured. + # (advanced) Path to the key file for the client certificate. Also requires + # the client certificate to be configured. # CLI flag: -querier.store-gateway-client.tls-key-path [tls_key_path: | default = ""] - # Path to the CA certificates file to validate server certificate against. - # If not set, the host's root CA certificates are used. + # (advanced) Path to the CA certificates file to validate server certificate + # against. If not set, the host's root CA certificates are used. # CLI flag: -querier.store-gateway-client.tls-ca-path [tls_ca_path: | default = ""] - # Override the expected name on the server certificate. + # (advanced) Override the expected name on the server certificate. # CLI flag: -querier.store-gateway-client.tls-server-name [tls_server_name: | default = ""] - # Skip validating server certificate. + # (advanced) Skip validating server certificate. # CLI flag: -querier.store-gateway-client.tls-insecure-skip-verify [tls_insecure_skip_verify: | default = false] @@ -168,9 +161,9 @@ querier: # > 0, queriers fetch in-memory series from the minimum set of required # ingesters, selecting only ingesters which may have received series since # 'now - lookback period'. The lookback period should be greater or equal than - # the configured 'query store after' and 'query ingesters within'. If this - # setting is 0, queriers always query all ingesters (ingesters shuffle - # sharding on read path is disabled). + # the configured -querier.query-store-after and + # -querier.query-ingesters-within. If this setting is 0, queriers always query + # all ingesters (ingesters shuffle sharding on read path is disabled). # CLI flag: -querier.shuffle-sharding-ingesters-lookback-period [shuffle_sharding_ingesters_lookback_period: | default = 0s] @@ -189,11 +182,6 @@ querier: # CLI flag: -querier.max-samples [max_samples: | default = 50000000] - # Enable the @ modifier in PromQL. This config option should be set on - # query-frontend too when query sharding is enabled. - # CLI flag: -querier.at-modifier-enabled - [at_modifier_enabled: | default = false] - # The default evaluation interval or step size for subqueries. This config # option should be set on query-frontend too when query sharding is enabled. # CLI flag: -querier.default-evaluation-interval @@ -206,7 +194,7 @@ querier: [lookback_delta: | default = 5m] ``` -### `blocks_storage_config` +### blocks_storage_config The `blocks_storage_config` configures the blocks storage. @@ -241,14 +229,14 @@ blocks_storage: # CLI flag: -blocks-storage.s3.access-key-id [access_key_id: | default = ""] - # If enabled, use http:// for the S3 endpoint instead of https://. This - # could be useful in local dev/test environments while using an - # S3-compatible backend storage, like Minio. + # (advanced) If enabled, use http:// for the S3 endpoint instead of + # https://. This could be useful in local dev/test environments while using + # an S3-compatible backend storage, like Minio. # CLI flag: -blocks-storage.s3.insecure [insecure: | default = false] - # The signature version to use for authenticating against S3. Supported - # values are: v4, v2. + # (advanced) The signature version to use for authenticating against S3. + # Supported values are: v4, v2. # CLI flag: -blocks-storage.s3.signature-version [signature_version: | default = "v4"] @@ -257,40 +245,41 @@ blocks_storage: [sse: ] http: - # The time an idle connection will remain idle before closing. + # (advanced) The time an idle connection will remain idle before closing. # CLI flag: -blocks-storage.s3.http.idle-conn-timeout [idle_conn_timeout: | default = 1m30s] - # The amount of time the client will wait for a servers response headers. + # (advanced) The amount of time the client will wait for a servers + # response headers. # CLI flag: -blocks-storage.s3.http.response-header-timeout [response_header_timeout: | default = 2m] - # If the client connects to S3 via HTTPS and this option is enabled, the - # client will accept any certificate and hostname. + # (advanced) If the client connects to S3 via HTTPS and this option is + # enabled, the client will accept any certificate and hostname. # CLI flag: -blocks-storage.s3.http.insecure-skip-verify [insecure_skip_verify: | default = false] - # Maximum time to wait for a TLS handshake. 0 means no limit. + # (advanced) Maximum time to wait for a TLS handshake. 0 means no limit. # CLI flag: -blocks-storage.s3.tls-handshake-timeout [tls_handshake_timeout: | default = 10s] - # The time to wait for a server's first response headers after fully - # writing the request headers if the request has an Expect header. 0 to - # send the request body immediately. + # (advanced) The time to wait for a server's first response headers after + # fully writing the request headers if the request has an Expect header. 0 + # to send the request body immediately. # CLI flag: -blocks-storage.s3.expect-continue-timeout [expect_continue_timeout: | default = 1s] - # Maximum number of idle (keep-alive) connections across all hosts. 0 - # means no limit. + # (advanced) Maximum number of idle (keep-alive) connections across all + # hosts. 0 means no limit. # CLI flag: -blocks-storage.s3.max-idle-connections [max_idle_connections: | default = 100] - # Maximum number of idle (keep-alive) connections to keep per-host. If 0, - # a built-in default value is used. + # (advanced) Maximum number of idle (keep-alive) connections to keep + # per-host. If 0, a built-in default value is used. # CLI flag: -blocks-storage.s3.max-idle-connections-per-host [max_idle_connections_per_host: | default = 100] - # Maximum number of connections per host. 0 means no limit. + # (advanced) Maximum number of connections per host. 0 means no limit. # CLI flag: -blocks-storage.s3.max-connections-per-host [max_connections_per_host: | default = 0] @@ -324,17 +313,18 @@ blocks_storage: # CLI flag: -blocks-storage.azure.endpoint-suffix [endpoint_suffix: | default = ""] - # Number of retries for recoverable errors + # (advanced) Number of retries for recoverable errors # CLI flag: -blocks-storage.azure.max-retries [max_retries: | default = 20] - # If set, this URL is used instead of + # (advanced) If set, this URL is used instead of # https://. for obtaining # ServicePrincipalToken from MSI. # CLI flag: -blocks-storage.azure.msi-resource [msi_resource: | default = ""] - # User assigned identity. If empty, then System assigned identity is used. + # (advanced) User assigned identity. If empty, then System assigned identity + # is used. # CLI flag: -blocks-storage.azure.user-assigned-id [user_assigned_id: | default = ""] @@ -401,17 +391,17 @@ blocks_storage: # CLI flag: -blocks-storage.swift.container-name [container_name: | default = ""] - # Max retries on requests error. + # (advanced) Max retries on requests error. # CLI flag: -blocks-storage.swift.max-retries [max_retries: | default = 3] - # Time after which a connection attempt is aborted. + # (advanced) Time after which a connection attempt is aborted. # CLI flag: -blocks-storage.swift.connect-timeout [connect_timeout: | default = 10s] - # Time after which an idle request is aborted. The timeout watchdog is reset - # each time some data is received, so the timeout triggers after X time no - # data is received on a request. + # (advanced) Time after which an idle request is aborted. The timeout + # watchdog is reset each time some data is received, so the timeout triggers + # after X time no data is received on a request. # CLI flag: -blocks-storage.swift.request-timeout [request_timeout: | default = 5s] @@ -423,37 +413,39 @@ blocks_storage: # This configures how the querier and store-gateway discover and synchronize # blocks stored in the bucket. bucket_store: - # Directory to store synchronized TSDB index headers. + # Directory to store synchronized TSDB index headers. This directory is not + # required to be persisted between restarts, but it's highly recommended in + # order to improve the store-gateway startup time. # CLI flag: -blocks-storage.bucket-store.sync-dir - [sync_dir: | default = "tsdb-sync"] + [sync_dir: | default = "./tsdb-sync/"] - # How frequently to scan the bucket, or to refresh the bucket index (if - # enabled), in order to look for changes (new blocks shipped by ingesters - # and blocks deleted by retention or compaction). + # (advanced) How frequently to scan the bucket, or to refresh the bucket + # index (if enabled), in order to look for changes (new blocks shipped by + # ingesters and blocks deleted by retention or compaction). # CLI flag: -blocks-storage.bucket-store.sync-interval [sync_interval: | default = 15m] - # Max number of concurrent queries to execute against the long-term storage. - # The limit is shared across all tenants. + # (advanced) Max number of concurrent queries to execute against the + # long-term storage. The limit is shared across all tenants. # CLI flag: -blocks-storage.bucket-store.max-concurrent [max_concurrent: | default = 100] - # Maximum number of concurrent tenants synching blocks. + # (advanced) Maximum number of concurrent tenants synching blocks. # CLI flag: -blocks-storage.bucket-store.tenant-sync-concurrency [tenant_sync_concurrency: | default = 10] - # Maximum number of concurrent blocks synching per tenant. + # (advanced) Maximum number of concurrent blocks synching per tenant. # CLI flag: -blocks-storage.bucket-store.block-sync-concurrency [block_sync_concurrency: | default = 20] - # Number of Go routines to use when syncing block meta files from object - # storage per tenant. + # (advanced) Number of Go routines to use when syncing block meta files from + # object storage per tenant. # CLI flag: -blocks-storage.bucket-store.meta-sync-concurrency [meta_sync_concurrency: | default = 20] - # Minimum age of a block before it's being read. Set it to safe value (e.g - # 30m) if your object storage is eventually consistent. GCS and S3 are - # (roughly) strongly consistent. + # (advanced) Minimum age of a block before it's being read. Set it to safe + # value (e.g 30m) if your object storage is eventually consistent. GCS and + # S3 are (roughly) strongly consistent. # CLI flag: -blocks-storage.bucket-store.consistency-delay [consistency_delay: | default = 0s] @@ -483,30 +475,30 @@ blocks_storage: # blocks-storage.bucket-store.chunks-cache [memcached: ] - # Size of each subrange that bucket object is split into for better - # caching. + # (advanced) Size of each subrange that bucket object is split into for + # better caching. # CLI flag: -blocks-storage.bucket-store.chunks-cache.subrange-size [subrange_size: | default = 16000] - # Maximum number of sub-GetRange requests that a single GetRange request - # can be split into when fetching chunks. Zero or negative value = - # unlimited number of sub-requests. + # (advanced) Maximum number of sub-GetRange requests that a single + # GetRange request can be split into when fetching chunks. Zero or + # negative value = unlimited number of sub-requests. # CLI flag: -blocks-storage.bucket-store.chunks-cache.max-get-range-requests [max_get_range_requests: | default = 3] - # TTL for caching object attributes for chunks. If the metadata cache is - # configured, attributes will be stored under this cache backend, + # (advanced) TTL for caching object attributes for chunks. If the metadata + # cache is configured, attributes will be stored under this cache backend, # otherwise attributes are stored in the chunks cache backend. # CLI flag: -blocks-storage.bucket-store.chunks-cache.attributes-ttl [attributes_ttl: | default = 168h] - # Maximum number of object attribute items to keep in a first level - # in-memory LRU cache. Metadata will be stored and fetched in-memory + # (advanced) Maximum number of object attribute items to keep in a first + # level in-memory LRU cache. Metadata will be stored and fetched in-memory # before hitting the cache backend. 0 to disable the in-memory cache. # CLI flag: -blocks-storage.bucket-store.chunks-cache.attributes-in-memory-max-items [attributes_in_memory_max_items: | default = 0] - # TTL for caching individual chunks subranges. + # (advanced) TTL for caching individual chunks subranges. # CLI flag: -blocks-storage.bucket-store.chunks-cache.subrange-ttl [subrange_ttl: | default = 24h] @@ -520,220 +512,251 @@ blocks_storage: # blocks-storage.bucket-store.metadata-cache [memcached: ] - # How long to cache list of tenants in the bucket. + # (advanced) How long to cache list of tenants in the bucket. # CLI flag: -blocks-storage.bucket-store.metadata-cache.tenants-list-ttl [tenants_list_ttl: | default = 15m] - # How long to cache list of blocks for each tenant. + # (advanced) How long to cache list of blocks for each tenant. # CLI flag: -blocks-storage.bucket-store.metadata-cache.tenant-blocks-list-ttl [tenant_blocks_list_ttl: | default = 5m] - # How long to cache list of chunks for a block. + # (advanced) How long to cache list of chunks for a block. # CLI flag: -blocks-storage.bucket-store.metadata-cache.chunks-list-ttl [chunks_list_ttl: | default = 24h] - # How long to cache information that block metafile exists. Also used for - # user deletion mark file. + # (advanced) How long to cache information that block metafile exists. + # Also used for user deletion mark file. # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-exists-ttl [metafile_exists_ttl: | default = 2h] - # How long to cache information that block metafile doesn't exist. Also - # used for user deletion mark file. + # (advanced) How long to cache information that block metafile doesn't + # exist. Also used for user deletion mark file. # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-doesnt-exist-ttl [metafile_doesnt_exist_ttl: | default = 5m] - # How long to cache content of the metafile. + # (advanced) How long to cache content of the metafile. # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-content-ttl [metafile_content_ttl: | default = 24h] - # Maximum size of metafile content to cache in bytes. Caching will be - # skipped if the content exceeds this size. This is useful to avoid - # network round trip for large content if the configured caching backend - # has an hard limit on cached items size (in this case, you should set - # this limit to the same limit in the caching backend). + # (advanced) Maximum size of metafile content to cache in bytes. Caching + # will be skipped if the content exceeds this size. This is useful to + # avoid network round trip for large content if the configured caching + # backend has an hard limit on cached items size (in this case, you should + # set this limit to the same limit in the caching backend). # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-max-size-bytes [metafile_max_size_bytes: | default = 1048576] - # How long to cache attributes of the block metafile. + # (advanced) How long to cache attributes of the block metafile. # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-attributes-ttl [metafile_attributes_ttl: | default = 168h] - # How long to cache attributes of the block index. + # (advanced) How long to cache attributes of the block index. # CLI flag: -blocks-storage.bucket-store.metadata-cache.block-index-attributes-ttl [block_index_attributes_ttl: | default = 168h] - # How long to cache content of the bucket index. + # (advanced) How long to cache content of the bucket index. # CLI flag: -blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl [bucket_index_content_ttl: | default = 5m] - # Maximum size of bucket index content to cache in bytes. Caching will be - # skipped if the content exceeds this size. This is useful to avoid - # network round trip for large content if the configured caching backend - # has an hard limit on cached items size (in this case, you should set - # this limit to the same limit in the caching backend). + # (advanced) Maximum size of bucket index content to cache in bytes. + # Caching will be skipped if the content exceeds this size. This is useful + # to avoid network round trip for large content if the configured caching + # backend has an hard limit on cached items size (in this case, you should + # set this limit to the same limit in the caching backend). # CLI flag: -blocks-storage.bucket-store.metadata-cache.bucket-index-max-size-bytes [bucket_index_max_size_bytes: | default = 1048576] - # Duration after which the blocks marked for deletion will be filtered out - # while fetching blocks. The idea of ignore-deletion-marks-delay is to - # ignore blocks that are marked for deletion with some delay. This ensures - # store can still serve blocks that are meant to be deleted but do not have - # a replacement yet. Default is 6h, half of the default value for - # -compactor.deletion-delay. + # (advanced) Duration after which the blocks marked for deletion will be + # filtered out while fetching blocks. The idea of + # ignore-deletion-marks-delay is to ignore blocks that are marked for + # deletion with some delay. This ensures store can still serve blocks that + # are meant to be deleted but do not have a replacement yet. # CLI flag: -blocks-storage.bucket-store.ignore-deletion-marks-delay - [ignore_deletion_mark_delay: | default = 6h] + [ignore_deletion_mark_delay: | default = 1h] bucket_index: - # True to enable querier and store-gateway to discover blocks in the - # storage via bucket index instead of bucket scanning. + # If enabled, queriers and store-gateways discover blocks by reading a + # bucket index (created and updated by the compactor) instead of + # periodically scanning the bucket. # CLI flag: -blocks-storage.bucket-store.bucket-index.enabled - [enabled: | default = false] + [enabled: | default = true] - # How frequently a bucket index, which previously failed to load, should - # be tried to load again. This option is used only by querier. + # (advanced) How frequently a bucket index, which previously failed to + # load, should be tried to load again. This option is used only by + # querier. # CLI flag: -blocks-storage.bucket-store.bucket-index.update-on-error-interval [update_on_error_interval: | default = 1m] - # How long a unused bucket index should be cached. Once this timeout - # expires, the unused bucket index is removed from the in-memory cache. - # This option is used only by querier. + # (advanced) How long a unused bucket index should be cached. Once this + # timeout expires, the unused bucket index is removed from the in-memory + # cache. This option is used only by querier. # CLI flag: -blocks-storage.bucket-store.bucket-index.idle-timeout [idle_timeout: | default = 1h] - # The maximum allowed age of a bucket index (last updated) before queries - # start failing because the bucket index is too old. The bucket index is - # periodically updated by the compactor, while this check is enforced in - # the querier (at query time). + # (advanced) The maximum allowed age of a bucket index (last updated) + # before queries start failing because the bucket index is too old. The + # bucket index is periodically updated by the compactor, while this check + # is enforced in the querier (at query time). # CLI flag: -blocks-storage.bucket-store.bucket-index.max-stale-period [max_stale_period: | default = 1h] - # Blocks with minimum time within this duration are ignored, and not loaded - # by store-gateway. Useful when used together with + # (advanced) Blocks with minimum time within this duration are ignored, and + # not loaded by store-gateway. Useful when used together with # -querier.query-store-after to prevent loading young blocks, because there # are usually many of them (depending on number of ingesters) and they are # not yet compacted. Negative values or 0 disable the filter. # CLI flag: -blocks-storage.bucket-store.ignore-blocks-within [ignore_blocks_within: | default = 0s] - # Max size - in bytes - of a chunks pool, used to reduce memory allocations. - # The pool is shared across all tenants. 0 to disable the limit. + # (advanced) Max size - in bytes - of a chunks pool, used to reduce memory + # allocations. The pool is shared across all tenants. 0 to disable the + # limit. # CLI flag: -blocks-storage.bucket-store.max-chunk-pool-bytes [max_chunk_pool_bytes: | default = 2147483648] - # Max size - in bytes - of the in-memory series hash cache. The cache is - # shared across all tenants and it's used only when query sharding is - # enabled. + # (advanced) Size - in bytes - of the smallest chunks pool bucket. + # CLI flag: -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes + [chunk_pool_min_bucket_size_bytes: | default = 16000] + + # (advanced) Size - in bytes - of the largest chunks pool bucket. + # CLI flag: -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes + [chunk_pool_max_bucket_size_bytes: | default = 50000000] + + # (advanced) Max size - in bytes - of the in-memory series hash cache. The + # cache is shared across all tenants and it's used only when query sharding + # is enabled. # CLI flag: -blocks-storage.bucket-store.series-hash-cache-max-size-bytes [series_hash_cache_max_size_bytes: | default = 1073741824] - # If enabled, store-gateway will lazy load an index-header only once - # required by a query. + # (advanced) If enabled, store-gateway will lazy load an index-header only + # once required by a query. # CLI flag: -blocks-storage.bucket-store.index-header-lazy-loading-enabled [index_header_lazy_loading_enabled: | default = true] - # If index-header lazy loading is enabled and this setting is > 0, the - # store-gateway will offload unused index-headers after 'idle timeout' - # inactivity. + # (advanced) If index-header lazy loading is enabled and this setting is > + # 0, the store-gateway will offload unused index-headers after 'idle + # timeout' inactivity. # CLI flag: -blocks-storage.bucket-store.index-header-lazy-loading-idle-timeout [index_header_lazy_loading_idle_timeout: | default = 1h] + # (advanced) Max size - in bytes - of a gap for which the partitioner + # aggregates together two bucket GET object requests. + # CLI flag: -blocks-storage.bucket-store.partitioner-max-gap-bytes + [partitioner_max_gap_bytes: | default = 524288] + + # (advanced) Controls what is the ratio of postings offsets that the store + # will hold in memory. + # CLI flag: -blocks-storage.bucket-store.posting-offsets-in-mem-sampling + [postings_offsets_in_mem_sampling: | default = 32] + tsdb: - # Local directory to store TSDBs in the ingesters. + # Directory to store TSDBs (including WAL) in the ingesters. This directory + # is required to be persisted between restarts. # CLI flag: -blocks-storage.tsdb.dir - [dir: | default = "tsdb"] + [dir: | default = "./tsdb/"] - # TSDB blocks range period. + # (advanced) TSDB blocks range period. # CLI flag: -blocks-storage.tsdb.block-ranges-period [block_ranges_period: | default = 2h0m0s] # TSDB blocks retention in the ingester before a block is removed. This - # should be larger than the block_ranges_period and large enough to give - # store-gateways and queriers enough time to discover newly uploaded blocks. + # should be larger than the -blocks-storage.tsdb.block-ranges-period, + # -querier.query-store-after and large enough to give store-gateways and + # queriers enough time to discover newly uploaded blocks. # CLI flag: -blocks-storage.tsdb.retention-period - [retention_period: | default = 6h] + [retention_period: | default = 24h] - # How frequently the TSDB blocks are scanned and new ones are shipped to the - # storage. 0 means shipping is disabled. + # (advanced) How frequently the TSDB blocks are scanned and new ones are + # shipped to the storage. 0 means shipping is disabled. # CLI flag: -blocks-storage.tsdb.ship-interval [ship_interval: | default = 1m] - # Maximum number of tenants concurrently shipping blocks to the storage. + # (advanced) Maximum number of tenants concurrently shipping blocks to the + # storage. # CLI flag: -blocks-storage.tsdb.ship-concurrency [ship_concurrency: | default = 10] - # How frequently ingesters try to compact TSDB head. Block is only created - # if data covers smallest block range. Must be greater than 0 and max 5 - # minutes. + # (advanced) How frequently ingesters try to compact TSDB head. Block is + # only created if data covers smallest block range. Must be greater than 0 + # and max 5 minutes. # CLI flag: -blocks-storage.tsdb.head-compaction-interval [head_compaction_interval: | default = 1m] - # Maximum number of tenants concurrently compacting TSDB head into a new - # block + # (advanced) Maximum number of tenants concurrently compacting TSDB head + # into a new block # CLI flag: -blocks-storage.tsdb.head-compaction-concurrency [head_compaction_concurrency: | default = 5] - # If TSDB head is idle for this duration, it is compacted. Note that up to - # 25% jitter is added to the value to avoid ingesters compacting + # (advanced) If TSDB head is idle for this duration, it is compacted. Note + # that up to 25% jitter is added to the value to avoid ingesters compacting # concurrently. 0 means disabled. # CLI flag: -blocks-storage.tsdb.head-compaction-idle-timeout [head_compaction_idle_timeout: | default = 1h] - # The write buffer size used by the head chunks mapper. Lower values reduce - # memory utilisation on clusters with a large number of tenants at the cost - # of increased disk I/O operations. + # (advanced) The write buffer size used by the head chunks mapper. Lower + # values reduce memory utilisation on clusters with a large number of + # tenants at the cost of increased disk I/O operations. # CLI flag: -blocks-storage.tsdb.head-chunks-write-buffer-size-bytes [head_chunks_write_buffer_size_bytes: | default = 4194304] - # The number of shards of series to use in TSDB (must be a power of 2). - # Reducing this will decrease memory footprint, but can negatively impact - # performance. + # (experimental) How much variance (as percentage between 0 and 1) should be + # applied to the chunk end time, to spread chunks writing across time. + # Doesn't apply to the last chunk of the chunk range. 0 means no variance. + # CLI flag: -blocks-storage.tsdb.head-chunks-end-time-variance + [head_chunks_end_time_variance: | default = 0] + + # (advanced) The number of shards of series to use in TSDB (must be a power + # of 2). Reducing this will decrease memory footprint, but can negatively + # impact performance. # CLI flag: -blocks-storage.tsdb.stripe-size [stripe_size: | default = 16384] - # True to enable TSDB WAL compression. + # (advanced) True to enable TSDB WAL compression. # CLI flag: -blocks-storage.tsdb.wal-compression-enabled [wal_compression_enabled: | default = false] - # TSDB WAL segments files max size (bytes). + # (advanced) TSDB WAL segments files max size (bytes). # CLI flag: -blocks-storage.tsdb.wal-segment-size-bytes [wal_segment_size_bytes: | default = 134217728] - # True to flush blocks to storage on shutdown. If false, incomplete blocks - # will be reused after restart. + # (advanced) True to flush blocks to storage on shutdown. If false, + # incomplete blocks will be reused after restart. # CLI flag: -blocks-storage.tsdb.flush-blocks-on-shutdown [flush_blocks_on_shutdown: | default = false] - # If TSDB has not received any data for this duration, and all blocks from - # TSDB have been shipped, TSDB is closed and deleted from local disk. If set - # to positive value, this value should be equal or higher than + # (advanced) If TSDB has not received any data for this duration, and all + # blocks from TSDB have been shipped, TSDB is closed and deleted from local + # disk. If set to positive value, this value should be equal or higher than # -querier.query-ingesters-within flag to make sure that TSDB is not closed # prematurely, which could cause partial query results. 0 or negative value # disables closing of idle TSDB. # CLI flag: -blocks-storage.tsdb.close-idle-tsdb-timeout - [close_idle_tsdb_timeout: | default = 0s] + [close_idle_tsdb_timeout: | default = 13h] - # True to enable snapshotting of in-memory TSDB data on disk when shutting - # down. + # (experimental) True to enable snapshotting of in-memory TSDB data on disk + # when shutting down. # CLI flag: -blocks-storage.tsdb.memory-snapshot-on-shutdown [memory_snapshot_on_shutdown: | default = false] - # The size of the write queue used by the head chunks mapper. Lower values - # reduce memory utilisation at the cost of potentially higher ingest - # latency. 0 disables the use of the chunk write queue. + # (experimental) The size of the write queue used by the head chunks mapper. + # Lower values reduce memory utilisation at the cost of potentially higher + # ingest latency. Value of 0 switches chunks mapper to implementation + # without a queue. # CLI flag: -blocks-storage.tsdb.head-chunks-write-queue-size [head_chunks_write_queue_size: | default = 0] - # Enables TSDB isolation feature. Disabling may improve performance. + # (advanced) Enables TSDB isolation feature. Disabling may improve + # performance. # CLI flag: -blocks-storage.tsdb.isolation-enabled [isolation_enabled: | default = true] - # Max size - in bytes - of the in-memory series hash cache. The cache is - # shared across all tenants and it's used only when query sharding is - # enabled. + # (advanced) Max size - in bytes - of the in-memory series hash cache. The + # cache is shared across all tenants and it's used only when query sharding + # is enabled. # CLI flag: -blocks-storage.tsdb.series-hash-cache-max-size-bytes [series_hash_cache_max_size_bytes: | default = 1073741824] - # limit the number of concurrently opening TSDB's on startup + # (advanced) limit the number of concurrently opening TSDB's on startup # CLI flag: -blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup [max_tsdb_opening_concurrency_on_startup: | default = 10] ``` diff --git a/docs/sources/blocks-storage/querier.template b/docs/sources/architecture/querier.template similarity index 99% rename from docs/sources/blocks-storage/querier.template rename to docs/sources/architecture/querier.template index ee944d3d07a..58bcefa1684 100644 --- a/docs/sources/blocks-storage/querier.template +++ b/docs/sources/architecture/querier.template @@ -1,8 +1,7 @@ --- title: "Querier" -linkTitle: "Querier" -weight: 2 -slug: querier +description: "Overview of the querier microservice." +weight: 10 --- {{ .GeneratedFileWarning }} diff --git a/docs/sources/architecture/query-frontend.md b/docs/sources/architecture/query-frontend.md new file mode 100644 index 00000000000..9744839a095 --- /dev/null +++ b/docs/sources/architecture/query-frontend.md @@ -0,0 +1,36 @@ +--- +title: "Query-frontend" +description: "Overview of the query-frontend microservice." +weight: 10 +--- + +# Query-frontend + +The **query-frontend** is an **optional service** that provides the querier's API endpoints and can be used to accelerate the read path. When the query frontend is in place, incoming query requests should be directed to the query frontend instead of the queriers. The querier service will still be required within the cluster, in order to execute the actual queries. + +The query frontend internally performs some query adjustments and holds queries in an internal queue. In this setup, queriers act as workers which pull jobs from the queue, execute them, and return them to the query-frontend for aggregation. Queriers need to be configured with the query frontend address (via the `-querier.frontend-address` CLI flag) to allow them to connect to the query frontends. + +Query frontends are **stateless**. However, due to how the internal queue works, it's recommended to run a few query frontend replicas to reap the benefit of fair scheduling. Two replicas should suffice in most cases. + +Flow of the query in the system when using query-frontend: + +1. Query is received by query frontend, which can optionally split it or serve from the cache. +2. Query frontend stores the query into an in-memory queue, where it waits for some querier to pick it up. +3. Querier picks up the query, and executes it. +4. Querier sends result back to query-frontend, which then forwards it to the client. + +#### Queueing + +The query frontend queuing mechanism is used to: + +- Ensure that large queries, that could cause an out-of-memory (OOM) error in the querier, will be retried on failure. This allows administrators to under-provision memory for queries, or optimistically run more small queries in parallel, which helps to reduce the TCO. +- Prevent multiple large requests from being convoyed on a single querier by distributing them across all queriers using a first-in/first-out queue (FIFO). +- Prevent a single tenant from denial-of-service-ing (DOSing) other tenants by fairly scheduling queries between tenants. + +#### Splitting + +The query frontend splits multi-day queries into multiple single-day queries, executing these queries in parallel on downstream queriers and stitching the results back together again. This prevents large (multi-day) queries from causing out of memory issues in a single querier and helps to execute them faster. + +#### Caching + +The query frontend supports caching query results and reuses them on subsequent queries. If the cached results are incomplete, the query frontend calculates the required subqueries and executes them in parallel on downstream queriers. The query frontend can optionally align queries with their step parameter to improve the cacheability of the query results. The result cache is compatible with any Mimir caching backend (currently memcached, Redis, and an in-memory cache). diff --git a/docs/sources/architecture/query-scheduler.md b/docs/sources/architecture/query-scheduler.md new file mode 100644 index 00000000000..c45b400932b --- /dev/null +++ b/docs/sources/architecture/query-scheduler.md @@ -0,0 +1,23 @@ +--- +title: "Query-scheduler" +description: "Overview of the query-scheduler microservice." +weight: 10 +--- + +# Query-scheduler + +Query-scheduler is an **optional** service that moves the internal queue from the query frontend into a separate component. +This enables independent scaling of query frontends and number of queues (query-scheduler). + +In order to use query-scheduler, both query frontend and queriers must be configured with the query-scheduler address +(using `-frontend.scheduler-address` and `-querier.scheduler-address` options respectively). + +Flow of the query in the system changes when using query-scheduler: + +1. Query is received by query frontend, which can optionally split it or serve from the cache. +2. Query frontend forwards the query to random query-scheduler process. +3. query-scheduler stores the query into an in-memory queue, where it waits for some querier to pick it up. +4. Querier picks up the query, and executes it. +5. Querier sends result back to query-frontend, which then forwards it to the client. + +Query-schedulers are **stateless**. It is recommended to run two replicas to make sure queries can still be serviced while one replica is restarting. diff --git a/docs/sources/guides/sharded_ruler.md b/docs/sources/architecture/ruler.md similarity index 58% rename from docs/sources/guides/sharded_ruler.md rename to docs/sources/architecture/ruler.md index dadc659165f..3440e3c0bdd 100644 --- a/docs/sources/guides/sharded_ruler.md +++ b/docs/sources/architecture/ruler.md @@ -1,35 +1,27 @@ --- -title: "Config for horizontally scaling the Ruler" -linkTitle: "Config for horizontally scaling the Ruler" -weight: 10 -slug: ruler-sharding +title: "(Optional) Ruler" +description: "Overview of the ruler microservice." +weight: 20 --- -## Context +# (Optional) Ruler -One option to scale the ruler is by scaling it horizontally. However, with multiple ruler instances running they will need to coordinate to determine which instance will evaluate which rule. Similar to the ingesters, the rulers establish a hash ring to divide up the responsibilities of evaluating rules. - -## Config - -In order to enable sharding in the ruler the following flag needs to be set: - -``` - -ruler.enable-sharding=true -``` +The **ruler** is an **optional service** that executes PromQL queries for recording rules and alerts. The ruler requires object storage to store the recording rules and alerts for each tenant. -In addition the ruler requires it's own ring to be configured, for instance: +Ruler is **semi-stateful** and can be scaled horizontally. +Running rules internally have state, as well as the ring the rulers initiate. +However, if the rulers all fail and restart, +Prometheus alert rules have a feature where an alert is restored and returned to a firing state +if it would have been active in its for period. +However, there would be gaps in the series generated by the recording rules. -``` - -ruler.ring.consul.hostname=consul.dev.svc.cluster.local:8500 -``` - -The only configuration that is required is to enable sharding and configure a key value store. From there the rulers will shard and handle the division of rules automatically. +One option to scale the ruler is by scaling it horizontally. However, with multiple ruler instances running they will need to coordinate to determine which instance will evaluate which rule. Similar to the ingesters, the rulers establish a hash ring to divide up the responsibilities of evaluating rules. -Unlike ingesters, rulers do not hand over responsibility: all rules are re-sharded randomly every time a ruler is added to or removed from the ring. +## Storage -## Ruler Storage +The ruler supports several kinds of storage (azure, gcs, s3, swift, filesystem, local). Most kinds of storage work with the sharded ruler configuration in an obvious way. i.e. configure all rulers to use the same backend. -The ruler supports six kinds of storage (azure, gcs, s3, swift, local). Most kinds of storage work with the sharded ruler configuration in an obvious way. i.e. configure all rulers to use the same backend. +> **Note:** The `filesystem` storage backend should only be used for testing purposes. The local implementation reads [Prometheus recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) off of the local filesystem. This is a read only backend that does not support the creation and deletion of rules through [the API](../api/_index.md#ruler). Despite the fact that it reads the local filesystem this method can still be used in a sharded ruler configuration if the operator takes care to load the same rules to every ruler. For instance this could be accomplished by mounting a [Kubernetes ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) onto every ruler pod. diff --git a/docs/sources/blocks-storage/store-gateway.md b/docs/sources/architecture/store-gateway.md similarity index 70% rename from docs/sources/blocks-storage/store-gateway.md rename to docs/sources/architecture/store-gateway.md index 5a0d3b183ad..58e09c2b775 100644 --- a/docs/sources/blocks-storage/store-gateway.md +++ b/docs/sources/architecture/store-gateway.md @@ -1,12 +1,17 @@ --- title: "Store-gateway" -linkTitle: "Store-gateway" -weight: 3 -slug: store-gateway +description: "Overview of the store-gateway microservice." +weight: 10 --- + + +The **[store-gateway](./store-gateway.md)** is responsible to query blocks and is used by the [querier](./querier.md) at query time. The store-gateway is required when running the blocks storage. + + + The store-gateway is the Cortex service responsible to query series from blocks. The store-gateway is required when running the Cortex blocks storage. The store-gateway is **semi-stateful**. @@ -44,22 +49,17 @@ Store-gateways continuously monitor the ring state and whenever the ring topolog For each block belonging to a store-gateway shard, the store-gateway loads its `meta.json`, the `deletion-mark.json` and the index-header. Once a block is loaded on the store-gateway, it's ready to be queried by queriers. When the querier queries blocks through a store-gateway, the response will contain the list of actually queried block IDs. If a querier tries to query a block which has not been loaded by a store-gateway, the querier will either retry on a different store-gateway (if blocks replication is enabled) or fail the query. -Blocks can be replicated across multiple store-gateway instances based on a replication factor configured via `-store-gateway.sharding-ring.replication-factor`. The blocks replication is used to protect from query failures caused by some blocks not loaded by any store-gateway instance at a given time like, for example, in the event of a store-gateway failure or while restarting a store-gateway instance (e.g. during a rolling update). +Blocks are replicated across multiple store-gateway instances based on a replication factor configured via `-store-gateway.sharding-ring.replication-factor`. The blocks replication is used to protect from query failures caused by some blocks not loaded by any store-gateway instance at a given time like, for example, in the event of a store-gateway failure or while restarting a store-gateway instance (e.g. during a rolling update). -This feature can be enabled via `-store-gateway.sharding-enabled=true` and requires the backend [hash ring](../architecture.md#the-hash-ring) to be configured via `-store-gateway.sharding-ring.*` flags (or their respective YAML config options). +This feature requires the backend [hash ring](../architecture.md#the-hash-ring) to be configured via `-store-gateway.sharding-ring.*` flags (or their respective YAML config options). ### Sharding strategies -The store-gateway supports two sharding strategies: - -- `default` -- `shuffle-sharding` +The store-gateway uses shuffle-sharding to spread the blocks of each tenant across a subset of store-gateway instances. The number of store-gateway instances loading blocks of a single tenant is limited and the blast radius of any issue that could be introduced by the tenant's workload is limited to its shard instances. -The **`default`** sharding strategy spreads the blocks of each tenant across all store-gateway instances. It's the easiest form of sharding supported, but doesn't provide any workload isolation between different tenants. +The default number of store-gateway instances per tenant is configured using the `-store-gateway.tenant-shard-size` flag (or their respective YAML config options). The shard size can then be overridden on a per-tenant basis setting the `store_gateway_tenant_shard_size` in the limits overrides. -The **`shuffle-sharding`** strategy spreads the blocks of a tenant across a subset of store-gateway instances. This way, the number of store-gateway instances loading blocks of a single tenant is limited and the blast radius of any issue that could be introduced by the tenant's workload is limited to its shard instances. - -The shuffle sharding strategy can be enabled via `-store-gateway.sharding-strategy=shuffle-sharding` and requires the `-store-gateway.tenant-shard-size` flag (or their respective YAML config options) to be set to the default shard size, which is the default number of store-gateway instances each tenant should be sharded to. The shard size can then be overridden on a per-tenant basis setting the `store_gateway_tenant_shard_size` in the limits overrides. +Default value for `-store-gateway.tenant-shard-size` is 0, which means that tenant's blocks are sharded across all store-gateway instances. _Please check out the [shuffle sharding documentation](../guides/shuffle-sharding.md) for more information about how it works._ @@ -177,20 +177,13 @@ _The same memcached backend cluster should be shared between store-gateways and This section described the store-gateway configuration. For the general Cortex configuration and references to common config blocks, please refer to the [configuration documentation](../configuration/config-file-reference.md). -### `store_gateway_config` +### store_gateway_config The `store_gateway_config` configures the store-gateway service. ```yaml store_gateway: - # Shard blocks across multiple store gateway instances. This option needs be - # set both on the store-gateway and querier when running in microservices - # mode. - # CLI flag: -store-gateway.sharding-enabled - [sharding_enabled: | default = false] - - # The hash ring configuration. This option is required only if blocks sharding - # is enabled. + # The hash ring configuration. sharding_ring: # The key-value store used to share the hash ring across multiple instances. # This option needs be set both on the store-gateway and querier when @@ -199,9 +192,9 @@ store_gateway: # Backend storage to use for the ring. Supported values are: consul, etcd, # inmemory, memberlist, multi. # CLI flag: -store-gateway.sharding-ring.store - [store: | default = "consul"] + [store: | default = "memberlist"] - # The prefix for the keys in the store. Should end with a /. + # (advanced) The prefix for the keys in the store. Should end with a /. # CLI flag: -store-gateway.sharding-ring.prefix [prefix: | default = "collectors/"] @@ -216,19 +209,19 @@ store_gateway: [etcd: ] multi: - # Primary backend storage used by multi-client. + # (advanced) Primary backend storage used by multi-client. # CLI flag: -store-gateway.sharding-ring.multi.primary [primary: | default = ""] - # Secondary backend storage used by multi-client. + # (advanced) Secondary backend storage used by multi-client. # CLI flag: -store-gateway.sharding-ring.multi.secondary [secondary: | default = ""] - # Mirror writes to secondary store. + # (advanced) Mirror writes to secondary store. # CLI flag: -store-gateway.sharding-ring.multi.mirror-enabled [mirror_enabled: | default = false] - # Timeout for storing value to secondary store. + # (advanced) Timeout for storing value to secondary store. # CLI flag: -store-gateway.sharding-ring.multi.mirror-timeout [mirror_timeout: | default = 2s] @@ -258,9 +251,10 @@ store_gateway: # CLI flag: -store-gateway.sharding-ring.zone-awareness-enabled [zone_awareness_enabled: | default = false] - # Minimum time to wait for ring stability at startup. 0 to disable. + # Minimum time to wait for ring stability at startup, if set to positive + # value. # CLI flag: -store-gateway.sharding-ring.wait-stability-min-duration - [wait_stability_min_duration: | default = 1m] + [wait_stability_min_duration: | default = 0s] # Maximum time to wait for ring stability at startup. If the store-gateway # ring keeps changing after this period of time, the store-gateway will @@ -280,14 +274,9 @@ store_gateway: # Unregister from the ring upon clean shutdown. # CLI flag: -store-gateway.sharding-ring.unregister-on-shutdown [unregister_on_shutdown: | default = true] - - # The sharding strategy to use. Supported values are: default, - # shuffle-sharding. - # CLI flag: -store-gateway.sharding-strategy - [sharding_strategy: | default = "default"] ``` -### `blocks_storage_config` +### blocks_storage_config The `blocks_storage_config` configures the blocks storage. @@ -322,14 +311,14 @@ blocks_storage: # CLI flag: -blocks-storage.s3.access-key-id [access_key_id: | default = ""] - # If enabled, use http:// for the S3 endpoint instead of https://. This - # could be useful in local dev/test environments while using an - # S3-compatible backend storage, like Minio. + # (advanced) If enabled, use http:// for the S3 endpoint instead of + # https://. This could be useful in local dev/test environments while using + # an S3-compatible backend storage, like Minio. # CLI flag: -blocks-storage.s3.insecure [insecure: | default = false] - # The signature version to use for authenticating against S3. Supported - # values are: v4, v2. + # (advanced) The signature version to use for authenticating against S3. + # Supported values are: v4, v2. # CLI flag: -blocks-storage.s3.signature-version [signature_version: | default = "v4"] @@ -338,40 +327,41 @@ blocks_storage: [sse: ] http: - # The time an idle connection will remain idle before closing. + # (advanced) The time an idle connection will remain idle before closing. # CLI flag: -blocks-storage.s3.http.idle-conn-timeout [idle_conn_timeout: | default = 1m30s] - # The amount of time the client will wait for a servers response headers. + # (advanced) The amount of time the client will wait for a servers + # response headers. # CLI flag: -blocks-storage.s3.http.response-header-timeout [response_header_timeout: | default = 2m] - # If the client connects to S3 via HTTPS and this option is enabled, the - # client will accept any certificate and hostname. + # (advanced) If the client connects to S3 via HTTPS and this option is + # enabled, the client will accept any certificate and hostname. # CLI flag: -blocks-storage.s3.http.insecure-skip-verify [insecure_skip_verify: | default = false] - # Maximum time to wait for a TLS handshake. 0 means no limit. + # (advanced) Maximum time to wait for a TLS handshake. 0 means no limit. # CLI flag: -blocks-storage.s3.tls-handshake-timeout [tls_handshake_timeout: | default = 10s] - # The time to wait for a server's first response headers after fully - # writing the request headers if the request has an Expect header. 0 to - # send the request body immediately. + # (advanced) The time to wait for a server's first response headers after + # fully writing the request headers if the request has an Expect header. 0 + # to send the request body immediately. # CLI flag: -blocks-storage.s3.expect-continue-timeout [expect_continue_timeout: | default = 1s] - # Maximum number of idle (keep-alive) connections across all hosts. 0 - # means no limit. + # (advanced) Maximum number of idle (keep-alive) connections across all + # hosts. 0 means no limit. # CLI flag: -blocks-storage.s3.max-idle-connections [max_idle_connections: | default = 100] - # Maximum number of idle (keep-alive) connections to keep per-host. If 0, - # a built-in default value is used. + # (advanced) Maximum number of idle (keep-alive) connections to keep + # per-host. If 0, a built-in default value is used. # CLI flag: -blocks-storage.s3.max-idle-connections-per-host [max_idle_connections_per_host: | default = 100] - # Maximum number of connections per host. 0 means no limit. + # (advanced) Maximum number of connections per host. 0 means no limit. # CLI flag: -blocks-storage.s3.max-connections-per-host [max_connections_per_host: | default = 0] @@ -405,17 +395,18 @@ blocks_storage: # CLI flag: -blocks-storage.azure.endpoint-suffix [endpoint_suffix: | default = ""] - # Number of retries for recoverable errors + # (advanced) Number of retries for recoverable errors # CLI flag: -blocks-storage.azure.max-retries [max_retries: | default = 20] - # If set, this URL is used instead of + # (advanced) If set, this URL is used instead of # https://. for obtaining # ServicePrincipalToken from MSI. # CLI flag: -blocks-storage.azure.msi-resource [msi_resource: | default = ""] - # User assigned identity. If empty, then System assigned identity is used. + # (advanced) User assigned identity. If empty, then System assigned identity + # is used. # CLI flag: -blocks-storage.azure.user-assigned-id [user_assigned_id: | default = ""] @@ -482,17 +473,17 @@ blocks_storage: # CLI flag: -blocks-storage.swift.container-name [container_name: | default = ""] - # Max retries on requests error. + # (advanced) Max retries on requests error. # CLI flag: -blocks-storage.swift.max-retries [max_retries: | default = 3] - # Time after which a connection attempt is aborted. + # (advanced) Time after which a connection attempt is aborted. # CLI flag: -blocks-storage.swift.connect-timeout [connect_timeout: | default = 10s] - # Time after which an idle request is aborted. The timeout watchdog is reset - # each time some data is received, so the timeout triggers after X time no - # data is received on a request. + # (advanced) Time after which an idle request is aborted. The timeout + # watchdog is reset each time some data is received, so the timeout triggers + # after X time no data is received on a request. # CLI flag: -blocks-storage.swift.request-timeout [request_timeout: | default = 5s] @@ -504,37 +495,39 @@ blocks_storage: # This configures how the querier and store-gateway discover and synchronize # blocks stored in the bucket. bucket_store: - # Directory to store synchronized TSDB index headers. + # Directory to store synchronized TSDB index headers. This directory is not + # required to be persisted between restarts, but it's highly recommended in + # order to improve the store-gateway startup time. # CLI flag: -blocks-storage.bucket-store.sync-dir - [sync_dir: | default = "tsdb-sync"] + [sync_dir: | default = "./tsdb-sync/"] - # How frequently to scan the bucket, or to refresh the bucket index (if - # enabled), in order to look for changes (new blocks shipped by ingesters - # and blocks deleted by retention or compaction). + # (advanced) How frequently to scan the bucket, or to refresh the bucket + # index (if enabled), in order to look for changes (new blocks shipped by + # ingesters and blocks deleted by retention or compaction). # CLI flag: -blocks-storage.bucket-store.sync-interval [sync_interval: | default = 15m] - # Max number of concurrent queries to execute against the long-term storage. - # The limit is shared across all tenants. + # (advanced) Max number of concurrent queries to execute against the + # long-term storage. The limit is shared across all tenants. # CLI flag: -blocks-storage.bucket-store.max-concurrent [max_concurrent: | default = 100] - # Maximum number of concurrent tenants synching blocks. + # (advanced) Maximum number of concurrent tenants synching blocks. # CLI flag: -blocks-storage.bucket-store.tenant-sync-concurrency [tenant_sync_concurrency: | default = 10] - # Maximum number of concurrent blocks synching per tenant. + # (advanced) Maximum number of concurrent blocks synching per tenant. # CLI flag: -blocks-storage.bucket-store.block-sync-concurrency [block_sync_concurrency: | default = 20] - # Number of Go routines to use when syncing block meta files from object - # storage per tenant. + # (advanced) Number of Go routines to use when syncing block meta files from + # object storage per tenant. # CLI flag: -blocks-storage.bucket-store.meta-sync-concurrency [meta_sync_concurrency: | default = 20] - # Minimum age of a block before it's being read. Set it to safe value (e.g - # 30m) if your object storage is eventually consistent. GCS and S3 are - # (roughly) strongly consistent. + # (advanced) Minimum age of a block before it's being read. Set it to safe + # value (e.g 30m) if your object storage is eventually consistent. GCS and + # S3 are (roughly) strongly consistent. # CLI flag: -blocks-storage.bucket-store.consistency-delay [consistency_delay: | default = 0s] @@ -564,30 +557,30 @@ blocks_storage: # blocks-storage.bucket-store.chunks-cache [memcached: ] - # Size of each subrange that bucket object is split into for better - # caching. + # (advanced) Size of each subrange that bucket object is split into for + # better caching. # CLI flag: -blocks-storage.bucket-store.chunks-cache.subrange-size [subrange_size: | default = 16000] - # Maximum number of sub-GetRange requests that a single GetRange request - # can be split into when fetching chunks. Zero or negative value = - # unlimited number of sub-requests. + # (advanced) Maximum number of sub-GetRange requests that a single + # GetRange request can be split into when fetching chunks. Zero or + # negative value = unlimited number of sub-requests. # CLI flag: -blocks-storage.bucket-store.chunks-cache.max-get-range-requests [max_get_range_requests: | default = 3] - # TTL for caching object attributes for chunks. If the metadata cache is - # configured, attributes will be stored under this cache backend, + # (advanced) TTL for caching object attributes for chunks. If the metadata + # cache is configured, attributes will be stored under this cache backend, # otherwise attributes are stored in the chunks cache backend. # CLI flag: -blocks-storage.bucket-store.chunks-cache.attributes-ttl [attributes_ttl: | default = 168h] - # Maximum number of object attribute items to keep in a first level - # in-memory LRU cache. Metadata will be stored and fetched in-memory + # (advanced) Maximum number of object attribute items to keep in a first + # level in-memory LRU cache. Metadata will be stored and fetched in-memory # before hitting the cache backend. 0 to disable the in-memory cache. # CLI flag: -blocks-storage.bucket-store.chunks-cache.attributes-in-memory-max-items [attributes_in_memory_max_items: | default = 0] - # TTL for caching individual chunks subranges. + # (advanced) TTL for caching individual chunks subranges. # CLI flag: -blocks-storage.bucket-store.chunks-cache.subrange-ttl [subrange_ttl: | default = 24h] @@ -601,220 +594,251 @@ blocks_storage: # blocks-storage.bucket-store.metadata-cache [memcached: ] - # How long to cache list of tenants in the bucket. + # (advanced) How long to cache list of tenants in the bucket. # CLI flag: -blocks-storage.bucket-store.metadata-cache.tenants-list-ttl [tenants_list_ttl: | default = 15m] - # How long to cache list of blocks for each tenant. + # (advanced) How long to cache list of blocks for each tenant. # CLI flag: -blocks-storage.bucket-store.metadata-cache.tenant-blocks-list-ttl [tenant_blocks_list_ttl: | default = 5m] - # How long to cache list of chunks for a block. + # (advanced) How long to cache list of chunks for a block. # CLI flag: -blocks-storage.bucket-store.metadata-cache.chunks-list-ttl [chunks_list_ttl: | default = 24h] - # How long to cache information that block metafile exists. Also used for - # user deletion mark file. + # (advanced) How long to cache information that block metafile exists. + # Also used for user deletion mark file. # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-exists-ttl [metafile_exists_ttl: | default = 2h] - # How long to cache information that block metafile doesn't exist. Also - # used for user deletion mark file. + # (advanced) How long to cache information that block metafile doesn't + # exist. Also used for user deletion mark file. # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-doesnt-exist-ttl [metafile_doesnt_exist_ttl: | default = 5m] - # How long to cache content of the metafile. + # (advanced) How long to cache content of the metafile. # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-content-ttl [metafile_content_ttl: | default = 24h] - # Maximum size of metafile content to cache in bytes. Caching will be - # skipped if the content exceeds this size. This is useful to avoid - # network round trip for large content if the configured caching backend - # has an hard limit on cached items size (in this case, you should set - # this limit to the same limit in the caching backend). + # (advanced) Maximum size of metafile content to cache in bytes. Caching + # will be skipped if the content exceeds this size. This is useful to + # avoid network round trip for large content if the configured caching + # backend has an hard limit on cached items size (in this case, you should + # set this limit to the same limit in the caching backend). # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-max-size-bytes [metafile_max_size_bytes: | default = 1048576] - # How long to cache attributes of the block metafile. + # (advanced) How long to cache attributes of the block metafile. # CLI flag: -blocks-storage.bucket-store.metadata-cache.metafile-attributes-ttl [metafile_attributes_ttl: | default = 168h] - # How long to cache attributes of the block index. + # (advanced) How long to cache attributes of the block index. # CLI flag: -blocks-storage.bucket-store.metadata-cache.block-index-attributes-ttl [block_index_attributes_ttl: | default = 168h] - # How long to cache content of the bucket index. + # (advanced) How long to cache content of the bucket index. # CLI flag: -blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl [bucket_index_content_ttl: | default = 5m] - # Maximum size of bucket index content to cache in bytes. Caching will be - # skipped if the content exceeds this size. This is useful to avoid - # network round trip for large content if the configured caching backend - # has an hard limit on cached items size (in this case, you should set - # this limit to the same limit in the caching backend). + # (advanced) Maximum size of bucket index content to cache in bytes. + # Caching will be skipped if the content exceeds this size. This is useful + # to avoid network round trip for large content if the configured caching + # backend has an hard limit on cached items size (in this case, you should + # set this limit to the same limit in the caching backend). # CLI flag: -blocks-storage.bucket-store.metadata-cache.bucket-index-max-size-bytes [bucket_index_max_size_bytes: | default = 1048576] - # Duration after which the blocks marked for deletion will be filtered out - # while fetching blocks. The idea of ignore-deletion-marks-delay is to - # ignore blocks that are marked for deletion with some delay. This ensures - # store can still serve blocks that are meant to be deleted but do not have - # a replacement yet. Default is 6h, half of the default value for - # -compactor.deletion-delay. + # (advanced) Duration after which the blocks marked for deletion will be + # filtered out while fetching blocks. The idea of + # ignore-deletion-marks-delay is to ignore blocks that are marked for + # deletion with some delay. This ensures store can still serve blocks that + # are meant to be deleted but do not have a replacement yet. # CLI flag: -blocks-storage.bucket-store.ignore-deletion-marks-delay - [ignore_deletion_mark_delay: | default = 6h] + [ignore_deletion_mark_delay: | default = 1h] bucket_index: - # True to enable querier and store-gateway to discover blocks in the - # storage via bucket index instead of bucket scanning. + # If enabled, queriers and store-gateways discover blocks by reading a + # bucket index (created and updated by the compactor) instead of + # periodically scanning the bucket. # CLI flag: -blocks-storage.bucket-store.bucket-index.enabled - [enabled: | default = false] + [enabled: | default = true] - # How frequently a bucket index, which previously failed to load, should - # be tried to load again. This option is used only by querier. + # (advanced) How frequently a bucket index, which previously failed to + # load, should be tried to load again. This option is used only by + # querier. # CLI flag: -blocks-storage.bucket-store.bucket-index.update-on-error-interval [update_on_error_interval: | default = 1m] - # How long a unused bucket index should be cached. Once this timeout - # expires, the unused bucket index is removed from the in-memory cache. - # This option is used only by querier. + # (advanced) How long a unused bucket index should be cached. Once this + # timeout expires, the unused bucket index is removed from the in-memory + # cache. This option is used only by querier. # CLI flag: -blocks-storage.bucket-store.bucket-index.idle-timeout [idle_timeout: | default = 1h] - # The maximum allowed age of a bucket index (last updated) before queries - # start failing because the bucket index is too old. The bucket index is - # periodically updated by the compactor, while this check is enforced in - # the querier (at query time). + # (advanced) The maximum allowed age of a bucket index (last updated) + # before queries start failing because the bucket index is too old. The + # bucket index is periodically updated by the compactor, while this check + # is enforced in the querier (at query time). # CLI flag: -blocks-storage.bucket-store.bucket-index.max-stale-period [max_stale_period: | default = 1h] - # Blocks with minimum time within this duration are ignored, and not loaded - # by store-gateway. Useful when used together with + # (advanced) Blocks with minimum time within this duration are ignored, and + # not loaded by store-gateway. Useful when used together with # -querier.query-store-after to prevent loading young blocks, because there # are usually many of them (depending on number of ingesters) and they are # not yet compacted. Negative values or 0 disable the filter. # CLI flag: -blocks-storage.bucket-store.ignore-blocks-within [ignore_blocks_within: | default = 0s] - # Max size - in bytes - of a chunks pool, used to reduce memory allocations. - # The pool is shared across all tenants. 0 to disable the limit. + # (advanced) Max size - in bytes - of a chunks pool, used to reduce memory + # allocations. The pool is shared across all tenants. 0 to disable the + # limit. # CLI flag: -blocks-storage.bucket-store.max-chunk-pool-bytes [max_chunk_pool_bytes: | default = 2147483648] - # Max size - in bytes - of the in-memory series hash cache. The cache is - # shared across all tenants and it's used only when query sharding is - # enabled. + # (advanced) Size - in bytes - of the smallest chunks pool bucket. + # CLI flag: -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes + [chunk_pool_min_bucket_size_bytes: | default = 16000] + + # (advanced) Size - in bytes - of the largest chunks pool bucket. + # CLI flag: -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes + [chunk_pool_max_bucket_size_bytes: | default = 50000000] + + # (advanced) Max size - in bytes - of the in-memory series hash cache. The + # cache is shared across all tenants and it's used only when query sharding + # is enabled. # CLI flag: -blocks-storage.bucket-store.series-hash-cache-max-size-bytes [series_hash_cache_max_size_bytes: | default = 1073741824] - # If enabled, store-gateway will lazy load an index-header only once - # required by a query. + # (advanced) If enabled, store-gateway will lazy load an index-header only + # once required by a query. # CLI flag: -blocks-storage.bucket-store.index-header-lazy-loading-enabled [index_header_lazy_loading_enabled: | default = true] - # If index-header lazy loading is enabled and this setting is > 0, the - # store-gateway will offload unused index-headers after 'idle timeout' - # inactivity. + # (advanced) If index-header lazy loading is enabled and this setting is > + # 0, the store-gateway will offload unused index-headers after 'idle + # timeout' inactivity. # CLI flag: -blocks-storage.bucket-store.index-header-lazy-loading-idle-timeout [index_header_lazy_loading_idle_timeout: | default = 1h] + # (advanced) Max size - in bytes - of a gap for which the partitioner + # aggregates together two bucket GET object requests. + # CLI flag: -blocks-storage.bucket-store.partitioner-max-gap-bytes + [partitioner_max_gap_bytes: | default = 524288] + + # (advanced) Controls what is the ratio of postings offsets that the store + # will hold in memory. + # CLI flag: -blocks-storage.bucket-store.posting-offsets-in-mem-sampling + [postings_offsets_in_mem_sampling: | default = 32] + tsdb: - # Local directory to store TSDBs in the ingesters. + # Directory to store TSDBs (including WAL) in the ingesters. This directory + # is required to be persisted between restarts. # CLI flag: -blocks-storage.tsdb.dir - [dir: | default = "tsdb"] + [dir: | default = "./tsdb/"] - # TSDB blocks range period. + # (advanced) TSDB blocks range period. # CLI flag: -blocks-storage.tsdb.block-ranges-period [block_ranges_period: | default = 2h0m0s] # TSDB blocks retention in the ingester before a block is removed. This - # should be larger than the block_ranges_period and large enough to give - # store-gateways and queriers enough time to discover newly uploaded blocks. + # should be larger than the -blocks-storage.tsdb.block-ranges-period, + # -querier.query-store-after and large enough to give store-gateways and + # queriers enough time to discover newly uploaded blocks. # CLI flag: -blocks-storage.tsdb.retention-period - [retention_period: | default = 6h] + [retention_period: | default = 24h] - # How frequently the TSDB blocks are scanned and new ones are shipped to the - # storage. 0 means shipping is disabled. + # (advanced) How frequently the TSDB blocks are scanned and new ones are + # shipped to the storage. 0 means shipping is disabled. # CLI flag: -blocks-storage.tsdb.ship-interval [ship_interval: | default = 1m] - # Maximum number of tenants concurrently shipping blocks to the storage. + # (advanced) Maximum number of tenants concurrently shipping blocks to the + # storage. # CLI flag: -blocks-storage.tsdb.ship-concurrency [ship_concurrency: | default = 10] - # How frequently ingesters try to compact TSDB head. Block is only created - # if data covers smallest block range. Must be greater than 0 and max 5 - # minutes. + # (advanced) How frequently ingesters try to compact TSDB head. Block is + # only created if data covers smallest block range. Must be greater than 0 + # and max 5 minutes. # CLI flag: -blocks-storage.tsdb.head-compaction-interval [head_compaction_interval: | default = 1m] - # Maximum number of tenants concurrently compacting TSDB head into a new - # block + # (advanced) Maximum number of tenants concurrently compacting TSDB head + # into a new block # CLI flag: -blocks-storage.tsdb.head-compaction-concurrency [head_compaction_concurrency: | default = 5] - # If TSDB head is idle for this duration, it is compacted. Note that up to - # 25% jitter is added to the value to avoid ingesters compacting + # (advanced) If TSDB head is idle for this duration, it is compacted. Note + # that up to 25% jitter is added to the value to avoid ingesters compacting # concurrently. 0 means disabled. # CLI flag: -blocks-storage.tsdb.head-compaction-idle-timeout [head_compaction_idle_timeout: | default = 1h] - # The write buffer size used by the head chunks mapper. Lower values reduce - # memory utilisation on clusters with a large number of tenants at the cost - # of increased disk I/O operations. + # (advanced) The write buffer size used by the head chunks mapper. Lower + # values reduce memory utilisation on clusters with a large number of + # tenants at the cost of increased disk I/O operations. # CLI flag: -blocks-storage.tsdb.head-chunks-write-buffer-size-bytes [head_chunks_write_buffer_size_bytes: | default = 4194304] - # The number of shards of series to use in TSDB (must be a power of 2). - # Reducing this will decrease memory footprint, but can negatively impact - # performance. + # (experimental) How much variance (as percentage between 0 and 1) should be + # applied to the chunk end time, to spread chunks writing across time. + # Doesn't apply to the last chunk of the chunk range. 0 means no variance. + # CLI flag: -blocks-storage.tsdb.head-chunks-end-time-variance + [head_chunks_end_time_variance: | default = 0] + + # (advanced) The number of shards of series to use in TSDB (must be a power + # of 2). Reducing this will decrease memory footprint, but can negatively + # impact performance. # CLI flag: -blocks-storage.tsdb.stripe-size [stripe_size: | default = 16384] - # True to enable TSDB WAL compression. + # (advanced) True to enable TSDB WAL compression. # CLI flag: -blocks-storage.tsdb.wal-compression-enabled [wal_compression_enabled: | default = false] - # TSDB WAL segments files max size (bytes). + # (advanced) TSDB WAL segments files max size (bytes). # CLI flag: -blocks-storage.tsdb.wal-segment-size-bytes [wal_segment_size_bytes: | default = 134217728] - # True to flush blocks to storage on shutdown. If false, incomplete blocks - # will be reused after restart. + # (advanced) True to flush blocks to storage on shutdown. If false, + # incomplete blocks will be reused after restart. # CLI flag: -blocks-storage.tsdb.flush-blocks-on-shutdown [flush_blocks_on_shutdown: | default = false] - # If TSDB has not received any data for this duration, and all blocks from - # TSDB have been shipped, TSDB is closed and deleted from local disk. If set - # to positive value, this value should be equal or higher than + # (advanced) If TSDB has not received any data for this duration, and all + # blocks from TSDB have been shipped, TSDB is closed and deleted from local + # disk. If set to positive value, this value should be equal or higher than # -querier.query-ingesters-within flag to make sure that TSDB is not closed # prematurely, which could cause partial query results. 0 or negative value # disables closing of idle TSDB. # CLI flag: -blocks-storage.tsdb.close-idle-tsdb-timeout - [close_idle_tsdb_timeout: | default = 0s] + [close_idle_tsdb_timeout: | default = 13h] - # True to enable snapshotting of in-memory TSDB data on disk when shutting - # down. + # (experimental) True to enable snapshotting of in-memory TSDB data on disk + # when shutting down. # CLI flag: -blocks-storage.tsdb.memory-snapshot-on-shutdown [memory_snapshot_on_shutdown: | default = false] - # The size of the write queue used by the head chunks mapper. Lower values - # reduce memory utilisation at the cost of potentially higher ingest - # latency. 0 disables the use of the chunk write queue. + # (experimental) The size of the write queue used by the head chunks mapper. + # Lower values reduce memory utilisation at the cost of potentially higher + # ingest latency. Value of 0 switches chunks mapper to implementation + # without a queue. # CLI flag: -blocks-storage.tsdb.head-chunks-write-queue-size [head_chunks_write_queue_size: | default = 0] - # Enables TSDB isolation feature. Disabling may improve performance. + # (advanced) Enables TSDB isolation feature. Disabling may improve + # performance. # CLI flag: -blocks-storage.tsdb.isolation-enabled [isolation_enabled: | default = true] - # Max size - in bytes - of the in-memory series hash cache. The cache is - # shared across all tenants and it's used only when query sharding is - # enabled. + # (advanced) Max size - in bytes - of the in-memory series hash cache. The + # cache is shared across all tenants and it's used only when query sharding + # is enabled. # CLI flag: -blocks-storage.tsdb.series-hash-cache-max-size-bytes [series_hash_cache_max_size_bytes: | default = 1073741824] - # limit the number of concurrently opening TSDB's on startup + # (advanced) limit the number of concurrently opening TSDB's on startup # CLI flag: -blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup [max_tsdb_opening_concurrency_on_startup: | default = 10] ``` diff --git a/docs/sources/blocks-storage/store-gateway.template b/docs/sources/architecture/store-gateway.template similarity index 87% rename from docs/sources/blocks-storage/store-gateway.template rename to docs/sources/architecture/store-gateway.template index 68b787f1ab4..2205a1ab1e9 100644 --- a/docs/sources/blocks-storage/store-gateway.template +++ b/docs/sources/architecture/store-gateway.template @@ -1,12 +1,17 @@ --- title: "Store-gateway" -linkTitle: "Store-gateway" -weight: 3 -slug: store-gateway +description: "Overview of the store-gateway microservice." +weight: 10 --- {{ .GeneratedFileWarning }} + + +The **[store-gateway](./store-gateway.md)** is responsible to query blocks and is used by the [querier](./querier.md) at query time. The store-gateway is required when running the blocks storage. + + + The store-gateway is the Cortex service responsible to query series from blocks. The store-gateway is required when running the Cortex blocks storage. The store-gateway is **semi-stateful**. @@ -44,22 +49,17 @@ Store-gateways continuously monitor the ring state and whenever the ring topolog For each block belonging to a store-gateway shard, the store-gateway loads its `meta.json`, the `deletion-mark.json` and the index-header. Once a block is loaded on the store-gateway, it's ready to be queried by queriers. When the querier queries blocks through a store-gateway, the response will contain the list of actually queried block IDs. If a querier tries to query a block which has not been loaded by a store-gateway, the querier will either retry on a different store-gateway (if blocks replication is enabled) or fail the query. -Blocks can be replicated across multiple store-gateway instances based on a replication factor configured via `-store-gateway.sharding-ring.replication-factor`. The blocks replication is used to protect from query failures caused by some blocks not loaded by any store-gateway instance at a given time like, for example, in the event of a store-gateway failure or while restarting a store-gateway instance (e.g. during a rolling update). +Blocks are replicated across multiple store-gateway instances based on a replication factor configured via `-store-gateway.sharding-ring.replication-factor`. The blocks replication is used to protect from query failures caused by some blocks not loaded by any store-gateway instance at a given time like, for example, in the event of a store-gateway failure or while restarting a store-gateway instance (e.g. during a rolling update). -This feature can be enabled via `-store-gateway.sharding-enabled=true` and requires the backend [hash ring](../architecture.md#the-hash-ring) to be configured via `-store-gateway.sharding-ring.*` flags (or their respective YAML config options). +This feature requires the backend [hash ring](../architecture.md#the-hash-ring) to be configured via `-store-gateway.sharding-ring.*` flags (or their respective YAML config options). ### Sharding strategies -The store-gateway supports two sharding strategies: - -- `default` -- `shuffle-sharding` - -The **`default`** sharding strategy spreads the blocks of each tenant across all store-gateway instances. It's the easiest form of sharding supported, but doesn't provide any workload isolation between different tenants. +The store-gateway uses shuffle-sharding to spread the blocks of each tenant across a subset of store-gateway instances. The number of store-gateway instances loading blocks of a single tenant is limited and the blast radius of any issue that could be introduced by the tenant's workload is limited to its shard instances. -The **`shuffle-sharding`** strategy spreads the blocks of a tenant across a subset of store-gateway instances. This way, the number of store-gateway instances loading blocks of a single tenant is limited and the blast radius of any issue that could be introduced by the tenant's workload is limited to its shard instances. +The default number of store-gateway instances per tenant is configured using the `-store-gateway.tenant-shard-size` flag (or their respective YAML config options). The shard size can then be overridden on a per-tenant basis setting the `store_gateway_tenant_shard_size` in the limits overrides. -The shuffle sharding strategy can be enabled via `-store-gateway.sharding-strategy=shuffle-sharding` and requires the `-store-gateway.tenant-shard-size` flag (or their respective YAML config options) to be set to the default shard size, which is the default number of store-gateway instances each tenant should be sharded to. The shard size can then be overridden on a per-tenant basis setting the `store_gateway_tenant_shard_size` in the limits overrides. +Default value for `-store-gateway.tenant-shard-size` is 0, which means that tenant's blocks are sharded across all store-gateway instances. _Please check out the [shuffle sharding documentation](../guides/shuffle-sharding.md) for more information about how it works._ diff --git a/docs/sources/blocks-storage/_index.md b/docs/sources/blocks-storage/_index.md index a183d223193..724ecf4414c 100644 --- a/docs/sources/blocks-storage/_index.md +++ b/docs/sources/blocks-storage/_index.md @@ -12,52 +12,7 @@ The supported backends for the blocks storage are: - [Amazon S3](https://aws.amazon.com/s3) - [Google Cloud Storage](https://cloud.google.com/storage/) - [Microsoft Azure Storage](https://azure.microsoft.com/en-us/services/storage/) -- [OpenStack Swift](https://wiki.openstack.org/wiki/Swift) (experimental) +- [OpenStack Swift](https://wiki.openstack.org/wiki/Swift) - [Local Filesystem](https://thanos.io/storage.md/#filesystem) (single node only) _Internally, some components are based on [Thanos](https://thanos.io), but no Thanos knowledge is required in order to run it._ - -## Architecture - -When running the Mimir blocks storage, the Mimir architecture doesn't significantly change and thus the [general architecture documentation](../architecture.md) applies to the blocks storage as well. However, there are two additional Mimir services when running the blocks storage: - -- [Store-gateway](./store-gateway.md) -- [Compactor](./compactor.md) - -![Architecture](/images/blocks-storage/architecture.png) - - - -The **[store-gateway](./store-gateway.md)** is responsible to query blocks and is used by the [querier](./querier.md) at query time. The store-gateway is required when running the blocks storage. - -The **[compactor](./compactor.md)** is responsible to merge and deduplicate smaller blocks into larger ones, in order to reduce the number of blocks stored in the long-term storage for a given tenant and query them more efficiently. It also keeps the [bucket index](./bucket-index.md) updated and, for this reason, it's a required component. - -The `alertmanager` and `ruler` components can also use object storage to store its configurations and rules uploaded by users. In that case a separate bucket should be created to store alertmanager configurations and rules: using the same bucket between ruler/alertmanager and blocks will cause issue with the **[compactor](./compactor.md)**. - -### The write path - -**Ingesters** receive incoming samples from the distributors. Each push request belongs to a tenant, and the ingester appends the received samples to the specific per-tenant TSDB stored on the local disk. The received samples are both kept in-memory and written to a write-ahead log (WAL) and used to recover the in-memory series in case the ingester abruptly terminates. The per-tenant TSDB is lazily created in each ingester as soon as the first samples are received for that tenant. - -The in-memory samples are periodically flushed to disk - and the WAL truncated - when a new TSDB block is created, which by default occurs every 2 hours. Each newly created block is then uploaded to the long-term storage and kept in the ingester until the configured `-blocks-storage.tsdb.retention-period` expires, in order to give [queriers](./querier.md) and [store-gateways](./store-gateway.md) enough time to discover the new block on the storage and download its index-header. - -In order to effectively use the **WAL** and being able to recover the in-memory series upon ingester abruptly termination, the WAL needs to be stored to a persistent disk which can survive in the event of an ingester failure (ie. AWS EBS volume or GCP persistent disk when running in the cloud). For example, if you're running the Mimir cluster in Kubernetes, you may use a StatefulSet with a persistent volume claim for the ingesters. The location on the filesystem where the WAL is stored is the same where local TSDB blocks (compacted from head) are stored and cannot be decoupled. See also the [timeline of block uploads](production-tips/#how-to-estimate--querierquery-store-after) and [disk space estimate](production-tips/#ingester-disk-space). - -#### Distributor series sharding and replication - -Due to the replication factor N (typically 3), each time series is stored by N ingesters, and each ingester writes its own block to the long-term storage. [Compactor](./compactor.md) merges blocks from multiple ingesters into a single block, and removes duplicate samples. After blocks compaction, the storage utilization is significantly reduced. - -For more information, see [Compactor](./compactor.md) and [Production tips](./production-tips.md). - -### The read path - -[Queriers](./querier.md) and [store-gateways](./store-gateway.md) periodically iterate over the storage bucket to discover blocks recently uploaded by ingesters. - -For each discovered block, queriers only download the block's `meta.json` file (containing some metadata including min and max timestamp of samples within the block), while store-gateways download the `meta.json` as well as the index-header, which is a small subset of the block's index used by the store-gateway to lookup series at query time. - -Queriers use the blocks metadata to compute the list of blocks that need to be queried at query time and fetch matching series from the store-gateway instances holding the required blocks. - -For more information, please refer to the following dedicated sections: - -- [Querier](./querier.md) -- [Store-gateway](./store-gateway.md) -- [Production tips](./production-tips.md) diff --git a/docs/sources/blocks-storage/bucket-index.md b/docs/sources/blocks-storage/bucket-index.md index d4aab94ec80..8a4b3cadc9e 100644 --- a/docs/sources/blocks-storage/bucket-index.md +++ b/docs/sources/blocks-storage/bucket-index.md @@ -7,7 +7,7 @@ slug: bucket-index The bucket index is a **per-tenant file containing the list of blocks and block deletion marks** in the storage. The bucket index itself is stored in the backend object storage, is periodically updated by the compactor, and used by queriers, store-gateways and rulers to discover blocks in the storage. -The bucket index usage is **optional** and can be enabled via `-blocks-storage.bucket-store.bucket-index.enabled=true` (or its respective YAML config option). +The bucket index is enabled by default, but it is **optional**. It can be disabled via `-blocks-storage.bucket-store.bucket-index.enabled=false` (or its respective YAML config option). Disabling the bucket index is not recommended. ## Benefits @@ -34,7 +34,7 @@ The `bucket-index.json.gz` contains: The [compactor](./compactor.md) periodically scans the bucket and uploads an updated bucket index to the storage. The frequency at which the bucket index is updated can be configured via `-compactor.cleanup-interval`. -Despite using the bucket index is optional, the index itself is built and updated by the compactor even if `-blocks-storage.bucket-store.bucket-index.enabled` has **not** been enabled. This is intentional, so that once a Cortex cluster operator decides to enable the bucket index in a live cluster, the bucket index for any tenant is already existing and query results consistency is guaranteed. The overhead introduced by keeping the bucket index updated is expected to be non significative. +The use of the bucket index is optional, but the index is built and updated by the compactor even if `-blocks-storage.bucket-store.bucket-index.enabled=false`. This behavior ensures the bucket index for any tenant already exists and that query result consistency is guaranteed if a Grafana Mimir cluster operator decides to enable the bucket index in a live cluster. The overhead introduced by keeping the bucket index updated is not signifcant. ## How it's used by the querier diff --git a/docs/sources/blocks-storage/migrate-storage-from-thanos-and-prometheus.md b/docs/sources/blocks-storage/migrate-storage-from-thanos-and-prometheus.md deleted file mode 100644 index 779a6f813a2..00000000000 --- a/docs/sources/blocks-storage/migrate-storage-from-thanos-and-prometheus.md +++ /dev/null @@ -1,132 +0,0 @@ ---- -title: "Migrate the storage from Thanos and Prometheus" -linkTitle: "Migrate the storage from Thanos and Prometheus" -weight: 7 -slug: migrate-storage-from-thanos-and-prometheus ---- - -The Cortex blocks storage engine stores series in TSDB blocks uploaded in the storage bucket. This makes very easy to migrate the storage from Thanos and/or Prometheus to Cortex, when running the blocks storage. - -## Cortex blocks storage requirements - -The Cortex blocks storage has few requirements that should be considered when migrating TSDB blocks from Thanos / Prometheus to Cortex: - -1. **The blocks in the bucket should be located at `bucket:///`**
- Cortex isolates blocks on a per-tenant basis in the bucket and, for this reason, each tenant blocks should be uploaded to a different location in the bucket. The bucket prefix, where a specific tenant blocks should be uploaded, is `//`; if Cortex is running with auth disabled (no multi-tenancy) then the `` to use is `fake`. -2. **Remove Thanos external labels and inject `__org_id__` into each block's `meta.json`**
- Every block has a little metadata file named `meta.json`. Thanos stores external labels at `thanos` > `labels`, which should be all removed when migrating to Cortex, while the `"__org_id__": ""` added. - -## How to migrate the storage - -### Upload TSDB blocks to Cortex bucket - -TSDB blocks stored in Prometheus local disk or Thanos bucket should be copied/uploaded to the Cortex bucket at the location `bucket:///` (when Cortex is running with auth disabled then `` must be `fake`). - -### Migrate block metadata (`meta.json`) to Cortex - -For each block copied/uploaded to the Cortex bucket, there are a few changes required to the `meta.json`. - -#### Automatically migrate metadata using the `thanosconvert` tool - -`thanosconvert` can iterate over a Cortex bucket and make sure that each `meta.json` has the correct `thanos` > `labels` layout. - -⚠ Warning ⚠ `thanosconvert` will modify files in the bucket you specify. It's recommended that you have backups or enable object versioning before running this tool. - -To run `thanosconvert`, you need to provide it with the bucket configuration in the same format as the [blocks storage bucket configuration](../configuration/config-file-reference.md#blocks_storage_config). - -```yaml -# bucket-config.yaml -backend: s3 -s3: - endpoint: s3.us-east-1.amazonaws.com - bucket_name: my-cortex-bucket -``` - -You can run thanosconvert directly using Go: - -```bash -go install github.com/cortexproject/cortex/cmd/thanosconvert -thanosconvert -``` - -Or use the provided docker image: - -```bash -docker run quay.io/cortexproject/thanosconvert -``` - -You can run the tool in dry-run mode first to find out what which blocks it will migrate: - -```bash -thanosconvert -config ./bucket-config.yaml -dry-run -``` - -Once you're happy with the results, you can run without dry run to migrate blocks: - -```bash -thanosconvert -config ./bucket-config.yaml -``` - -You can cancel a conversion in progress (with Ctrl+C) and rerun `thanosconvert`. It won't change any blocks which have been written by Cortex or already converted from Thanos, so you can run `thanosconvert` multiple times. - -#### Migrate metadata manually - -If you need to migrate the block metadata manually, you need to: - -1. Download the `meta.json` to the local filesystem -2. Decode the JSON -3. Manipulate the data structure (_see below_) -4. Re-encode the JSON -5. Re-upload it to the bucket (overwriting the previous version of the `meta.json` file) - -The `meta.json` should be manipulated in order to ensure: - -- It contains the `thanos` root-level entry -- The `thanos` > `labels` do not contain any Thanos-specific external label -- The `thanos` > `labels` contain the Cortex-specific external label `"__org_id__": ""` - -##### When migrating from Thanos - -When migrating from Thanos, the easiest approach would be keep the existing `thanos` root-level entry as is, except: - -1. Completely remove the content of `thanos` > `labels` -2. Add `"__org_id__": ""` to `thanos` > `labels` - -For example, when migrating a block from Thanos for the tenant `user-1`, the `thanos` root-level property within the `meta.json` file will look like: - -```json -{ - "thanos": { - "labels": { - "__org_id__": "user-1" - }, - "downsample": { - "resolution": 0 - }, - "source": "compactor" - } -} -``` - -##### When migrating from Prometheus - -When migrating from Prometheus, the `meta.json` file will not contain any `thanos` root-level entry and, for this reason, it would need to be generated: - -1. Create the `thanos` root-level entry (_see below_) -2. Add `"__org_id__": ""` to `thanos` > `labels` - -For example, when migrating a block from Prometheus for the tenant `user-1`, the `thanos` root-level property within the `meta.json` file should be as follow: - -```json -{ - "thanos": { - "labels": { - "__org_id__": "user-1" - }, - "downsample": { - "resolution": 0 - }, - "source": "compactor" - } -} -``` diff --git a/docs/sources/blocks-storage/production-tips.md b/docs/sources/blocks-storage/production-tips.md index 6f828a564d9..30b6dbfe7d9 100644 --- a/docs/sources/blocks-storage/production-tips.md +++ b/docs/sources/blocks-storage/production-tips.md @@ -51,7 +51,7 @@ Because of this, we would suggest to avoid querying non compacted blocks. In ord 3. Configure queriers `-querier.query-store-after` large enough to give compactor enough time to compact newly uploaded blocks (_see below_) 4. Configure queriers `-querier.query-ingesters-within` equal to `-querier.query-store-after` plus 5m (5 minutes is just a delta to query the boundary both from ingesters and queriers) 5. Configure ingesters `-blocks-storage.tsdb.retention-period` at least as `-querier.query-ingesters-within` -6. Lower `-blocks-storage.bucket-store.ignore-deletion-marks-delay` to 1h, otherwise non compacted blocks could be queried anyway, even if their compacted replacement is available +6. Ensure `-blocks-storage.bucket-store.ignore-deletion-marks-delay` is not too large (the default is suggested), otherwise non compacted blocks could be queried anyway, even if their compacted replacement is available #### How to estimate `-querier.query-store-after` diff --git a/docs/sources/configuration/_index.md b/docs/sources/configuration/_index.md index 4e75e965479..91dd38ef1d4 100644 --- a/docs/sources/configuration/_index.md +++ b/docs/sources/configuration/_index.md @@ -1,7 +1,5 @@ --- title: "Configuration" -linkTitle: "Configuration" -no_section_index_title: true -weight: 5 -menu: +description: "" +weight: 10000 --- diff --git a/docs/sources/configuration/arguments.md b/docs/sources/configuration/about-grafana-mimir-arguments.md similarity index 94% rename from docs/sources/configuration/arguments.md rename to docs/sources/configuration/about-grafana-mimir-arguments.md index 228026610c2..053e1486495 100644 --- a/docs/sources/configuration/arguments.md +++ b/docs/sources/configuration/about-grafana-mimir-arguments.md @@ -1,10 +1,11 @@ --- -title: "Mimir arguments" -linkTitle: "Mimir arguments explained" -weight: 2 -slug: arguments +title: "About Grafana Mimir arguments" +description: "" +weight: 10 --- +# About Grafana Mimir arguments + ## General notes Mimir has evolved over several years, and the command-line options sometimes reflect this heritage. In some cases the default value for options is not the recommended value, and in some cases names do not reflect the true meaning. We do intend to clean this up, but it requires a lot of care to avoid breaking existing installations. In the meantime we regret the inconvenience. @@ -16,7 +17,7 @@ Duration arguments should be specified with a unit like `5s` or `3h`. Valid time - `-querier.max-concurrent` The maximum number of top-level PromQL queries that will execute at the same time, per querier process. - If using the query frontend, this should be set to at least (`-querier.worker-parallelism` \* number of query frontend replicas). Otherwise queries may queue in the queriers and not the frontend, which will affect QoS. Alternatively, consider using `-querier.worker-match-max-concurrent` to force worker parallelism to match `-querier.max-concurrent`. + If using the query-frontend or query-scheduler, this also set the number of query execution workers (running inside the a querier process) which will connect respectively to the query-frontend or query-scheduler. - `-querier.timeout` @@ -40,16 +41,6 @@ The next three options only apply when the querier is used together with the Que How often the workers will query DNS to re-check where the query frontend or query scheduler is. -- `-querier.worker-parallelism` - - Number of simultaneous queries to process, per query frontend or scheduler. - See note on `-querier.max-concurrent` - -- `-querier.worker-match-max-concurrent` - - Force worker concurrency to match the -querier.max-concurrent option. Overrides `-querier.worker-parallelism`. - See note on `-querier.max-concurrent` - ## Querier and Ruler The ingester query API was improved over time, but defaults to the old behaviour for backwards-compatibility. For best results both of these next two flags should be set to `true`: @@ -313,8 +304,6 @@ Example runtime configuration file: overrides: tenant1: ingestion_rate: 10000 - tenant2: - max_samples_per_query: 1000000 multi_kv_config: mirror_enabled: false @@ -339,8 +328,6 @@ The `overrides` field is a map of tenant ID (same values as passed in the `X-Sco overrides: tenant1: ingestion_rate: 10000 - tenant2: - max_samples_per_query: 1000000 ``` Valid per-tenant limits are (with their corresponding flags for default values): @@ -365,18 +352,14 @@ Valid per-tenant limits are (with their corresponding flags for default values): Enforced by the ingesters; limits the number of in-memory series a user (or a given metric) can have. A series is kept in memory if a sample has been written since the last TSDB head compaction (occurring every 2h) or in the last 1h (regardless when the last TSDB head compaction occurred). The limit is enforced across the cluster. Each ingester is configured with a local limit based on the replication factor and the current number of healthy ingesters. The local limit is updated whenever the number of ingesters change. - Requires `-distributor.replication-factor`, `-distributor.sharding-strategy` and `-distributor.zone-awareness-enabled` set for the ingesters too. - -- `max_samples_per_query` / `-ingester.max-samples-per-query` - - Limits on the number of timeseries and samples returns by a single ingester during a query. + Requires `-distributor.replication-factor` and `-distributor.zone-awareness-enabled` set for the ingesters too. - `max_global_metadata_per_user` / `-ingester.max-global-metadata-per-user` - `max_global_metadata_per_metric` / `-ingester.max-global-metadata-per-metric` Enforced by the ingesters; limits the number of active metadata a user (or a given metric) can have. The limit is enforced across the cluster. Each ingester is configured with a local limit based on the replication factor and the current number of healthy ingesters. The local limit is updated whenever the number of ingesters change. - Requires `-distributor.replication-factor`, `-distributor.sharding-strategy` and `-distributor.zone-awareness-enabled` set for the ingesters too. + Requires `-distributor.replication-factor` and `-distributor.zone-awareness-enabled` set for the ingesters too. - `max_fetched_series_per_query` / `querier.max-fetched-series-per-query` This limit is enforced in the queriers on unique series fetched from ingesters and store-gateways (long-term storage). diff --git a/docs/sources/configuration/about-versioning.md b/docs/sources/configuration/about-versioning.md new file mode 100644 index 00000000000..5f6bdea49d0 --- /dev/null +++ b/docs/sources/configuration/about-versioning.md @@ -0,0 +1,63 @@ +--- +title: "About versioning" +description: "" +weight: 10 +--- + +# About versioning + +For the v1.0 release, we want to provide the following guarantees: + +## Flags, Config and minor version upgrades + +Upgrading Grafana Mimir from one minor version to the next should "just work"; that being said, we don't want to bump the major version every time we remove a configuration parameter, so we will keep deprecated flags and YAML config parameters around for 2 minor releases. There is a metric (`deprecated_flags_inuse_total`) you can alert on to find out if you're using a deprecated flag. + +These guarantees don't apply for [experimental features](#experimental-features). + +## Reading old data + +The Grafana Mimir maintainers commit to ensuring future versions can read data written by versions up to two years old. In practice we expect to be able to read more, but this is our guarantee. + +## API Compatibility + +Grafana Mimir strives to be 100% API compatible with Prometheus (under `/prometheus/*`); any deviation from this is considered a bug, except: + +- Additional API endpoints for creating, removing and modifying alerts and recording rules. +- Additional API around pushing metrics (under `/prometheus/api/push`). +- Additional API endpoints for management of Grafana Mimir itself, such as the ring. These APIs are not part of the any compatibility guarantees. + +_For more information, please refer to the [limitations](../guides/limitations.md) doc._ + +## Experimental features + +Grafana Mimir is an actively developed project and we want to encourage the introduction of new features and capabilities. As such, not everything in each release of Grafana Mimir is considered "production-ready". Features not considered "production-ready" and the flags used to enable and/or configure these features will be marked "Experimental". There are no backwards compatibility guarantees on anything marked experimental. Configuration and flags are subject to change. + +Currently experimental features are: + +- Ruler: tenant federation. +- Distributor: metrics relabeling. +- Purger: tenant deletion API. +- Exemplar storage + - `-ingester.max-global-exemplars-per-user` + - `-ingester.exemplars-update-period` + - API endpoint `/api/v1/query_exemplars` +- Hash ring + - Disabling ring heartbeat timeouts + - `-distributor.ring.heartbeat-timeout=0` + - `-ring.heartbeat-timeout=0` + - `-ruler.ring.heartbeat-timeout=0` + - `-alertmanager.sharding-ring.heartbeat-timeout=0` + - `-compactor.ring.heartbeat-timeout=0` + - `-store-gateway.sharding-ring.heartbeat-timeout=0` + - Disabling ring heartbeats + - `-distributor.ring.heartbeat-period=0` + - `-ingester.heartbeat-period=0` + - `-ruler.ring.heartbeat-period=0` + - `-alertmanager.sharding-ring.heartbeat-period=0` + - `-compactor.ring.heartbeat-period=0` + - `-store-gateway.sharding-ring.heartbeat-period=0` + - Exclude ingesters running in specific zones (`-distributor.excluded-zones`) +- Ingester + - Add variance to chunks end time to spread writing across time (`-blocks-storage.tsdb.head-chunks-end-time-variance`) + - Using queue and asynchronous chunks disk mapper (`-blocks-storage.tsdb.head-chunks-write-queue-size`) + - Snapshotting of in-memory TSDB data on disk when shutting down (`-blocks-storage.tsdb.memory-snapshot-on-shutdown`) diff --git a/docs/sources/configuration/config-file-reference.template b/docs/sources/configuration/config-file-reference.template deleted file mode 100644 index 9a4352d13e4..00000000000 --- a/docs/sources/configuration/config-file-reference.template +++ /dev/null @@ -1,55 +0,0 @@ ---- -title: "Configuration file" -linkTitle: "Configuration file" -weight: 1 -slug: configuration-file ---- - -{{ .GeneratedFileWarning }} - -Cortex can be configured using a YAML file - specified using the `-config.file` flag - or CLI flags. In case you combine both, CLI flags take precedence over the YAML config file. - -The current configuration of any Cortex component can be seen by visiting the `/config` HTTP path. -Passwords are filtered out of this endpoint. - -## Reference - -To specify which configuration file to load, pass the `-config.file` flag at the command line. The file is written in [YAML format](https://en.wikipedia.org/wiki/YAML), defined by the scheme below. Brackets indicate that a parameter is optional. - -### Generic placeholders - -- ``: a boolean that can take the values `true` or `false` -- ``: any integer matching the regular expression `[1-9]+[0-9]*` -- ``: a duration matching the regular expression `[0-9]+(ns|us|µs|ms|s|m|h|d|w|y)` where y = 365 days. -- ``: a regular string -- ``: an URL -- ``: a CLI flag prefix based on the context (look at the parent configuration block to see which CLI flags prefix should be used) -- ``: a [Prometheus relabeling configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config). -- `