From 0fb5afc57923fcab3e6763cedfd438d4bdbaaf81 Mon Sep 17 00:00:00 2001 From: Alex R Date: Sun, 19 Nov 2023 17:07:01 +0100 Subject: [PATCH] add mimir support --- README.md | 83 ++++++++++++++++++++++++++++--------------------------- 1 file changed, 43 insertions(+), 40 deletions(-) diff --git a/README.md b/README.md index 7ee2206..f3b3de2 100644 --- a/README.md +++ b/README.md @@ -1,80 +1,73 @@ # thanos-kit -Aftermarket thanos tools and utilities. +Tooling to work with Thanos blocks in object storage. -- `inspect` - Inspect all blocks in the bucket in detailed, table-like way (same as `thanos tools bucket inspect`) -- `analyze` - Analyze churn, label pair cardinality for specific block -- `dump` - Dump samples from a TSDB to text format -- `import` - Import samples to TSDB blocks. Read about [Backfill](#backfill) below +- **ls** - List all blocks ULIDs in the bucket, also show ULID as time (same as `thanos tools bucket ls` but with mimir support) +- **inspect** - Inspect all blocks in the bucket in detailed, table-like way (same as `thanos tools bucket inspect` but with mimir support) +- **analyze** - Analyze churn, label pair cardinality for specific block. (same as `promtool tsdb analyze` but also show Labels suitable for block split) +- **dump** - Dump samples from a TSDB to text format (same as `promtool tsdb dump` but to promtext format) +- **import** - Import samples to TSDB blocks (same as `promtool tsdb create-blocks-from openmetrics` but from promtext format). Read more about [backfill](#backfill) below -Cli arguments mostly are the same as for `thanos`, help is available for each sub-command: +Cli arguments are mostly the same as for `thanos`, help is available for each sub-command: ``` -$ ./thanos-kit -h +$ docker run sepa/thanos-kit -h usage: thanos-kit [] [ ...] -Tooling to work with Thanos blobs in object storage +Tooling to work with Thanos blocks in object storage Flags: -h, --help Show context-sensitive help (also try --help-long and --help-man). --version Show application version. --log.level=info Log filtering level (info, debug) + --objstore.config-file= + Path to YAML file that contains object store%s configuration. See format details: https://thanos.io/tip/thanos/storage.md/ + --objstore.config= + Alternative to 'objstore.config-file' flag (mutually exclusive). Content of YAML file that contains object store%s configuration. See format details: https://thanos.io/tip/thanos/storage.md/ Commands: help [...] Show help. + ls [] + List all blocks in the bucket. + inspect [] Inspect all blocks in the bucket in detailed, table-like way analyze [] - Analyze churn, label pair cardinality + Analyze churn, label pair cardinality and find labels to split on dump [] ... - Dump samples from a TSDB + Dump samples from a TSDB to text - import --label=="" [] - Import samples to TSDB blocks + import --input-file=INPUT-FILE --label=="" [] + Import samples from text to TSDB blocks ``` ### Get it -Master builds are available on [Docker Hub](https://hub.docker.com/repository/docker/sepa/thanos-kit/tags) +Docker images are available on [Docker Hub](https://hub.docker.com/repository/docker/sepa/thanos-kit/tags) ### Backfill - ([Original PR](https://github.com/prometheus/prometheus/pull/7586)) -You can use it to restore another prometheus partial dump, or any metrics exported from any system. The only supported input format is Prometheus text format. +Supported input format is Prometheus text format. You are free to export/convert your existing data to this format, into one **time-sorted** text file. -Sample file `rrd_exported_data.txt` (`[metric]{[labels]} [number value] [timestamp ms]`): +`metric{[labels]} value timestamp_ms` + +For example: ```ini -collectd_df_complex{host="myserver.fqdn.com",df="var-log",dimension="free"} 5.8093906125e+10 1599771600000 -collectd_varnish_derive{host="myserver.fqdn.com",varnish="request_rate",dimension="MAIN.client_req"} 2.3021666667e+01 1599771600000 -collectd_df_complex{host="myserver.fqdn.com",df="var-log",dimension="free"} 5.8093906125e+10 1599771615000 -collectd_load{host="myserver.fqdn.com",type="midterm"} 0.0155 1599771615000 -collectd_varnish_derive{host="myserver.fqdn.com",varnish="request_rate",dimension="MAIN.client_req"} 2.3021666667e+01 1599771630000 -collectd_load{host="myserver.fqdn.com",type="midterm"} 1.5500000000e-02 1599771630000 -collectd_varnish_derive{host="myserver.fqdn.com",varnish="request_rate",dimension="MAIN.cache_hit"} 3.8054166667e+01 1599771645000 -collectd_df_complex{host="myserver.fqdn.com",df="var-log",dimension="free"} 5.8093906125e+10 1599771645000 -collectd_varnish_derive{host="myserver.fqdn.com",varnish="request_rate",dimension="MAIN.s_pipe"} 0 1599771660000 -collectd_varnish_derive{host="myserver.fqdn.com",varnish="request_rate",dimension="MAIN.cache_hit"} 3.8054166667e+01 1599771675000 -collectd_load{host="myserver.fqdn.com",type="shortterm"} 1.1000000000e-02 1599771675000 -collectd_varnish_derive{host="myserver.fqdn.com",varnish="request_rate",dimension="MAIN.client_req"} 2.3021666667e+01 1599771690000 -collectd_load{host="myserver.fqdn.com",type="shortterm"} 1.1000000000e-02 1599771690000 -collectd_load{host="myserver.fqdn.com",type="shortterm"} 1.1000000000e-02 1599771705000 -collectd_load{host="myserver.fqdn.com",type="longterm"} 2.5500000000e-02 1599771720000 -collectd_varnish_derive{host="myserver.fqdn.com",varnish="request_rate",dimension="MAIN.s_pipe"} 0 1599771735000 -collectd_load{host="myserver.fqdn.com"type="longterm"} 2.5500000000e-02 1599771735000 -collectd_varnish_derive{host="myserver.fqdn.com",varnish="request_rate",dimension="MAIN.cache_hit"} 3.8054166667e+01 1599771750000 -collectd_load{host="myserver.fqdn.com",type="midterm"} 1.5500000000e-02 1599771750000 +k8s_ns_hourly_cost{namespace="kube-system"} 5.7 1599771600000 +k8s_ns_hourly_cost{namespace="kube-system"} 5.0 1599771630000 +... ``` -Note, `[number value]` can be mixed as normal or scientific number as per your preference. You are free to put custom labels on each metric, don't forget that "relabelling rules" defined in prometheus will not be applied on them! You should produce the final labels on your import file. +Note, `value` can be mixed as normal or scientific number as per your preference. This format is simple to produce, but not optimized or compressed, so it's normal if your data file is huge. -Example of a 19G OpenMetrics file, with ~20k timeseries and 200M data points (samples) on 2y period. Globally resolution is very very low in this example. -Import will take around 2h and uncompacted new TSDB blocks will be around 2.1G for 7600 blocks. When prometheus scan them, it starts automatically compacting them in the background. Once compaction is completed (~30min), TSDB blocks will be around 970M for 80 blocks (without loss of data points). +Example of a 19G OpenMetrics file, with ~20k timeseries and 200M data points (samples) on 2y period. Globally resolution is very low in this example. +Import will take around 2h and uncompacted new TSDB blocks will be around 2.1G for 7600 blocks. When thanos-compact scan them, it starts automatically compacting them in the background. Once compaction is completed (~30min), TSDB blocks will be around 970M for 80 blocks. The size, and number of blocks depends on timeseries numbers and metrics resolution, but it gives you an order of sizes. -Apart from labels set for each metric in text file, you would also need to set Thanos Metadata Labels for the whole batch of blocks you are importing (consider this as prometheus `external_labels` which scraped the metrics from text file) +Apart from labels set for each metric in text file, you would also need to set Thanos Metadata Labels for the whole batch of blocks you are importing (consider this as prometheus `external_labels` which scraped the metrics from the text file) Example of command for importing data from `data.prom` (above) to GCS bucket `bucketname`: ```bash @@ -87,4 +80,14 @@ docker run -it --rm \ --label=replica=\"prom-a\" \ --label=location=\"us-east1\" ``` -Current implementation reads all imported data into memory, and then flushes as TSDB 2h blocks. In case you have OOM but want to import large amount of data, you can split imported file by days (or 2h windows) and do multiple separate imports. Anyway these separate blocks would be compacted to larger one on bucket side via your compactor. Please note that compactor has default `--consistency-delay=30m` which is based on file upload time (not ULID), so it could take some time before compactor would start processing these blocks. +Please note that compactor has default `--consistency-delay=30m` which is based on file upload time (not ULID), so it could take some time before compactor would start processing these blocks. + +### Cache dir +By default, `thanos-kit` will cache blocks from object storage to `./data` directory, and the dir is not cleaned up on exit. This is to speed up subsequent runs, and to avoid deleting user data when `--data-dir=/tmp` is used for example. + +Important note that `dump` command downloads specified blocks to cache dir, but then dump TSDB as a whole (including blocks already present there) + +### Alternatives +- [thanos tools bucket](https://thanos.io/tip/components/tools.md/#bucket) +- [promtool tsdb](https://prometheus.io/docs/prometheus/latest/command-line/promtool/#promtool-tsdb) +- [mimirtool](https://grafana.com/docs/mimir/latest/manage/tools/mimirtool/#backfill) \ No newline at end of file