Allow serving compresses responses via Prometheus plugin

[Lusitaniae]

Is your feature request related to a problem? Please describe.

I'm using Prometheus plugin to export metrics based on logs ingested by fluentd.

In some servers these metrics can amount to millions of lines, which reach payload sizes of 50/60Mb and either reach the scraping limit (set by vmagent), or they failed to be transitted within the timeout period (10s)

Compressing data would be a clear winner to solve this problem

Describe the solution you'd like

Allow prometheus plugin to send compressed responses

Describe alternatives you've considered

Bumping payload limits / timeouts

Additional context

$ curl -s http://node1:9106/metrics | wc -c
37420840
$ curl -s http://node1:9106/metrics | gzip -c | wc -c
1877477

Cross posted from fluent/fluent-plugin-prometheus#219 in case this is the right place to implement such feature

[Athishpranav2003]

@Lusitaniae I have one query with this. I understand you want to expose it in compressed format but can you name few metrics collectors that scrape in compressed format and decompresses it?
I have worked with Telegraf and chronocollector and both don't support scraping metrics that are in compressed format

I guess if this is not the standard then not sure if its a genuine feature request

[Lusitaniae]

Should work for prometheus

https://github.com/prometheus/prometheus/blob/main/scrape/scrape.go#L729

https://prometheus.io/docs/instrumenting/exposition_formats/

[Athishpranav2003]

I see
For now it seems only gzip format is supported
Maybe I can pick this up in the following week

[Lusitaniae]

Much appreciated 🙏

[Athishpranav2003]

@Lusitaniae This would require to contribute to https://github.com/prometheus/client_ruby/blob/main/lib/prometheus/client/formats/text.rb the official prometheus client for ruby. The hackaround for this is we break the abstractions in the Fluentd prometheus plugin to fetch the metrics info and compress and host directly. Not sure which is a better call because maybe first is overkill but I feel that's the write method

[Lusitaniae]

Doesn't this solve it?

It's highly recommended to enable gzip compression for the metrics endpoint, for example by including the Rack::Deflater middleware.

https://github.com/prometheus/client_ruby?tab=readme-ov-file#rack-middleware

[Athishpranav2003]

Oh yah
i missed this infact. Thanks for pointing it. I will try to check this in my free time.

[Athishpranav2003]

I was checking this @Lusitaniae. So for the Rack middleware to be used we need to have the app(http server) exposed and get overriden with the middleware. But i guess the code abstraction in fluentd has made use of a plugin helper so seems like its not as direct as we see. I am actively checking how to make use of it but incase if you also find something interesting please let me know

For now i am not using rack but trying to compress the data directly and put it in the response body

[Athishpranav2003]

@kenhys the PR is done for this and reviewed by @ashie. I guess u can change the label to the appropriate one
fluent/fluent-plugin-prometheus#223

[Lusitaniae]

Maybe can close this one, since it was implemented on the plugin itself, and I also filed an issue there

fluent/fluent-plugin-prometheus#219