feat(otel): add ability to setup and create metrics, initially for Prometheus #45
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
This is part of the work for adding custom metrics into our applications, and I'm not super sure on the best approach especially when it comes to gokit, so this draft PR is here to serve as a testing ground on how we want this all to really work.
I've been battling on the best way to really deal with this, as
SetupGlobalState
is nautrally already hard to test properly, however making it as easy as possible to begin adding metrics - primarily around ease of use for setting up the Prometheus endpoint - I decided to stuff a bit more logic into the function... as bloated as it currently is.I decided that since otelkit already sets up most of what we need for OTEL, it'd be the best place to stuff the metrics stuff into it - though that now makes the
/tracing/
part of its path slightly misleading 😓I was also struggling to determine between either:
In the end I felt that setting up a Prometheus endpoint would give us a bit more flexibilty, as we can manage aspects such as scrape interval; filtering out boring metrics we don't want to pay for, etc. within Managed Prometheus rather than the application's code/config.
Prometheus also makes it easier for us to inspect metrics locally, as we don't need to make local dev apps send things into Cloud Monitoring (conflicting with other people's local deployments and any live dev deployments).
Since we use GRPC for the most part I decided to add
WithPrometheusMetricExporterAutoServer
that makes it simple to add in the Prometheus endpoint, but this resulted in some additional complexity within otelkit's side. I'm not really sold on this idea, but wanted to pitch it.(Minor point right now but Managed Prometheus is cheaper than direct API calls as well).
This PR
This PR:
SetupGlobalState
so that it will also configured aMeterProvider
(defaulting to a noop one) alongside all the other bits of OTEL state we setup there.SetupGlobalState
so that it'll run a deferred list of functions at the end of its logic. The main use case right now is so thatWithPrometheusMetricExporterAutoServer
can setup it's simple HTTP server.WithPrometheusMetricExporter
andWithPrometheusMetricExporterAutoServer
which will configure OTEL to use Prometheus for its metrics backend.## What I'm looking for
I'm mainly looking for general feedback on whether this is the kind of approach we'd like to go down and explore, and if so I can pick a service to do a test implementation of to make sure we're all happy with this approach. The setup of otelkit is kind of secondary I guess.
As with tracing this is mainly just a light wrapper around setting up OTEL in general. When it comes to producing metrics it'll generally look like normal OTEL SDK stuff (except we use our own
.Meter
function).I should note that the GCP SDKs seem to automatically use OTEL for generating metrics, so it's not just our own custom ones we have a chance to export here.