Support for custom metrics

[emschwartz]

Whoa, that doesn't sound automatic, does it? Stay with me :)

Arguably, the main benefits of autometrics come not just from automatic instrumentation or from function-level metrics but from standardizing the production, naming, and labeling of metrics so that we can then write queries, build dashboards, and set up alerts based on them.

A number of people have asked about whether autometrics lets you add custom metrics. The current answer is that you can simply use one of the underlying metrics libraries to create a custom metric, which will then be exported alongside autometrics' metrics.

This is okay, but not ideal, because once you create a custom metric you're on your own as far as writing queries and adding them to the dashboard.

"Standardizing" custom metrics

While this might sound like a contradiction, there may be ways to standardize either metric names or labels such that we could still help you write certain queries and add useful charts to the dashboard.

For example, it seems like a common custom metric people might want to add is some kind of overall tracker of "the good thing" that their system does. That's probably not something you'd creates alerts based off of, but it is useful to get some overall sense of how much your system is being used. I sometimes refer to these as vanity metrics (and that's not meant as a knock against them. For my side projects, I am obsessive about checking their usage metrics).

If we standardize the metric name and/or some labels, we would then know that a certain metric is a counter tracking one of "the good things" for your system. We could probably insert links to some useful charts into the code doc comments and/or add these charts to the dashboard.

@hatchan made the good point that it might be better to standardize the labels for these rather than the metrics. There's a good chance that the various metrics would be counting different things, so it would be a more faithful use of Prometheus to create separate metrics for them but use some standard labels to make the time series identifiable.

Using log levels for metrics

One approach for this would be to call the "counter of the good thing" an info_counter. "Info" is trying to leverage the idea from log levels where you have a scheme that is understandable by both humans and computers. By telling us this is a good thing, rather than a bad thing, maybe we'd show you both the chart of how much this thing happens over time (but maybe on a longer interval than a per-second rate) and an overall total.

What would the equivalents of debug, warn, and error be? In some ways, debug is everything, because any metric you collect can help you debug things. warn and error probably need to be tied to some threshold because it's more like "warn me if this happens 'too much'", but you need a definition of 'too much'.

Maybe using those log levels would also let you enable/disable them at runtime using an environment variable like you can for logs?

Custom metrics aside from counters?

What are some other types of "custom" metrics you might want to include?

• A gauge tracking the number of connected users
• Maybe a histogram to track the latency of some specific block of code. (I'm skeptical about this one because why not just refactor that code into a separate function and use stock autometrics instrumentation?)
• ...?

If the libraries provided some functions for creating these other metrics and identifying (roughly) what they're for, we could use some labels to help us understand (for example in the dashboard), what types of charts would be most helpful to show for those time series.

[adamchalmers]

Personally I can't think of a reason why I'd need metrics outside of function annotations. Now, there might certainly be a good reason to support this. But at least for my projects, I can't think of one. So I suggest not prioritizing a feature like this until someone has a really good reason they need it.

If you need it, consider this an invitation to describe your use-case :)

[IvanMerrill]

My two cents on this:

• autometrics can provide a simple way of creating such a metric I guess, but I'd be very wary of doing much more than this. Log levels etc to me seems to introduce a level of complexity that just isn't needed.
• if people have a custom metric that's so important to them, then nothing is stopping them from adding it using the underlying metric libraries. They'll do this if they need it that badly.
• even with a lot of guard rails, people can still find a way to do silly things. That's in part why autometrics works so well, the chance of doing something bad with labels is taken away. Be wary of diluting this to satisfy other low-value features