poc: a metrics module for pebble

[IronCore864]

A self-implemented metrics module for Pebble. I referred to the Prometheus Golang client a bit, not so much to runtime/metrics because it seems it's too complicated.

According to the spec, it seems we only need the counter type and the gauge type, but I added the histogram type anyway. Prometheus Golang client has more types but I don't think we will need it any time soon, so I ignored it.

Binary size doesn't increase (no extra libs used), still ~7.8MB with -trimpath -ldflags='-s -w'.

Manual test:

$ curl localhost:8000/metrics
# HELP my_counter A simple counter
# TYPE my_counter counter
my_counter 3

Some basic unit test is added to make sure the functionalities work as expected and there is no deadlock issue.

TODO: investigate how the official Prometheus Golang client gathers basic metrics like CPU usage and stuff.

[IronCore864]

Trying to integrate with Prometheus to make sure they are compatible:

[IronCore864]

Some investigation into the default metrics that come with the Prometheus Go client:

1 List of Metrics from Go runtime/metrics

c.a. 80 items.

• /cgo/go-to-c-calls:calls
• /cpu/*
• /gc/cycles/*
• /gc/gogc:percent
• /gc/gomemlimit:bytes
• /gc/heap/*
• /gc/limiter/last-enabled:gc-cycle
• /gc/pauses:seconds
• /gc/scan/*
• /gc/stack/starting-size:bytes
• /godebug/*
• /memory/*
• /sched/gomaxprocs:threads
• /sched/goroutines:goroutines
• /sched/latencies:seconds
• /sched/pauses/*
• /sync/mutex/wait/total:seconds

---

2 List of Default Metrics from prometheus/client_golang

c.a. 40 items.

• go_gc_duration_seconds{quantile="*"}
• go_gc_duration_seconds_sum
• go_gc_duration_seconds_count
• go_gc_gogc_percent
• go_gc_gomemlimit_bytes
• go_goroutines
• go_info{version="go1.23.1"}
• go_memstats_*
• go_sched_gomaxprocs_threads
• go_threads
• promhttp_metric_handler_requests_in_flight
• promhttp_metric_handler_requests_total{code="200"}
• promhttp_metric_handler_requests_total{code="500"}
• promhttp_metric_handler_requests_total{code="503"}

---

3 How Are the Default Metrics from prometheus/client_golang Fetched/Calculated

3.1 From runtime and runtime/debug

• go_goroutines
• go_threads
• go_gc_duration_seconds
• go_memstats_last_gc_time_seconds
• go_info

3.2 From runtime.MemStats with Some Calculations

• go_memstats_alloc_bytes
• go_memstats_alloc_bytes_total
• go_memstats_buck_hash_sys_bytes
• go_memstats_frees_total
• go_memstats_gc_sys_bytes
• go_memstats_heap_alloc_bytes
• go_memstats_heap_idle_bytes
• go_memstats_heap_inuse_bytes
• go_memstats_heap_objects
• go_memstats_heap_released_bytes
• go_memstats_heap_sys_bytes
• go_memstats_last_gc_time_seconds
• go_memstats_mallocs_total
• go_memstats_mcache_inuse_bytes
• go_memstats_mcache_sys_bytes
• go_memstats_mspan_inuse_bytes
• go_memstats_mspan_sys_bytes
• go_memstats_next_gc_bytes
• go_memstats_other_sys_bytes
• go_memstats_stack_inuse_bytes
• go_memstats_stack_sys_bytes
• go_memstats_sys_bytes

3.3 Directly from runtime/metrics but Go-Version-Dependant

• go_gc_gogc_percent
• go_gc_gomemlimit_bytes
• go_sched_gomaxprocs_threads

3.4 Prometheus-Related

• promhttp_metric_handler_requests_in_flight
• promhttp_metric_handler_requests_total{code="200"}
• promhttp_metric_handler_requests_total{code="500"}
• promhttp_metric_handler_requests_total{code="503"}

Self-processed.

4 Summary

If we want to include these metrics in our self-implemented metrics module, a lot of duplicated work (duplicated with Prometheus) needs to be done, some of which seem to be Go-version-dependent, which means extra work and tests. This creates a bunch of operational overhead.

If these metrics are not necessary, we probably should go with a self-implemented module but if they are nice to have, a 2.2MB size increase of the Pebble binary is worth the price in the long run.