[RFE] pmproxy live mode for pmseries queries

[natoscott]

We've had multiple requests for this feature now, so I'm going to brain dump and gather any other ideas people may have here.

Currently, pmproxy will only index PCP metrics into Redis that have been archived locally, and then retrieved using the discovery functionality in pmproxy. It would be beneficial for some use cases (e.g. edge devices and other minimal install setups) to have the option of pmproxy directly sampling metrics from pmcd itself and then sending that to Redis in the same form as the archive based mechanisms we have today.

One subtlety here is configuration - i.e. choosing which metrics to index and sampling intervals - this state is inherent in the pmlogger approach as pmlogconf/manual pmlogger configuration decides these things. With a live mode, pmproxy would need to be configured in terms of which metrics and intervals - something like the pmrepconf extension to pmlogconf might be the way to go.

Another aspect to consider is whether to allow this for use with the localhost pmcd only, or to allow sampling from multiple remote pmcd servers through a single pmproxy. I'm inclined to go with the localhost only model for simplicity (perhaps with a dedicated libuv thread - communication with pmcd uses the non-async PMAPI), and insist on a pmproxy server on each host that wants to play in this sandpit. This has the benefit of also ensuring every edge server has the PCP REST API available, and avoids remote-access permissions setup requirements.

[christianhorn]

Seems like it would be doing pretty much what pmlogger is doing, just with "target redis" instead of archive files.. interesting implementation in pmproxy would be easier than in pmlogger?

I wonder if users would then start to wonder more heavily about redis dumps/recovering these, while currently the PCP strategy is AFAIK to consider redis contents ephemeral, and if required fill redis again with archive files.

[natoscott]

@christianhorn that is a good point. In the past there's been discussions about making pmlogger able to write different formats, so this would indeed be an alternative.

From an implementation POV, its far easier in pmproxy since it already has all the logic there for doing the archive -> Redis communication via libpcp_web, whereas pmlogger does not use this library at all (pmlogger is designed to be non-async, whereas ilbpcp_web and pmproxy are designed to communicate async with redis via hiredis and hiredis-cluster APIs).