[APM] Throughput becomes inaccurate when instances become inactive

[sorenlouv]

Currently throughout is calculated as the total number of requests per min throughout the entire range. This means that an instance that has stopped will see a falling average throughput. This makes it harder to compare the throughput of instances.

Questions:

• For instances specifically, should throughput be calculated for the entire time range (like today) or only for the "active" period of an instance?
• In general, should throughput only be calculated for the "active" periods, thus excluding periods with no data?

The following example shows how throughput for the same instances change over time even though it should (intuitively) stay the same (at least for instance 1 and 2)

3 days range

7 days range

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[alex-fedotyev]

@felixbarny - I just realized that APM agents deliver metrics already, which might be useful for this.
It looks like every agent except PHP reports "system.cpu.total.norm.pct", could you confirm?

The problem is that we are looking for a reliable way to calculate lifetime of the service instance, in order to make sure we have proper throughout calculation for 1 instance of a service, and I guess same applied for a set of instances same way too.

[felixbarny]

Yes, according to their metrics documentation page, that's true for all agents except PHP.

However, metrics sending can be disabled, and older versions of an agent may not be sending these metrics.

To circumvent the first issue, we could send heartbeat metrics in the agents: elastic/apm#414

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sorenlouv]

@alex-fedotyev Is this something we are still considering or can we close this issue?