[Elastic Agent] Read out apm data for diagnostics purpose

[ruflin]

As part of the apm instrumenation of Elastic Agent a discussion (elastic/beats#29031 (comment)) was started around how to use the apm data for diagnostic purpose. This issue is to continue this discussion.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[axw]

@ruflin:

One thing I'm curious in the context of diagnostics: Assuming someone is not shipping the apm data to ES, would it still be possible to fetch some of it as part of the diagnostics call? Does that even make sense?

@axw:

not out of the box, but it would be possible to do something like this by implementing a custom transport. e.g. we have a transport which decodes events and stores them in memory for testing purposes: https://github.com/elastic/apm-agent-go/blob/master/transport/transporttest/recorder.go

@ruflin:

Is this something that could be enabled "on-demand"? What I have in mind is something like:

User triggers elastic-agent diagnostic command
Recorder is enabled for x seconds
Recording is read out and stored on disk
Recorder is disabled again
Diagnostics finished

Yes, this is possible with a combination of APIs:

• apm.Tracer.SetRecording
• The aforementioned Transport' interface

i.e. elastic-agent would set up a custom transport that writes events to disk, and would call the tracer's SetRecording method on demand. Alternatively, have the tracer always "recording", but dynamically control the custom transport implementation instead. That might be better, as that way you duplicate events to the custom transport (for on-demand event collection) and to a standard APM Server transport.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[lizozom]

Hey :-)

I'm coordinating APM data collection for Kibana for internal uses.
I'd love to help and coordinate about this!