[Feature] Transaction sample min duration

[nhlushak]

Is your feature request related to a problem? Please describe.
Configuring sample rate in APM agents is a great feature to decrease unwanted performance overhead on application and APM server itself, but on the other hand we faced with the problem when almost every of our "anomaly" traces are not covered up with transaction samples:

Describe the solution you'd like
APM agents already have a parameter to configure minimus duration for including a stacktrace for span (ELASTIC_APM_SPAN_FRAMES_MIN_DURATION). How about same for transaction samples? (e.g. ELASTIC_APM_TRANSACTION_SAMPLE_MIN_DURATION)

[axw]

@NikitaGl thanks for opening the issue!

Agents are not in a good position to make a sampling decision based on the transaction duration, as the sampling they perform is "head based" -- the decision is made as soon as the trace begins, to ensure the decision can be propagated and is coherent throughout the trace.

The problem you describe (but not the exact solution you describe) will be addressed by elastic/apm-server#4185.

[nhlushak]

@axw thanks for quick response!

Issue you addressed is exactly what we are looking for. But like we see on screenshot above, apm-server could potentially have no "interesting" samples to filter if sampling rate on agent is set to less than 100%. And then we are balancing between potential performance overhead or your tracing being less effective :)

Anyway, I've subscribed to future updates for tail-based sampling, so I guess, this issue could be closed?

[axw]

Issue you addressed is exactly what we are looking for. But like we see on screenshot above, apm-server could potentially have no "interesting" samples to filter if sampling rate on agent is set to less than 100%. And then we are balancing between potential performance overhead or your tracing being less effective :)

Yes, you're right that it's still not guaranteed. Our current approach to tail-based sampling is to prefer keeping slow transactions; the sampling probability will be weighted based on the transaction duration, rather than being uniform. Once the implementation is available for testing, your feedback will be greatly appreciated :)

Anyway, I've subscribed to future updates for tail-based sampling, so I guess, this issue could be closed?

Agreed, will close. Thanks again!