[META 497] Collecting statistics about dropped spans

[AlexanderWert]

As described in this spec, agents should collect statistics on dropped spans.

Meta issue:

[astorm]

Looking at the spec and using the java agent's implementation as a reference, it looks like span statistics make the assumption that even a span that's not kept still exists for the lifetime of a transaction. Specifically, the statistics include data that's only avaiable at span end (the duration.sum.us property). The Node Agent's current implementation doesn't share this assumption.

ex. when the Node.js agent exceeds the transaction-max-spans limit, our createSpan method skips creating a span. However, in the same transaction-max-spans scenario in the java agent, we mark the span as not recorded (aka not sampled), and it's not until the tracer's endSpan method that the sampled flag is checked and an early return prevents the span from being sent. (you can also see the code calls captureDroppedSpan and that this method eventually records the stats for both count and duraction)

Therefore, in order to implement span statistics, we'll need to change our implementation such that spans are created but their context marked to indicate that recorded/sampled=false when transaction-max-span is exceeded and to not report spans that are unsampled.

Work to be done

• Implement a setRecorded method for our TraceContext and TraceParent object(s) (using the traceparent flags)
• Change transactionMaxSpans check to no longer return nullearly, instead create the the span but call setRecorded(false). Also, we'll add add an additional check in span end to prevent recording of unsampled spans (Risk/Unknown: Where to add this check? In span.end? or instrumentation.addEndedSpan? Or just block _encodeAndSendSpan? Or some other thing? What will the effect be now doing all the additional span ending work be?)
• Implement logic to capture span statistics when sampling or dropping a span. Serialize these statistics on transaction end.