feat: add tail sampling for traces

[Lp-Francois]

Add tail sampling for traces. Useful when dropping health probes from your traces for instance, as they can be quite noisy 💸.

Documentation: https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/setup/sampling/tail/

Related issues:

• Format issue with otelcol.processor.filter (traces.span) #431
• How to do tail sampling to drop health probes? #443

Comments:

• How to generate the example properly? I didn't find a way to generate the examples/tail-sampling/*.alloy files.
• Is there an easy way of testing the generated config? I made sure the helm chart output looks OK (with helm template k8s-monitoring charts/k8s-monitoring -f examples/tail-sampling/values.yaml > examples/tail-sampling/output.yaml), but didn't tested it would work correctly.
  • If there is no official way, I guess I could copy paste from the helm chart output the generated config and pass it locally to the alloy cli?
• The allowed configuration is quite a lot: https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.tail_sampling/#blocks I wonder if there would be an easy way to pass config for the policies? Maybe like the extraConfig is working, just let the user pass any string?

Note: It is really painful to work with these alloy_config/*.alloy.txt files 😅

[CLAassistant]

All committers have signed the CLA.

[Lp-Francois]

@t00mas I applied your reviews :)

Could you have another look if you find some time please 🙏 ?

[t00mas]

I'm OK with this, just take into account @rlankfo 's recommendation here #443 (comment)

[rlankfo]

I'd be concerned about adding the tail sampling processor at this stage of the pipeline. It won't be scalable -- to scale the tail sampling processor we recommend introducing a load balancing exporter in a separate layer that can route traces by trace ID. @petewall do you have any thoughts here?