Dynamic tenants via x-scope-orgid

[bputt-e]

We're planning on using OTEL for ingest with OAuth. We expect to send spans to kafka and are wondering how we can dynamically pass along the x-scope-orgid from kafka to cortex/loki/tempo?

We know that OTEL is still developing their auth capability and it's unclear how we'll be able to extract that data. Here's some potential solutions we're considering:

1. Add the auth-related information to the kafka payload and the grafana-agent can read from kafka and pull out the necessary information and send the x-scope-orgid to cortex/loki/tempo
2. Similar to ^, but add in an otel agent before grafana-agent and send the data to grafana-agent via otlp (if we can set an x-scope-orgid header from the otel agent to the grafana-agent, that seems feasible)

Does grafana-agent have anything that would support this use case or if not, would it make sense to in the foreseeable future?

[rfratto]

It sounds like there's a lot to unpack here:

1. We would need support for Kafka or OTLP
2. The metrics subsystem would need some way to retrieve auth information for a single metric (Prometheus can't do this today)
3. Even with the above, you may run into out of order errors from Cortex and Loki when using Kafka. I'm not aware of any ongoing efforts to do this with Cortex, but IIUC it's being considered for Loki.

Kafka is something I've been hearing increasingly more often and I suspect we'll need to support it in the future. We don't have plans to work on it any time soon, so we'd really need external contributors to take charge on this one with proposals and PRs for it to happen quickly.

It's a similar case with OTLP, though I think we're further off there. I'd personally want the OTLP protocol for logs to be stable before considering adding OTLP support for all telemetry types.

[rfratto]

Just a little bit more to summarize my thoughts here: the Grafana Agent is based off of pull-based software, and works best when you're pulling metrics and logs. Adding support for pushing is going to introduce a lot of challenges depending on your requirements.

One potential way to avoid:

The metrics subsystem would need some way to retrieve auth information for a single metric (Prometheus can't do this today)

is to have the Kafka payload specify which metrics config from the Agent should receive the metric. Metrics configs specify auth and where data should be sent. You can do something similar with logs. It's not the same as passing the org ID header but it's similar, and would be easier to pull off today.

[bputt-e]

appreciate the insight, FWIW, we'll be attempting to have OTEL->kafka->grafana-agent->[loki, cortex, tempo]....So we're definitely interested in a pull-based method for grafana-agent to pull from kafka