arch: Add OpenTelemetry (OTel) Distributed Tracing & Prometheus Metrics

[kavix]

🎯 Goal

Integrate OpenTelemetry (go.opentelemetry.io/otel) to instrument eko ai commands, tracing LLM API latency, CAS object retrieval, and SQLite query times.

[Sanchar127]

Hi! I'd like to work on this issue. I have experience with backend systems, distributed services, and observability, and I'm interested in implementing the OpenTelemetry tracing and Prometheus metrics integration. I'll first review the existing architecture and contributing guidelines, then work on a PR. Thanks!

[Sanchar127]

Hi @kavix, I’ve reviewed the existing architecture and started implementing the OTel tracing/metrics instrumentation and Prometheus exporter.
I have one architectural question regarding the Prometheus /metrics endpoint: Eko is currently a short-lived CLI, so an HTTP metrics server started during a CLI invocation would terminate when the command exits and wouldn't be reliably scrapeable by Prometheus.
Should use a persistent metrics/server process for Prometheus scraping, or should metrics be exported/flushed at the end of each CLI invocation?
I couldn't find an existing OTel/Prometheus design in the repository, so I wanted to confirm the intended approach before finalizing the implementation