[ci-auto] Enable prometheus/grafana/telemetry in chatqna

[CICD-at-OPEA]

GenAIExample ChatQnA compose.yaml got changed

Below files are changed in this commit

• ChatQnA/docker_compose/intel/cpu/xeon/compose.telemetry.yaml
• ChatQnA/docker_compose/intel/cpu/xeon/compose_tgi.telemetry.yaml
• ChatQnA/docker_compose/intel/hpu/gaudi/compose.telemetry.yaml
• ChatQnA/docker_compose/intel/hpu/gaudi/compose_tgi.telemetry.yaml

Please verify if the helm charts need to be changed accordingly.

This issue was created automatically by CI.

[eero-t]

@lianhao / @poussa There are quite a few differences in how telemetry works in that "GenAIExamples" Docker compose update vs. what is currently in "GenAIInfra" Helm charts: https://github.com/opea-project/GenAIInfra/blob/main/helm-charts/monitoring.md

What's in compose files vs. Helm charts:

• Prometheus + Grafana (container specs) + configs
  • => Installed separately with kube-prometheus Helm chart
• Monitoring configuration (just) for ChatQnA services
  • => Enabled with Helm chart global.monitoring option for all apps using those services
• Downloading of "GenAIEval" TGI/vLLM Grafana dashboard specs (but no instructions on how to install them)
  • => dashboard Helm chart installs generic OPEA app perf dashboards with most relevant metrics [1]
• Node-exporter container spec + dashboard download
  • => Included to kube-prometheus Helm chart
• Gaudi metric monitor v1.19 + dashboard download
  • => GenAIInfra has doc telling how to install v1.16 [2], linked by Helm chart monitoring doc
• Jaeger (container spec) + tracing endpoint configuration
  • => missing

---

[1] Helm chart option and corresponding dashboard(s):

• global.monitoring: LLM metrics dashboard for megaservice + TEI rerank/embed + TGI/vLLM services
• +autoscaling: Request rate/scaling dashboard for megaservice + retriever + TEI rerank/embed + llm-uservice + TGI/vLLM

[2] https://github.com/opea-project/GenAIInfra/tree/main/kubernetes-addons/Observability#device-metrics-for-gaudi-hw

[eero-t]

As to do what about the difference....

Helm charts try not to override users' own setup and to (some extent) avoid duplication, that's why monitoring is installed separately. In Kubernetes it's easy to connect things together, but if everything needs to be hard-coded (ports etc) for Docker compose, that would be much harder. Therefore I think it's valid for these to work differently, and just close functionality gaps.

Gaudi monitor

Instructions here: https://github.com/opea-project/GenAIInfra/tree/main/kubernetes-addons/Observability#device-metrics-for-gaudi-hw

Would need to be changed to:

• Use same version as "GenAIExamples" compose file(s)
• Get files from Habana: https://docs.habana.ai/en/v1.19.2/Orchestration/Prometheus_Metric_Exporter.html#deploying-prometheus-metric-exporter-in-kubernetes
• Get Dashboard from "GenAIEval" as "GenAIExamples" does

After which the current files under habana/ subdir can be dropped.

Jaeger install

I guess that needs:

• (New) tracing.md doc explaining Jaeger Kubernetes install, e.g. with: https://github.com/jaegertracing/helm-charts#readme
• Helm chart changes for configuring telemetry endpoint for all services

I'm not going to look into that myself though.

vLLM/TGI dashboards

I could add service specific dashboards for things like TGI & vLLM (which have a lot of metrics important for analyzing service performance in more detail), but I'm a bit worried about duplication of "GenAIEval" content.

In case there are many instances of LLMs running in a cluster, their Kubernetes dashboards would need a namespace selector, so that metrics for replicas of one app can be separated from replicas for another app. However, metric labels like namespace are missing for plain Docker installs => k8s dashboard variant might not work with metrics from Docker service.

One option to make maintaining (diffing etc) dashboards variants between GenAIEval & GenAIInfrace slightly easier, would be changing GenAIInfra to:

• Have dashboard specs as JSON files that can be loaded to & saved from Grafana directly, and easily compared against "GenAIEval" JSON versions
• A script to convert them to Helm configMap templates, based e.g. on this: https://github.com/opea-project/GenAIInfra/blob/main/kubernetes-addons/Observability/update-dashboards.sh
• Script, JSON files and configMap templates generated from JSON, all being in Git

Comments?

[CICD-at-OPEA]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[eero-t]

No progress on Gaudi monitoring or Jaeger install support.

I could add service specific dashboards for things like TGI & vLLM (which have a lot of metrics important for analyzing service performance in more detail), but I'm a bit worried about duplication of "GenAIEval" content.

In case there are many instances of LLMs running in a cluster, their Kubernetes dashboards would need a namespace selector, so that metrics for replicas of one app can be separated from replicas for another app. However, metric labels like namespace are missing for plain Docker installs => k8s dashboard variant might not work with metrics from Docker service.

Sakari's feedback was that it's enough to support vLLM, and document that for KubeAI.

That was merged in #1086 and #1144 PRs.

One option to make maintaining (diffing etc) dashboards variants between GenAIEval & GenAIInfrace slightly easier, would be changing GenAIInfra to:

• Have dashboard specs as JSON files that can be loaded to & saved from Grafana directly, and easily compared against "GenAIEval" JSON versions
• A script to convert them to Helm configMap templates, based e.g. on this: https://github.com/opea-project/GenAIInfra/blob/main/kubernetes-addons/Observability/update-dashboards.sh
• Script, JSON files and configMap templates generated from JSON, all being in Git

(Open) PR #1155 implements those.