Skip to content

GUIDELLM_DASHBOARD_SETUP.md

Latest commit

 

History

History
357 lines (269 loc) · 8.15 KB

GUIDELLM_DASHBOARD_SETUP.md

File metadata and controls

357 lines (269 loc) · 8.15 KB
Raw

GuideLLM Dashboard Setup Guide

Timeline Checklist

After Pushing Metrics (Now)

  • ✅ Metrics pushed to Thanos Receiver
  • ⏳ Wait ~30 minutes for TSDB compaction
  • ⏳ Wait ~2 hours total for S3 upload and Store Gateway sync

After 30 Minutes

Check if blocks are being created:

oc exec -n kserve-e2e-perf thanos-receiver-0 -- cat /var/thanos/receive/default-tenant/thanos.shipper.json

Should show new blocks uploaded.

After 2 Hours

Verify data is queryable from central cluster.

Step 1: Verify Data in S3 (After 2 Hours)

Switch to central cluster:

oc config use-context default/api-ocp4-intlab-redhat-com:6443/editor

Check Store Gateway has synced blocks:

# Get Store Gateway pod
STORE_POD=$(oc get pods -n psap-obs -l app=thanos-store-gateway -o jsonpath='{.items[0].metadata.name}')

# Check if it has guideLLM blocks
oc exec -n psap-obs $STORE_POD -- wget -qO- 'http://localhost:10902/api/v1/label/__name__/values' | jq -r '.data[]' | grep guidellm

Expected output:

guidellm_benchmark_duration_seconds
guidellm_benchmark_start_time_seconds
guidellm_output_tokens_per_second
guidellm_request_latency_seconds
guidellm_requests_errored_total
guidellm_requests_per_second
guidellm_requests_successful_total
guidellm_time_to_first_token_ms

Query sample data:

# Port-forward to Thanos Querier
oc port-forward -n psap-obs svc/thanos-querier 9090:9090

# In another terminal, query:
curl "http://localhost:9090/api/v1/query?query=guidellm_time_to_first_token_ms{cluster=\"mehulvalidation-wp4vb\"}" | jq '.data.result | length'

Should return > 0 (number of time series).

Step 2: Add Dashboard Variables

Add these variables to your existing Grafana dashboard for filtering.

Variable 1: guidellm_profile

Type: Query Name: guidellm_profile Label: GuideLLM Profile Query:

label_values(guidellm_time_to_first_token_ms{cluster=~"$cluster_name"}, profile)

Multi-value: Yes Include All: Yes

Variable 2: guidellm_concurrency

Type: Query Name: guidellm_concurrency Label: GuideLLM Concurrency Query:

label_values(guidellm_time_to_first_token_ms{cluster=~"$cluster_name", profile=~"$guidellm_profile"}, concurrency)

Multi-value: Yes Include All: Yes

Step 3: Dashboard Panels

Panel 1: GuideLLM TTFT by Concurrency

Panel Type: Time Series Title: GuideLLM - Time to First Token by Concurrency

Queries:

# P50 TTFT
guidellm_time_to_first_token_ms{
  cluster=~"$cluster_name",
  profile=~"$guidellm_profile",
  concurrency=~"$guidellm_concurrency",
  metric_type="p50"
}

# P90 TTFT
guidellm_time_to_first_token_ms{
  cluster=~"$cluster_name",
  profile=~"$guidellm_profile",
  concurrency=~"$guidellm_concurrency",
  metric_type="p90"
}

# P99 TTFT
guidellm_time_to_first_token_ms{
  cluster=~"$cluster_name",
  profile=~"$guidellm_profile",
  concurrency=~"$guidellm_concurrency",
  metric_type="p99"
}

Legend: {{profile}} - Concurrency {{concurrency}} - {{metric_type}}

Panel 2: GuideLLM Request Latency by Concurrency

Panel Type: Time Series Title: GuideLLM - End-to-End Request Latency

Queries:

# P50 Latency
guidellm_request_latency_seconds{
  cluster=~"$cluster_name",
  profile=~"$guidellm_profile",
  concurrency=~"$guidellm_concurrency",
  metric_type="p50"
}

# P90 Latency
guidellm_request_latency_seconds{
  cluster=~"$cluster_name",
  profile=~"$guidellm_profile",
  concurrency=~"$guidellm_concurrency",
  metric_type="p90"
}

Legend: {{profile}} - Concurrency {{concurrency}} - {{metric_type}} Unit: seconds (s)

Panel 3: GuideLLM Throughput by Concurrency

Panel Type: Time Series Title: GuideLLM - Tokens Per Second

Queries:

# Mean throughput
guidellm_output_tokens_per_second{
  cluster=~"$cluster_name",
  profile=~"$guidellm_profile",
  concurrency=~"$guidellm_concurrency",
  metric_type="mean"
}

Legend: {{profile}} - Concurrency {{concurrency}} Unit: tokens/sec

Panel 4: GuideLLM Request Success Rate

Panel Type: Stat Title: GuideLLM - Request Success Rate

Query:

(
  sum(guidellm_requests_successful_total{cluster=~"$cluster_name", profile=~"$guidellm_profile", concurrency=~"$guidellm_concurrency"})
  /
  (
    sum(guidellm_requests_successful_total{cluster=~"$cluster_name", profile=~"$guidellm_profile", concurrency=~"$guidellm_concurrency"})
    +
    sum(guidellm_requests_errored_total{cluster=~"$cluster_name", profile=~"$guidellm_profile", concurrency=~"$guidellm_concurrency"})
  )
) * 100

Unit: Percent (0-100) Thresholds: Red < 95%, Yellow < 99%, Green >= 99%

Panel 5: GuideLLM vs vLLM TTFT Comparison

Panel Type: Time Series Title: TTFT Comparison - GuideLLM (Client) vs vLLM (Server)

Queries:

# GuideLLM Client-side TTFT (p50)
guidellm_time_to_first_token_ms{
  cluster=~"$cluster_name",
  profile=~"$guidellm_profile",
  metric_type="p50"
} / 1000

# vLLM Server-side TTFT (p50)
histogram_quantile(0.50,
  rate(vllm:time_to_first_token_seconds_bucket{
    cluster=~"$cluster_name",
    deployment_uuid=~"$deployment_uuid"
  }[5m])
)

Legend:

  • GuideLLM Client - {{profile}} - C{{concurrency}}
  • vLLM Server - {{deployment_pod_name}}

Unit: seconds (s)

Panel 6: Concurrency vs GPU Utilization

Panel Type: Time Series Title: GuideLLM Concurrency vs GPU Utilization

Queries:

# Benchmark concurrency level
guidellm_request_concurrency{
  cluster=~"$cluster_name",
  profile=~"$guidellm_profile",
  metric_type="mean"
}

# GPU Utilization during benchmark
avg(DCGM_FI_DEV_GPU_UTIL{cluster=~"$cluster_name"})

Legend:

  • Concurrency - {{profile}}
  • GPU Utilization %

Y-axis: Dual axis (left: concurrency, right: %)

Panel 7: GuideLLM Benchmark Timeline

Panel Type: Table Title: GuideLLM Benchmark Runs

Query:

guidellm_benchmark_start_time_seconds{
  cluster=~"$cluster_name",
  profile=~"$guidellm_profile",
  concurrency=~"$guidellm_concurrency"
}

Transformations:

  1. Organize fields by name
  2. Add field from calculation: Start Time = from_unixtime($value)

Columns:

  • Profile
  • Concurrency
  • Model
  • Start Time
  • Duration

Step 4: Create Dashboard Row for GuideLLM

Add a new collapsed row titled "GuideLLM Benchmark Results" containing all the above panels.

Step 5: Upload Dashboard

Option 1: Via Grafana UI

  1. Open Grafana: https://grafana-psap-obs.apps.ocp4.intlab.redhat.com
  2. Navigate to existing dashboard (e.g., vLLM + DCGM Metrics - mehulvalidation-wp4vb)
  3. Click "Dashboard settings" (gear icon)
  4. Add variables: guidellm_profile and guidellm_concurrency
  5. Add new row: "GuideLLM Benchmark Results"
  6. Add panels using queries above
  7. Save dashboard

Option 2: Via JSON (Automated)

I'll create a Python script to programmatically add GuideLLM panels to your existing dashboard.

Verification Queries

After adding panels, verify data is showing:

  1. Check time range: Set Grafana to "Last 6 hours" or specific benchmark time
  2. Select variables:
    • guidellm_profile: profile-1
    • guidellm_concurrency: All or specific like 100
  3. Verify data points: Should see discrete data points at benchmark end times

Expected Results

Profile-1:

  • Concurrency levels: 1, 5, 50, 100, 200, 250
  • Data points: 5-6 (depending on what succeeded)
  • Timestamp: Around Oct 28, 2025 15:00-17:00 UTC

Profile-2:

  • Similar structure
  • Different ISL/OSL configuration

Profile-3:

  • Similar structure
  • Different ISL/OSL configuration

Troubleshooting

No data showing:

  1. Check time range matches benchmark timestamps
  2. Verify Store Gateway synced: oc logs -n psap-obs <store-gateway-pod> | grep sync
  3. Query Thanos Querier directly to verify data exists
  4. Check variable filters aren't excluding all data

Partial data:

  • Some concurrency levels may have failed to push (outside 6h window)
  • Check which ones succeeded during push

Data not correlating with vLLM:

  • Verify benchmark ran against same deployment_uuid
  • Check cluster labels match
  • Use absolute time ranges instead of relative