feat(rhaiis): add native HTTP profiler backend [Under test] - #182
feat(rhaiis): add native HTTP profiler backend [Under test]#182naveenmiriyaluredhat wants to merge 1 commit into
Conversation
Add a second profiler collection path that uses the engine's own /start_profile and /stop_profile APIs instead of the vLLM mutating webhook. Trace copy and S3 upload stay on the same path; only how capture is armed changes. Backends - webhook (default): unchanged. Label the ISVC, write /tmp/profiler_gate, run GuideLLM, copy /tmp/trace_*.json*. The webhook counts execute_model calls (typically 500-503); GuideLLM does not stop when that range ends. - native: deploy with vLLM --profiler-config (0.13+) or SGLang SGLANG_TORCH_PROFILER_DIR, POST /start_profile, run the same profiler GuideLLM load, POST /stop_profile (flush can take many minutes), copy traces_dir. /stop_profile 404 means the process was not started with profiler-config. GuideLLM during Phase 1 is a fixed wall clock: rhaiis.profiler.max_seconds (default 200s) at rhaiis.profiler.rates (default 200 concurrent). It is independent of the webhook call range and of native delay/max_iterations. Native kinds (vLLM --profiler-config.profiler) - torch: PyTorch profiler → Chrome/Perfetto json.gz (default). - cuda: CUDA Profiler API; set native.nsys_wrap to wrap the process in nsys (serving image must contain nsys). - proton: Triton Proton (CUPTI), enforce-eager; chrome_trace or hatchet. SGLang uses the same native HTTP flow (no webhook). Start body carries num_steps/start_step/activities. TRT-LLM is still unsupported. Implementation - orchestration/profiler.py builds profiler-config JSON and mutates ServingRuntime args/env. - toolbox/control_native_profiler POSTs start/stop with a long stop timeout. - copy_profiler_traces accepts remote_dir and normalizes native filenames so S3 still matches trace_*rank0*. - Nightlies that only set profiler.enabled keep the webhook backend. Reference presets (stack with a model + workload): profiler-webhook, profiler-native, profiler-native-short, profiler-native-window, profiler-native-cuda, profiler-native-proton, profiler-native-sglang --preset llama-8b --preset profile1 --preset profiler-native-short Co-authored-by: Cursor <cursoragent@cursor.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
naveenmiriyaluredhat
changed the title
|
@kpouget @Harshith-umesh I am still testing this feature |
| profiler-native-window: | ||
| rhaiis.profiler.enabled: true | ||
| rhaiis.profiler.backend: native | ||
| rhaiis.profiler.kind: torch |
There was a problem hiding this comment.
here you should use the extends flag:
profiler-native-short:
extends: [profiler-native]
rhaiis.profiler.rates: [1]
rhaiis.profiler.max_seconds: 60
that's a simple mechanism that just inserts the profile-native presets at the location of the extends marker
| "nsys", | ||
| "profile", | ||
| "--trace-fork-before-exec=true", | ||
| "--cuda-graph-trace=node", | ||
| "--capture-range=cudaProfilerApi", | ||
| "--capture-range-end=repeat", | ||
| f"--output={nsys_output}", |
There was a problem hiding this comment.
I try to keep all the constants in the config. That's not an absolute rule, more a guideline.
If you have:
nsys:
profile:
args:
trace_fork_before_exec: true
cuda_graph_trace: node
capture_range: cudaProfilerApi
capture_range_end: repeat
then maybe another way you (or someone else) will be happy to just have to define a preset to change
nsys.profile.args.cuda_graph_trace: cluster # random example
| env_vars.setdefault("VLLM_RPC_TIMEOUT", DEFAULT_RPC_TIMEOUT_MS) | ||
| env_vars.setdefault("VLLM_RPC_GET_DATA_TIMEOUT_MS", DEFAULT_RPC_TIMEOUT_MS) |
There was a problem hiding this comment.
default values can be in the config
same reason as above, if on a "busy day" you find that the timeout is too short, you'll be happy to extend it with a preset
| image=benchmark_cfg.get("image", "ghcr.io/vllm-project/guidellm:v0.6.0"), | ||
| timeout=benchmark_timeout, | ||
| pvc_size=benchmark_cfg.get("pvc_size", "5Gi"), | ||
| guidellm_args=guidellm_args, | ||
| hf_token_secret=benchmark_cfg.get("hf_token_secret", ""), | ||
| fs_group=benchmark_cfg.get("fs_group"), |
There was a problem hiding this comment.
shouldn't have default values here IMO, if the config is malformed you want to know it rightaway when you smoke test
2 participants
Add a second profiler collection path that uses the engine's own /start_profile and /stop_profile APIs instead of the vLLM mutating webhook. Trace copy and S3 upload stay on the same path; only how capture is armed changes.
Backends
GuideLLM during Phase 1 is a fixed wall clock: rhaiis.profiler.max_seconds (default 200s) at rhaiis.profiler.rates (default 200 concurrent). It is independent of the webhook call range and of native delay/max_iterations.
Native kinds (vLLM --profiler-config.profiler)
SGLang uses the same native HTTP flow (no webhook). Start body carries num_steps/start_step/activities. TRT-LLM is still unsupported.
Implementation
Reference presets (stack with a model + workload):
profiler-webhook, profiler-native, profiler-native-short,
profiler-native-window, profiler-native-cuda, profiler-native-proton,
profiler-native-sglang
--preset llama-8b --preset profile1 --preset profiler-native-short