[Profile] add op profile - #422
Open
cyber-pioneer wants to merge 10 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a graph-mode operator profiling workflow for vLLM-FL, including an opt-in CUDA Graph capture profiling hook, helper scripts to run/profile fixed requests, and documentation for reproducing and extracting operator shape/dtype inventories.
Changes:
- Add an opt-in Torch Profiler capture phase during CUDA Graph construction (
VLLM_FL_GRAPH_CAPTURE_PROFILE_DIR) and a kernel warmup opt-out (VLLM_FL_SKIP_KERNEL_WARMUP). - Add tooling under
tools/graph_operator_profile/to serve specific models, run a fixed profiling request, and extract operator shape/dtype rows into CSV/JSON. - Update docs (README + new guide) and adjust NVIDIA dispatch config to blacklist
emptyfor a DeepSeek-V4 FP8/FlagGems compatibility case.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| vllm_fl/worker/worker.py | Adds a DeepSeek-V4 Triton launch arg compat patch and an env-controlled kernel warmup skip. |
| vllm_fl/worker/model_runner.py | Adds an opt-in Torch Profiler wrapper around real CUDA Graph capture execution. |
| vllm_fl/dispatch/config/nvidia.yaml | Blacklists empty for a documented FlagGems + DeepSeek-V4 FP8 issue. |
| tools/graph_operator_profile/serve_qwen3_6_35b_a3b.sh | Launch script to run Qwen with capture/runtime profiling directories configured. |
| tools/graph_operator_profile/serve_deepseek_v4_flash.sh | Launch script to run DeepSeek-V4-Flash with capture/runtime profiling directories configured. |
| tools/graph_operator_profile/qwen3_6_35b_a3b_request.json | Fixed request payload for the Qwen profiling run. |
| tools/graph_operator_profile/deepseek_v4_flash_request.json | Fixed request payload for the DeepSeek profiling run. |
| tools/graph_operator_profile/profile_request.sh | Orchestrates warmup + runtime profiling request + extraction. |
| tools/graph_operator_profile/extract_operator_shapes.py | Parses profiler traces and emits per-phase and union CSV outputs plus summary JSON. |
| README.md | Adds a pointer to the new graph-mode operator profiling documentation. |
| GRAPH_OPERATOR_PROFILING.md | New end-to-end reproducibility guide for capture/runtime profiling and extraction. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+94
to
+95
| kernel = fused_inv_rope_fp8_quant._fused_inv_rope_fp8_quant_per_head | ||
| original_run = kernel.run |
Comment on lines
+24
to
+46
| prefix = source.read(4096) | ||
| source.seek(0) | ||
| if '"traceEvents"' not in prefix: | ||
| yield from json.load(source).get("traceEvents", []) | ||
| return | ||
| for line in source: | ||
| if '"traceEvents"' in line: | ||
| break | ||
| else: | ||
| raise ValueError(f"traceEvents not found: {path}") | ||
| event_lines: list[str] = [] | ||
| for line in source: | ||
| if not event_lines: | ||
| if line.startswith(" {"): | ||
| event_lines.append(line) | ||
| elif line.lstrip().startswith("]"): | ||
| return | ||
| continue | ||
| event_lines.append(line) | ||
| if line.startswith(" }"): | ||
| encoded = "".join(event_lines).rstrip().removesuffix(",") | ||
| yield json.loads(encoded) | ||
| event_lines.clear() |
Comment on lines
+11
to
+13
| RUN_ROOT=${PROFILE_RUN_ROOT:-/vllm-workspace/graph_operator_profile_runs} | ||
| RUN_DIR="$RUN_ROOT/$MODEL_CASE" | ||
| PROFILE_DIR="$RUN_DIR/profile" |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Suppressed comments (4)
vllm_fl/worker/worker.py:95
- _patch_deepseek_v4_launch_pdl can raise AttributeError if the DeepSeek-V4 module is present but its internal symbol layout changes (e.g., missing _fused_inv_rope_fp8_quant_per_head or .run). Since this runs during worker init, that would crash startup; it should fail closed (skip patch) with a small debug log.
kernel = fused_inv_rope_fp8_quant._fused_inv_rope_fp8_quant_per_head
original_run = kernel.run
tools/graph_operator_profile/profile_request.sh:14
- profile_request.sh writes responses under $RUN_DIR but never ensures the directory exists; running the script before (or without) the matching serve_*.sh will fail with "No such file or directory" on the first redirect. Creating $RUN_DIR makes the helper more robust and keeps failures focused on server health/profile endpoints.
RUN_ROOT=${PROFILE_RUN_ROOT:-/vllm-workspace/graph_operator_profile_runs}
RUN_DIR="$RUN_ROOT/$MODEL_CASE"
PROFILE_DIR="$RUN_DIR/profile"
TOOL_DIR=$(cd "$(dirname "$0")" && pwd)
tools/graph_operator_profile/extract_operator_shapes.py:40
- iter_events assumes torch profiler JSON always indents event objects with exactly two leading spaces (
" {"). JSON whitespace is not significant, so this can silently skip all events if indentation differs. Use a lstrip-based check so parsing is robust across torch/profiler versions.
if not event_lines:
if line.startswith(" {"):
event_lines.append(line)
elif line.lstrip().startswith("]"):
return
tools/graph_operator_profile/extract_operator_shapes.py:46
- Similarly, iter_events requires the closing brace line to start with exactly two spaces (
" }"). If the profiler emits a different indentation (or},), the parser won't emit any events. Use a lstrip-based check so end-of-object detection is indentation-insensitive.
event_lines.append(line)
if line.startswith(" }"):
encoded = "".join(event_lines).rstrip().removesuffix(",")
yield json.loads(encoded)
event_lines.clear()
| active=0 | ||
| stop_profile() { | ||
| if [[ "$active" -eq 1 ]]; then | ||
| curl -fsS -XPOST "$BASE_URL/stop_profile" |
Comment on lines
+136
to
+139
| "csv_event_output_coverage_pct": ( | ||
| sum(aggregate.values()) / total * 100 if total else 0.0 | ||
| ), | ||
| } |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Suppressed comments (4)
Previously missed (1) — in code that hasn't changed since the last review.
vllm_fl/worker/model_runner.py:6820
- The graph-capture profiling branch duplicates the full _dummy_run call across if/else. This increases the risk of the two branches drifting over time (e.g., one gets a new keyword arg and the other doesn’t). Consider refactoring to a single call (e.g., always wrap with _profile_graph_capture since it is already a no-op when the env var is unset, or extract the kwargs into a dict shared by both branches).
if os.environ.get("VLLM_FL_GRAPH_CAPTURE_PROFILE_DIR", ""):
with _profile_graph_capture(
desc.num_tokens,
cudagraph_runtime_mode,
):
vllm_fl/worker/worker.py:62
- The warmup import fallback only catches ImportError, but attribute or other initialization failures inside vLLM's warmup module (e.g., missing deep_gemm_warmup symbol) will currently crash worker import instead of gracefully degrading to the no-op fallback.
except ImportError:
# deep_gemm may be broken in some environments; provide a fallback
import logging as _logging
_logging.getLogger(__name__).warning(
"kernel_warmup import failed (likely deep_gemm issue), "
vllm_fl/worker/worker.py:116
- _patch_deepseek_v4_launch_pdl assumes private symbols (_fused_inv_rope_fp8_quant_per_head, kernel.run) exist once the import succeeds. If upstream changes these names, worker init will raise AttributeError and fail hard; this patch should degrade gracefully when the expected symbols are absent.
kernel = fused_inv_rope_fp8_quant._fused_inv_rope_fp8_quant_per_head
original_run = kernel.run
tools/graph_operator_profile/extract_operator_shapes.py:40
- iter_events relies on exact two-space indentation (line.startswith(" {") / " }") when streaming traceEvents. Torch profiler JSON formatting can vary, so this parser can silently skip events or produce empty output depending on whitespace. Use lstrip() for the brace checks to make the streaming mode indentation-agnostic.
if not event_lines:
if line.startswith(" {"):
event_lines.append(line)
elif line.lstrip().startswith("]"):
return
Comment on lines
+750
to
+756
| if _serialized_deep_gemm_warmup is not None: | ||
| kernel_warmup.__globals__["deep_gemm_warmup"] = ( | ||
| _serialized_deep_gemm_warmup | ||
| ) | ||
| kernel_warmup(self) | ||
| if torch.distributed.is_initialized(): | ||
| torch.distributed.barrier() |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Suppressed comments (2)
tools/graph_operator_profile/profile_request.sh:21
stop_profileruns in anEXITtrap underset -e, so a failure to reach$BASE_URL/stop_profilecan cause the trap itself to fail and mask the original error (or make the script fail on cleanup). Make the cleanup best-effort.
stop_profile() {
if [[ "$active" -eq 1 ]]; then
curl -fsS -XPOST "$BASE_URL/stop_profile"
active=0
fi
vllm_fl/worker/worker.py:754
compile_or_warm_up_modelpermanently mutateskernel_warmup.__globals__["deep_gemm_warmup"], which can leak a surprising behavior change beyond this warmup call (and makes repeated calls harder to reason about). Patch it only for the duration ofkernel_warmup(self)and then restore the original binding.
if _serialized_deep_gemm_warmup is not None:
kernel_warmup.__globals__["deep_gemm_warmup"] = (
_serialized_deep_gemm_warmup
)
kernel_warmup(self)
Comment on lines
+48
to
+61
| # FlagGems 5.3.4 cannot lower torch.float8_e8m0fnu allocations used by | ||
| # DeepSeek-V4 FP8 UE8M0 scale parameters. Fall back to native PyTorch. | ||
| - empty | ||
| - index_put_ | ||
| - index_put | ||
| - _index_put_impl_ | ||
| - nonzero | ||
| - copy_ | ||
| - to_copy | ||
| - index | ||
| # FlagGems linear/mm autotuning can access invalid memory for the dynamic | ||
| # DeepSeek-V4 logits-projection shapes in a concurrent prefill/decode batch. | ||
| - linear | ||
| - mm |
Comment on lines
+116
to
+120
| config = json.loads(args.config.read_text(encoding="utf-8")) | ||
| model = str(config["model"]) | ||
| concurrency = int(config["concurrency"]) | ||
| input_tokens = int(config["input_tokens"]) | ||
| output_tokens = int(config["output_tokens"]) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Category
PR Type
Description
Related Issues
Changes
Testing
Checklist