[Profile] add op profile

GRAPH_OPERATOR_PROFILING.md

@@ -0,0 +1,172 @@
+# Runtime Kernel Profiling
+
+This workflow records and aggregates the rank-0 GPU kernels generated by one
+fixed vLLM graph-mode workload. CUDA Graph construction is not profiled or
+included in the generated inventory.
+
+The workload contains 64 concurrent requests. Each request has exactly 4096
+input tokens and 1024 output tokens. A complete warmup batch finishes before
+`/start_profile`; only the second complete batch is inside the profiling
+window.
+
+## DeepSeek-V4-Flash
+
+Terminal A:
+
+```bash
+bash tools/graph_operator_profile/serve_deepseek_v4_flash.sh
+```
+
+Terminal B, after the server is ready:
+
+```bash
+bash tools/graph_operator_profile/profile_request.sh \
+  deepseek_v4_flash \
+  tools/graph_operator_profile/deepseek_v4_flash_request.json
+```
+
+Results:
+
+```text
+/vllm-workspace/graph_operator_profile_runs/deepseek_v4_flash/results/
+```
+
+## Qwen3.6-35B-A3B
+
+Terminal A:
+
+```bash
+bash tools/graph_operator_profile/serve_qwen3_6_35b_a3b.sh
+```
+
+Terminal B, after the server is ready:
+
+```bash
+bash tools/graph_operator_profile/profile_request.sh \
+  qwen3_6_35b_a3b \
+  tools/graph_operator_profile/qwen3_6_35b_a3b_request.json
+```
+
+Results:
+
+```text
+/vllm-workspace/graph_operator_profile_runs/qwen3_6_35b_a3b/results/
+```
+
+## Output files
+
+`kernel_summary.csv` is the compact physical inventory. It has one row per
+kernel and the following columns:
+
+- `kernel_name`
+- `total_call_count`
+- `total_time_us`
+- `percent`
+
+`percent` has three digits after the decimal point. Values smaller than
+`0.001%` are emitted as `<0.001%`.
+
+`kernel_details_report.csv` is the detailed aggregate. It has one row per
+kernel/operator/shape/dtype/mapping-status combination and the following
+columns:
+
+- `kernel_name`
+- `variant_index`: stable one-based index within a kernel
+- `mapping_status`
+- `operator_name`
+- `input_shapes`
+- `input_dtypes`
+- `candidate_operators`
+- `kernel_event_count`
+- `kernel_time_us`
+
+The summary contains each kernel once. The details report can contain multiple
+rows for one kernel, but its distinct `kernel_name` values have exactly the
+same order and set as the summary.
+
+Repeated physical events with the same kernel, mapping status, operator,
+shape, dtype, and candidate metadata are represented by one aggregate row.
+`kernel_event_count` and `kernel_time_us` preserve their complete physical
+totals.
+
+Shapes, dtypes, and candidate operators are compact JSON values stored inside
+CSV cells. Missing shape, dtype, operator, or candidate information is the
+literal `null`, not an empty cell. Standard CSV quoting protects embedded
+commas and quotes. Missing or ambiguous metadata never removes a kernel.
+`mapping_status` is one of:
+
+- `operator_shape_matched`
+- `operator_matched_shape_missing`
+- `operator_matched_dtype_missing`
+- `operator_matched_metadata_missing`
+- `shape_ambiguous`
+- `operator_ambiguous`
+- `missing_external_id`
+- `no_cpu_op_match`
+
+`mapping_status` totals are derived by grouping the details rows; they are not
+duplicated as separate rows.
+
+`summary.json` records collection scope, CPU and GPU event counts, mapping
+coverage, and conservation results. A valid extraction requires every boolean
+in `conservation` to be `true`. The checks prove:
+
+- summary and details kernel sets and order are identical
+- trace, summary, report, and mapping-status event counts are identical
+- trace, summary, report, and mapping-status kernel times are identical
+- every individual kernel preserves its count and time in all report variants
+- re-reading both CSV files preserves every kernel, count, and duration
+
+Kernel time is stored internally as integer nanoseconds and emitted in
+microseconds. This avoids floating-point drift in aggregate conservation.
+
+## Collection boundary
+
+The server scripts explicitly disable graph-construction profiling:
+
+```bash
+export VLLM_FL_ENABLE_GRAPH_CAPTURE_PROFILE=0
+unset VLLM_FL_GRAPH_CAPTURE_PROFILE_DIR
+```
+
+`VLLM_FL_ENABLE_GRAPH_CAPTURE_PROFILE` defaults to disabled. The upstream
+`/start_profile` and `/stop_profile` behavior remains unchanged and writes one
+raw runtime trace per rank. The extractor selects rank 0:
+
+```bash
+python3 tools/graph_operator_profile/extract_operator_shapes.py \
+  --runtime <runtime_trace_directory> \
+  --rank 0 \
+  --output-dir <result_directory>
+```
+
+The launch scripts use graph mode, `max_num_seqs=64`, and CUDA Graph capture
+sizes `[1,2,4,8,16,32,64]`. Prefix caching is disabled. DeepSeek uses the
+standard CUDA Graph path through `VLLM_USE_BREAKABLE_CUDAGRAPH=0` and disables
+async scheduling. The tested TP8 workload reproduced a CUDA launch failure in
+the asynchronous output-copy path after runtime profiling started. Disabling
+async scheduling removes that path while retaining CUDA Graph execution and
+the same request workload.
+
+## Completeness boundary
+
+The inventory preserves every kernel event emitted in the selected rank-0
+runtime trace through aggregate count and duration. It does not preserve
+per-event timestamps, process IDs, thread IDs, streams, or External ids.
+
+Memcpy and memset counts and durations remain in `summary.json`, but they are
+not kernel keys. CPU-only operators are not included because both primary files
+are kernel-keyed. Operators visible only during CUDA Graph construction are
+intentionally excluded.
+
+CUDA Graph replay exposes physical kernels but usually does not replay the
+original PyTorch CPU operators. A graph-internal kernel may therefore have no
+recoverable operator, input shape, or dtype in the runtime trace. Its count and
+time remain in both primary files, with the missing logical metadata recorded
+as `null` and an explicit `mapping_status` in the details report.
+
+Fused kernels may represent multiple logical operators and are not assigned by
+kernel-name guessing. Different requests, batch sizes, sequence lengths,
+sampling settings, TP ranks, or MoE routing can activate different kernels and
+shapes. Rank 0 is a reproducible single-rank view, not proof that another rank
+has no additional activity.

README.md

@@ -119,6 +119,10 @@ Operator adapters use the plugin dispatch manager, so backend selection,
 fallback, per-op policy, operator-list recording, and I/O diagnostics continue
 to follow the common FlagOS controls.
 
+For a reproducible procedure that extracts graph-mode logical operators,
+input shapes, dtypes, and call counts, see
+[Graph-mode operator profiling](./GRAPH_OPERATOR_PROFILING.md).
+
 4. (Optional) Install [FlagCX](https://github.com/flagos-ai/FlagCX/blob/main/docs/getting_started.md#build-and-installation)
 
     4.1 Clone the repository:

tools/graph_operator_profile/deepseek_v4_flash_request.json

@@ -0,0 +1,7 @@
+{
+  "model": "deepseek-v4-flash",
+  "concurrency": 64,
+  "input_tokens": 4096,
+  "output_tokens": 1024,
+  "seed": 0
+}