[Feature]: There is currently 0% AMD parity on BCFL tool calling nightly tests

[functionstackx]

Problem Statement

hi @chunfangamd @andyluo7

despite being an extremely popular router across Meta, Google Cloud Platform, Oracle Cloud, Alibaba Cloud ,TogetherAI

AMD currently has 0% parity on BCFL tool nightly tests due to lack of amd maintainers & espcially due to lack of AMD CI GPUs for SMG

https://github.com/smg-project/smg/blob/main/.github/workflows/nightly-bfcl.yml

Proposed Solution

amd contribute amd ci nodes & maintainers to fix this

Alternatives Considered

No response

Feature Area

Routing & Load Balancing

Affected Component(s)

No response

Use Case

i wanna use amd

Priority to You

Important

Contribution

• I am willing to contribute this feature (with guidance)
• I am willing to help test this feature
• I can provide more detailed requirements if needed

Additional Context

No response

[chunfangamd]

First results from running the BFCL A/B on AMD: on MI355X with ROCm, SMG main through vLLM's gRPC servicer passes the workflow's regression check against direct vLLM, for both a small model and gpt-oss-120b. In no run was SMG's unweighted overall more than the 2-point tolerance below vLLM's; the lowest was 1.01 points below. These are manual runs on our Slurm cluster, not a scheduled job yet.

Setup. Both arms ran on one MI355X node, one GPU each (TP1), from the same ROCm image (vLLM 0.30.1rc1.dev396+gac68c3087), the same weights and the same sampling. scripts/bfcl/launch_arm.sh and run_ab.py came unchanged from SMG ae3a61f2, and the SMG wheel, smg-grpc-proto and smg-grpc-servicer were built from that commit. BFCL was bfcl-eval==2026.3.23 in FC mode, through OpenAICompletionsHandler. For parsers, Qwen3-0.6B used hermes + qwen3 on vLLM and qwen + qwen3 on SMG, because its chat template has the model write tool calls as JSON inside <tool_call> tags (the nightly's Qwen3.8-27B leg uses the XML parsers instead); gpt-oss used the nightly leg's settings (openai on vLLM, nothing on SMG, so it takes the harmony path). For Qwen3-0.6B the arms build prompts of the same length: across 8,501 cases in five runs, the first prompt's token count was the same in all 8,478 cases where both arms recorded one. The other 23 are multi_turn_long_context cases in which one or both arms hit the 32,768-token context limit, so BFCL recorded no token counts for that arm.

Results (accuracy in %, vLLM / SMG, Δ = SMG − vLLM, in points; scores read from the BFCL score files; Δ is computed before rounding, so it can differ by 0.01 from the difference of the rounded scores):

Model	Categories (cases)	Runs	Unweighted vLLM / SMG (Δ)	Weighted vLLM / SMG (Δ)
Qwen3-0.6B	simple_python, irrelevance (640)	2	84.08 / 84.92 (+0.83); 84.75 / 84.33 (−0.42)	84.38 / 85.31 (+0.94); 85.00 / 84.38 (−0.62)
Qwen3-0.6B	the 7 non_live categories (1,390)	2	73.55 / 72.54 (−1.01); 71.99 / 72.43 (+0.44)	78.85 / 78.63 (−0.22); 78.42 / 78.20 (−0.22)
Qwen3-0.6B	the nightly's 17 categories (4,441)	1	51.85 / 54.00 (+2.15)	59.29 / 60.08 (+0.79)
Qwen3-0.6B, SMG arm on the Rust servicer (SMG_VLLM_SERVICER_IMPL=rust; the nightly has no such leg)	simple_python, irrelevance (640)	1	84.29 / 84.83 (+0.54)	84.84 / 85.00 (+0.16)
gpt-oss-120b	simple_python, irrelevance (640)	1	85.96 / 87.00 (+1.04)	86.09 / 86.88 (+0.78)
gpt-oss-120b	the nightly's 17 categories (4,441)	1	52.01 / 52.23 (+0.22)	66.70 / 66.81 (+0.11)

For reference, the 2026-10-05 H100 nightly (vLLM 0.27.1, TP2 per arm) scored the same 17 categories for gpt-oss-120b at 52.73 / 52.00 unweighted and 67.10 / 66.22 weighted. Per category, our numbers are within 3 points of the nightly's in 11 of the 13 single-turn categories; the other two are small (simple_javascript, 50 cases: SMG 4 points lower, 2 cases; live_relevance, 16 cases: vLLM 6.25 points lower, 1 case). The four multi-turn categories differ by up to 5 points. The vLLM versions differ and we have one gpt-oss run, so this is a sanity check, not a parity claim.

Things we ran into (items 1–4 also apply to the H100 nightly; 5 and 6 explain parts of our setup):

1. run_ab.py can read another category's score file (multiple matching parallel_multiple, and similar pairs). On our filesystem it misreported three of our runs and gave one false "REGRESSION" exit; in the gpt-oss 17-category run its table showed the unweighted Δ as +5.38 instead of +0.22. The 2026-10-05 nightly happens to be unaffected. fix(bfcl): read each category's own score file in run_ab.py #2808 fixes it and is merged; fix(bfcl): parse only the scores of the current evaluation in run_ab.py #2813, in review, also keeps run_ab.py from reading score files that earlier runs left in a reused project root.
2. gpt-oss-120b scores 0 on parallel, parallel_multiple, live_parallel and live_parallel_multiple, on both arms, in our run and in the H100 nightly alike. Those four categories carry no signal for that leg as it is set up today.
3. For gpt-oss, SMG's harmony rendering differs from vLLM's for requests with a system message or without tools. SMG keeps a system message as a second Harmony system message, where vLLM puts it in the developer message's instructions; SMG always adds a developer message, where vLLM adds one only when there are instructions or tools; and without tools SMG drops commentary from the valid channels, where vLLM keeps it. In the 17-category run this made the first prompt 1 or 2 tokens longer on SMG in 63 of the 4,441 cases: 59 of the 92 live cases with a system message (the other 33 differ in structure but happen to have the same length) and all 4 cases without tools. We filed it as Harmony chat path renders system messages and requests without tools differently from vLLM #2830. Qwen3-0.6B shows no such difference.
4. vLLM raises any nonzero temperature below 0.01 to 0.01 (sampling_params.py, in 0.27.1 as well), so the 0.001 that BFCL and run_ab.py use by default becomes 0.01. BFCL sends no top_p/top_k, so both arms sample with the model's defaults (Qwen3: top_p 0.95, top_k 20). In two identical runs, the same arm changed its pass/fail verdict on 2–13% of the cases in each category, and the number of cases where the arms disagreed was about the same. Between the identical runs, the unweighted Δ moved from +0.83 to −0.42 in the two-category runs and from −1.01 to +0.44 in the seven-category runs, so a 2-point tolerance on a single run leaves little margin.
5. The -FC entries bfcl-eval ships for small Qwen models (Qwen/Qwen3-0.6B-FC, Qwen/Qwen3-4B-Instruct-2507-FC, ...) use QwenFCHandler, which formats the prompt itself and calls /v1/completions, so neither arm's chat template or tool parser is exercised. register_bfcl_model.py leaves existing entries alone, so it does not switch them to OpenAICompletionsHandler. We served the model under a new id and registered that instead. The nightly's current legs are not affected: bfcl-eval 2026.3.23 ships neither openai/gpt-oss-120b-FC nor Qwen/Qwen3.8-27B-FC, so register_bfcl_model.py adds both with OpenAICompletionsHandler. A leg for one of those small Qwen models would be affected.
6. On ROCm, the image's vLLM pins grpcio and grpcio-reflection to 1.78.0 and the image has no grpcio-health-checking, while smg-grpc-servicer needs all three at >=1.81.1. We install the three at 1.81.1 in a venv on top of the image. In vLLM, only the gRPC launcher imports grpc, so this does not touch the direct-vLLM arm.

Next. Following how the H100 legs are run, we opened #2832 (a draft for now), which adds AMD legs to nightly-bfcl.yml, starting with the H100 legs' two models (gpt-oss-120b and Qwen3.8-27B). They get the same schedule, categories, sampling and tolerance as the H100 legs, the same job summary and artifacts, and their failures reach the existing nightly-triage issue. They are meant to run on AMD Instinct self-hosted runners provided by AMD; until such a runner is registered, the scheduled run will skip them, the way it skips the Blackwell legs today. Meanwhile we will keep running the A/B manually and post the results here. The PR also handles the run-to-run noise from item 4.