Skip to content

Conversation

TedThemistokleous
Copy link
Collaborator

@TedThemistokleous TedThemistokleous commented Aug 20, 2025

Motivation

Fixes an issue I was seeing with multi outptut fusions. Multi output would generate additional outputs and blow up so to speak for certain models and was causing broadcasts to be removed. Ensuring we fuse broadcasts after multioutput fusion pass is complte gives us a more stable run.

During debug I was seeing cases where fuse broadcast was creating a large amount of outputs resulting in a few cases where inputs would be mismatched on check during the fusion for multi output.

I don't see this anymore when allowing a reduce multioutput between broadcasts

Technical Details

Was hitting a failing case during MIGraphx compile related to multi output fusions being on. This was occurring within the customer model

Seeing about a 2-3 ms boost in perf run on fp16 and 5-6ms on fp32 runs now when I'm able to get multi output fusions running.

Test Plan

Running customer model without this change generates the erroneous case. Now able to run things end to end without needing to specify the MIGRAPHX_DISABLE_MULTI_OUTPUT_FUSION flag

Test Result

Pass/fail in this case.

Context to why this works

Relevant context for why this works - How we apply fusions.

void fuse_pointwise::apply(module_pass_manager& mpm) const
{
    mpm.run_pass(eliminate_identity{});
    create_pointwise_modules(mpm);
    mpm.run_pass(dead_code_elimination{});
    if(enabled(MIGRAPHX_DISABLE_POINTWISE_FUSION{}))
    {
        return;
    }
    for(int i = 0; i < 8; i++)
    {
        if(enable_rewrite_reshapes)
            mpm.run_pass(rewrite_reshapes<pointwise_reshape>{});
        if(enable_rewrite_broadcasts)
            rewrite_broadcasts(mpm);
        if(not find_pointwise_modules(mpm, enable_multi_output))
            break;
        mpm.run_pass(dead_code_elimination{});
    }
}

From what I was seeing when debugging customer model getting a large amount of inputs when using the rewrite broadcast pass that could potentially rewrite and change the number of inputs so that they don't match through a bunch of broadcasts. I probed and tried to track the find_inputs call which highlghted this.

eg) Rsz = result size, pSz is parameter size

before findinputs
pointwise(multibroadcast[out_lens={1, 64, 192, 256},out_dyn_dims={}], multibroadcast[out_lens={1, 64, 192, 256},out_dyn_dims={}], pointwise, multibroadcast[out_lens={1, 64, 192, 256},out_dyn_dims={}], pointwise) -> float_type, {1, 64, 192, 256}, {3145728, 49152, 256, 1}
pointwise(multibroadcast[out_lens={1, 64, 192, 256},out_dyn_dims={}], multibroadcast[out_lens={1, 64, 192, 256},out_dyn_dims={}], pointwise, multibroadcast[out_lens={1, 64, 192, 256},out_dyn_dims={}], pointwise) -> float_type, {1, 64, 192, 256}, {3145728, 49152, 256, 1}

...
.
.

try: 
multibroadcast[out_lens={1, 64, 192, 256},out_dyn_dims={}](get_tuple_elem[index=0]) -> float_type, {1, 64, 192, 256}, {64, 1, 0, 0}
a: 
multibroadcast[out_lens={1, 64, 192, 256},out_dyn_dims={}](get_tuple_elem[index=0]) -> float_type, {1, 64, 192, 256}, {64, 1, 0, 0}
try: 
mul(sub, @param:x2) -> float_type, {1}, {0}
no param
try: 
@param:x7 -> float_type, {1}, {0}
no input
try: 
sub(@param:x0, @param:x1) -> float_type, {1}, {0}
no param
try: 
add(add, add) -> float_type, {1}, {0}
no param
try: 
@param:x1 -> float_type, {1}, {0}
no input
try: 
mul(sub, @param:x5) -> float_type, {1}, {0}
no param
BAD BAD BAD
Rsz= 13
psZ= 14
driver: /workspace/AMDMIGraphX/src/param_utils.cpp:101: std::vector<instruction_ref> migraphx::find_inputs_impl(const std::unordered_map<instruction_ref, instruction_ref> &, const_module_ref, F) [F = (lambda at /workspace/AMDMIGraphX/src/param_utils.cpp:123:23)]: Assertion `not sub or result.size() == sub->get_parameter_shapes().size()' failed.

When I disabled the rewrite_broadcast pass, I was getting input sizes of at most 4 or 5 in this customer model, but with the rewrite pass the broadcasts start to get more inputs (10+) per module. If I interleave these between multi outputs things don't break and still allows us to handle the multi output pass.

…ns in same pass

This gives a perf boost but also fixes an issue I was seeing with multi outptu fusions. Multi output would generate additional outputs and blow up so to speak for certain models and was causing broadcasts to be removed. Ensuring we fuse broadcasts after multioutput fusion pass is complte gives us a more stable run.

During debug I was seeing cases where fuse broadcast was creating a large amount of outputs resulting in a few cases where inputs would be mismatched on check during the fusion for multi output.

I don't see this anymore when allowing a reduce multioutput between broadcasts
@TedThemistokleous TedThemistokleous self-assigned this Aug 20, 2025
@TedThemistokleous TedThemistokleous added bugfix Fixes a bug found in the code. simple small or simple changes Perf Improve labels Aug 20, 2025
@pfultz2
Copy link
Collaborator

pfultz2 commented Aug 20, 2025

It seems this isnt fixing the issue just bypassing it. The multi-output fusion shouldn't crash regardless of what was run before it.

@migraphx-bot
Copy link
Collaborator

Test Batch Rate new
55dd53
Rate old
5da1bd
Diff Compare
torchvision-resnet50 64 3,231.46 3,247.81 -0.50%
torchvision-resnet50_fp16 64 6,920.19 6,949.62 -0.42%
torchvision-densenet121 32 2,440.33 2,449.55 -0.38%
torchvision-densenet121_fp16 32 4,153.35 4,178.66 -0.61%
torchvision-inceptionv3 32 1,628.07 1,635.72 -0.47%
torchvision-inceptionv3_fp16 32 2,744.52 2,758.74 -0.52%
cadene-inceptionv4 16 766.83 770.87 -0.52%
cadene-resnext64x4 16 809.25 818.35 -1.11%
slim-mobilenet 64 7,425.04 7,449.28 -0.33%
slim-nasnetalarge 64 210.02 211.00 -0.46%
slim-resnet50v2 64 3,326.87 3,341.25 -0.43%
bert-mrpc-onnx 8 1,137.74 1,147.74 -0.87%
bert-mrpc-tf 1 445.14 444.91 0.05%
pytorch-examples-wlang-gru 1 299.61 355.05 -15.62% 🔴
pytorch-examples-wlang-lstm 1 411.58 407.70 0.95%
torchvision-resnet50_1 1 762.21 768.58 -0.83%
cadene-dpn92_1 1 390.74 391.19 -0.11%
cadene-resnext101_1 1 386.58 388.63 -0.53%
onnx-taau-downsample 1 393.81 395.86 -0.52%
dlrm-criteoterabyte 1 33.67 33.76 -0.27%
dlrm-criteoterabyte_fp16 1 51.13 51.17 -0.09%
agentmodel 1 8,598.08 8,973.04 -4.18% 🔴
unet_fp16 2 58.98 59.19 -0.35%
resnet50v1_fp16 1 973.86 985.57 -1.19%
resnet50v1_int8 1 1,043.85 1,028.23 1.52%
bert_base_cased_fp16 64 1,099.63 1,106.80 -0.65%
bert_large_uncased_fp16 32 344.02 345.19 -0.34%
bert_large_fp16 1 197.24 196.38 0.44%
distilgpt2_fp16 16 2,103.51 2,118.23 -0.69%
yolov5s 1 578.56 565.66 2.28%
tinyllama 1 43.78 43.95 -0.40%
vicuna-fastchat 1 45.11 45.10 0.01%
whisper-tiny-encoder 1 415.73 417.83 -0.50%
whisper-tiny-decoder 1 407.39 408.78 -0.34%
llama2_7b 1 19.12 19.16 -0.20%
qwen1.5-7b 1 23.42 23.53 -0.47%
phi3-3.8b 1 26.58 26.67 -0.36%
mask-rcnn 1 12.55 12.43 1.00%
llama3-8b 1 21.63 21.72 -0.41%
whisper-large-encoder 1 10.16 10.21 -0.43%
whisper-large-decoder 1 95.93 96.82 -0.93%
mistral-7b 1 23.63 23.72 -0.35%
FLUX.1-schnell 1 456.31 744.11 -38.68% 🔴
nan nan nan nan nan%

This build is not recommended to merge 🔴

@migraphx-bot
Copy link
Collaborator


     ✅ bert-mrpc-onnx: PASSED: MIGraphX meets tolerance

❌bert-mrpc-tf: ERROR - check error outputerror: unknown warning option '-Wnrvo' [-Werror,-Wunknown-warning-option]

error: unknown warning option '-Wnrvo' [-Werror,-Wunknown-warning-option]

2025-08-20 00:31:05.793061: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1755667871.028373 172386 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 62951 MB memory: -> device: 0, name: AMD Instinct MI250X/MI250, pci bus id: 0000:32:00.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1755667871.941853 172386 mlir_graph_optimization_pass.cc:401] MLIR V1 optimization pass is not enabled
2025-08-20 00:31:22.771172: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-08-20 00:31:22.771443: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-08-20 00:31:22.771491: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-08-20 00:31:22.771522: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-08-20 00:31:22.771567: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-08-20 00:31:22.771608: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-08-20 00:31:22.771653: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-08-20 00:31:22.771697: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
2025-08-20 00:31:22.772829: E tensorflow/compiler/mlir/tools/kernel_gen/tf_framework_c_interface.cc:228] INTERNAL: Generating device code failed.
2025-08-20 00:31:22.774017: W tensorflow/core/framework/op_kernel.cc:1829] UNKNOWN: JIT compilation failed.
2025-08-20 00:31:22.774036: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
2025-08-20 00:31:22.774046: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
2025-08-20 00:31:22.774061: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 11217777527359497193
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1407, in _do_call
return fn(*args)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1390, in _run_fn
return self._call_tf_sessionrun(options, feed_dict, fetch_list,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1483, in _call_tf_sessionrun
return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 359, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 335, in main
y_out = sess.run(y, feed_dict=tf_dict)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 977, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1220, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1400, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1426, in _do_call
raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.UnknownError: Graph execution error:

Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'import/bert/embeddings/LayerNorm/moments/SquaredDifference':


     ✅ pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

     ✅ pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

     ✅ dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

     ✅ agentmodel: PASSED: MIGraphX meets tolerance

     ✅ unet: PASSED: MIGraphX meets tolerance

     ✅ resnet50v1: PASSED: MIGraphX meets tolerance

     ✅ bert_base_cased_fp16: PASSED: MIGraphX meets tolerance

🔴bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output


     ✅ bert_large: PASSED: MIGraphX meets tolerance

     ✅ yolov5s: PASSED: MIGraphX meets tolerance

     ✅ tinyllama: PASSED: MIGraphX meets tolerance

     ✅ vicuna-fastchat: PASSED: MIGraphX meets tolerance

     ✅ whisper-tiny-encoder: PASSED: MIGraphX meets tolerance

     ✅ whisper-tiny-decoder: PASSED: MIGraphX meets tolerance

     ✅ distilgpt2_fp16: PASSED: MIGraphX meets tolerance

     ✅ llama2_7b: PASSED: MIGraphX meets tolerance

     ✅ qwen1.5-7b: PASSED: MIGraphX meets tolerance

     ✅ phi3-3.8b: PASSED: MIGraphX meets tolerance

🔴mask-rcnn: FAILED: MIGraphX is not within tolerance - check verbose output


     ✅ llama3-8b: PASSED: MIGraphX meets tolerance

     ✅ whisper-large-decoder: PASSED: MIGraphX meets tolerance

     ✅ mistral-7b: PASSED: MIGraphX meets tolerance

     ✅ FLUX.1-schnell: PASSED: MIGraphX meets tolerance

@CharlieL7 CharlieL7 removed their request for review September 16, 2025 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Fixes a bug found in the code. Perf Improve simple small or simple changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants