Integrate new CK API and operations with MIGraphX

[mirza-halilcevic]

• Resolves Integrate new CK API and operations with MIGraphX migraphx-benchmark/AMDMIGraphX#197

[migraphx-bot]

Test	Batch	Rate new cc546c	Rate old 3116c7	Diff	Compare
torchvision-resnet50	64	3,078.40	3,232.24	-4.76%	🔴
torchvision-resnet50_fp16	64	6,934.42	6,901.97	0.47%	✅
torchvision-densenet121	32	2,415.02	2,441.26	-1.07%	✅
torchvision-densenet121_fp16	32	4,188.64	4,183.95	0.11%	✅
torchvision-inceptionv3	32	1,636.98	1,628.18	0.54%	✅
torchvision-inceptionv3_fp16	32	2,748.03	2,747.37	0.02%	✅
cadene-inceptionv4	16	771.76	766.97	0.63%	✅
cadene-resnext64x4	16	813.67	813.79	-0.01%	✅
slim-mobilenet	64	7,466.53	7,430.84	0.48%	✅
slim-nasnetalarge	64	211.01	210.03	0.47%	✅
slim-resnet50v2	64	3,333.37	3,325.76	0.23%	✅
bert-mrpc-onnx	8	1,147.70	1,143.14	0.40%	✅
bert-mrpc-tf	1	457.95	458.44	-0.11%	✅
pytorch-examples-wlang-gru	1	344.44	343.86	0.17%	✅
pytorch-examples-wlang-lstm	1	439.53	474.60	-7.39%	🔴
torchvision-resnet50_1	1	790.31	789.53	0.10%	✅
cadene-dpn92_1	1	415.97	419.72	-0.89%	✅
cadene-resnext101_1	1	393.07	387.56	1.42%	✅
onnx-taau-downsample	1	396.03	394.23	0.46%	✅
dlrm-criteoterabyte	1	33.77	33.67	0.30%	✅
dlrm-criteoterabyte_fp16	1	51.19	51.07	0.24%	✅
agentmodel	1	10,251.35	10,510.49	-2.47%	✅
unet_fp16	2	58.08	60.59	-4.14%	🔴
resnet50v1_fp16	1	1,034.87	1,045.09	-0.98%	✅
resnet50v1_int8	1	1,057.09	1,069.22	-1.13%	✅
bert_base_cased_fp16	64	1,170.23	1,163.30	0.60%	✅
bert_large_uncased_fp16	32	361.37	359.63	0.49%	✅
bert_large_fp16	1	203.56	202.87	0.34%	✅
distilgpt2_fp16	16	2,241.45	2,229.71	0.53%	✅
yolov5s	1	539.13	544.90	-1.06%	✅
tinyllama	1	43.99	43.85	0.33%	✅
vicuna-fastchat	1	45.29	44.97	0.72%	✅
whisper-tiny-encoder	1	419.16	417.62	0.37%	✅
whisper-tiny-decoder	1	412.68	401.82	2.70%	✅
llama2_7b	1	19.18	19.13	0.28%	✅
qwen1.5-7b	1	23.65	23.53	0.49%	✅
phi3-3.8b	1	26.85	26.75	0.37%	✅
mask-rcnn	1	12.77	12.80	-0.23%	✅
llama3-8b	1	21.84	21.76	0.37%	✅
whisper-large-encoder	1	10.22	10.17	0.47%	✅
whisper-large-decoder	1	103.79	104.54	-0.71%	✅
mistral-7b	1	23.85	23.78	0.29%	✅
FLUX.1-schnell	1	770.42	771.60	-0.15%	✅
nan	nan	nan	nan	nan%	❌

This build is not recommended to merge 🔴

[migraphx-bot]

     ✅ bert-mrpc-onnx: PASSED: MIGraphX meets tolerance

❌bert-mrpc-tf: ERROR - check error output

error: unknown warning option '-Wnrvo' [-Werror,-Wunknown-warning-option]

error: unknown warning option '-Wnrvo' [-Werror,-Wunknown-warning-option]

2025-07-16 07:36:57.534346: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1752669422.870461 179142 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 62973 MB memory: -> device: 0, name: AMD Instinct MI250X/MI250, pci bus id: 0000:b3:00.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1752669423.750480 179142 mlir_graph_optimization_pass.cc:401] MLIR V1 optimization pass is not enabled
2025-07-16 07:37:14.669740: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-16 07:37:14.669983: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-16 07:37:14.670038: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-16 07:37:14.670085: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-16 07:37:14.670114: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-16 07:37:14.670166: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-16 07:37:14.670209: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-16 07:37:14.670263: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
2025-07-16 07:37:14.671449: E tensorflow/compiler/mlir/tools/kernel_gen/tf_framework_c_interface.cc:228] INTERNAL: Generating device code failed.
2025-07-16 07:37:14.672645: W tensorflow/core/framework/op_kernel.cc:1829] UNKNOWN: JIT compilation failed.
2025-07-16 07:37:14.672666: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
2025-07-16 07:37:14.672678: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
2025-07-16 07:37:14.672693: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 11217777527359497193
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1407, in _do_call
return fn(*args)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1390, in _run_fn
return self._call_tf_sessionrun(options, feed_dict, fetch_list,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1483, in _call_tf_sessionrun
return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 359, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 335, in main
y_out = sess.run(y, feed_dict=tf_dict)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 977, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1220, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1400, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1426, in _do_call
raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.UnknownError: Graph execution error:

Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'import/bert/embeddings/LayerNorm/moments/SquaredDifference':

     ✅ pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

     ✅ pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

     ✅ dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

     ✅ agentmodel: PASSED: MIGraphX meets tolerance

🔴unet: FAILED: MIGraphX is not within tolerance - check verbose output

     ✅ resnet50v1: PASSED: MIGraphX meets tolerance

     ✅ bert_base_cased_fp16: PASSED: MIGraphX meets tolerance

🔴bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output

     ✅ bert_large: PASSED: MIGraphX meets tolerance

     ✅ yolov5s: PASSED: MIGraphX meets tolerance

     ✅ tinyllama: PASSED: MIGraphX meets tolerance

     ✅ vicuna-fastchat: PASSED: MIGraphX meets tolerance

     ✅ whisper-tiny-encoder: PASSED: MIGraphX meets tolerance

     ✅ whisper-tiny-decoder: PASSED: MIGraphX meets tolerance

     ✅ distilgpt2_fp16: PASSED: MIGraphX meets tolerance

     ✅ llama2_7b: PASSED: MIGraphX meets tolerance

     ✅ qwen1.5-7b: PASSED: MIGraphX meets tolerance

     ✅ phi3-3.8b: PASSED: MIGraphX meets tolerance

🔴mask-rcnn: FAILED: MIGraphX is not within tolerance - check verbose output

     ✅ llama3-8b: PASSED: MIGraphX meets tolerance

     ✅ whisper-large-decoder: PASSED: MIGraphX meets tolerance

     ✅ mistral-7b: PASSED: MIGraphX meets tolerance

     ✅ FLUX.1-schnell: PASSED: MIGraphX meets tolerance

[mirza-halilcevic]

Moved to #4293