[DRAFT] Generalize MHA pattern

[gramalingam]

Generalize the MHA pattern (motivated by the Phi models). Specifically, we remove the initial MatMuls from the pattern (as being unnecessary). Phi uses packed MatMul (Q, K, and V are multiplied using a single MatMul and then sliced).

However, this is not sufficient yet, since Phi also uses partial rotary-embedding, which is not yet supported by the RotaryEmbedding pattern. I will separately work on the extension to the RotaryEmbedding pattern to handle partial embedding.

To fix the problem, we need to either remove the commented-out code or reinstate it if it is necessary for the functionality. Given the presence of TODO comments, it is likely that the commented-out code was intended to be revisited and possibly reinstated. Therefore, the best approach is to reinstate the commented-out code and ensure that it is functional.

• Reinstate the commented-out code on lines 161-162 and 173-178.
• Ensure that the reinstated code is properly integrated and does not cause any issues.

To fix the problem, we need to ensure that original_outputs is initialized before it is used. One way to achieve this is to initialize original_outputs to None before the conditional block and then check if it is None before using it. This ensures that original_outputs is always defined, and we can handle the case where it is not set due to the condition not being met.

[codecov]

❌ 83 Tests Failed:

Tests completed	Failed	Passed	Skipped
9647	83	9564	1946

View the top 2 failed test(s) by shortest run time

onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_0945_test_reshape_reordered_last_dims

Stack Traces | 0.003s run time

onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_reshape_reordered_last_dims'

The above exception was the direct cause of the following exception:
.nox\test_ort_nightly\Lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_reshape_reordered_last_dims' (e=No module named 'tests.onnx_backend_test_code.test_reshape_reordered_last_dims') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_reshape_reordered_last_dims.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_reshape_reordered_last_dims.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import FLOAT, INT64
E   from onnxscript.onnx_opset import opset21
E   
E   @script()
E   def bck_test_reshape_reordered_last_dims(data: FLOAT[2,3,4], shape: INT64[3]) -> (FLOAT[2,4,3]):
E       reshaped = opset21.Reshape(data, shape)
E       return reshaped

onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_1244_test_transpose_all_permutations_5

Stack Traces | 0.003s run time

onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_transpose_all_permutations_5'

The above exception was the direct cause of the following exception:
.nox\test\lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_transpose_all_permutations_5' (e=No module named 'tests.onnx_backend_test_code.test_transpose_all_permutations_5') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_transpose_all_permutations_5.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_transpose_all_permutations_5.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import FLOAT
E   from onnxscript.onnx_opset import opset21
E   
E   @script()
E   def bck_test_transpose_all_permutations_5(data: FLOAT[2,3,4]) -> (FLOAT[4,3,2]):
E       transposed = opset21.Transpose(data, perm=[2, 1, 0])
E       return transposed

View the full list of 1 ❄️ flaky tests

onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_1246_test_tril

Flake rate in main: 40.00% (Passed 3 times, Failed 2 times)

Stack Traces | 0.003s run time

onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_tril'

The above exception was the direct cause of the following exception:
.nox\test\Lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_tril' (e=No module named 'tests.onnx_backend_test_code.test_tril') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_tril.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_tril.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import INT64
E   from onnxscript.onnx_opset import opset14
E   
E   @script()
E   def bck_test_tril(x: INT64[4,5]) -> (INT64[4,5]):
E       y = opset14.Trilu(x, upper=0)
E       return y

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[justinchuby]

Can this be absorbed into return shape[dim]?

[justinchuby]

Is this used? If so

Suggested change

	ort_version = packaging.version.Version(onnxruntime.__version__)
	_ORT_VERSION = packaging.version.Version(onnxruntime.__version__)