Modify RunONNXModel.py for pip install #2857

chentong319 · 2024-06-27T15:57:02Z

The goal is to create a python package so that we can run an inference like onnxruntime in python script. Since the RunONNXModel.py provides most of the necessary functionality, I added extra support in that script. I tested with a local pip install. We can upload the package later.

Major changes:

The code for package, onnxmlir, is added in utils.
The main code is modified from utils/RunONNXModel.py. Therefore, we can run inference in onnxruntime way, or our RunONNXModel.py way.
Related document is added in docs.
Details of the package can be found at utils/onnxmlir/README.md, and example can be found in utils/onnxmlir/tests

Example:

import numpy as np
import onnxmlirrun

a = np.zeros((3,4,5), dtype=np.float32)
b = a + 4
r = onnxmlirrun.RunONNXModel.onnxmlirrun(compiled_so="/Users/chentong/Projects/onnx-mlir/build/test_add.so", inputs=[a,b])
print(r)
r = onnxmlirrun.RunONNXModel.onnxmlirrun(onnx_model="/Users/chentong/Projects/onnx-mlir/build/test_add.onnx", inputs=[a,b])
print(r)

We may have different name for the functions and the package. Comments are welcome.

Signed-off-by: Chen Tong <[email protected]>

Signed-off-by: Tong Chen <[email protected]>

Signed-off-by: chentong319 <[email protected]>

This reverts commit ebd27e2.

This reverts commit 432e4be.

Signed-off-by: chentong319 <[email protected]>

AlexandreEichenberger · 2024-06-27T16:04:10Z

@chentong319 Can you add a reference to the ORT interface that this PR is imitating.

In your first example

r = onnxmlirrun.RunONNXModel.onnxmlirrun(compiled_so="/Users/chentong/Projects/onnx-mlir/build/test_add.so", inputs=[a,b])

was the onnx_model parameter omitted on purpose? Just for me to understand better the interface.

chentong319 · 2024-06-27T18:29:01Z

@chentong319 Can you add a reference to the ORT interface that this PR is imitating.

In your first example
r = onnxmlirrun.RunONNXModel.onnxmlirrun(compiled_so="/Users/chentong/Projects/onnx-mlir/build/test_add.so", inputs=[a,b])
was the onnx_model parameter omitted on purpose? Just for me to understand better the interface.

Yes, onnx_model is not needed if the compiled .so is provided. By the way, the two onnxmilrrun.RunONNXModel.onnxmlirrun commands are two independent ways to do inference.

AlexandreEichenberger · 2024-06-27T18:54:34Z

Got it, so we can provide a onnx file or a pre-compiled binary, smart, thanks.

tungld · 2024-07-01T07:07:06Z

utils/RunONNXModel.py

+    if onnx_model :
+        args.model = onnx_model
+    if compiled_so :
+        args.load_so = compiled_so


Should we check if both onnx_model and compiled_so are not given?

Make sense, and maybe if none are given, an error?

I added the check whether both onnx_mdel and compiled_so are given or not give (XOR).

tungld · 2024-07-01T07:15:26Z

onnxmlirrun.RunONNXModel.onnxmlirrun

My two cents here. For the package name (the first onnxmlirrun), just onnxmlir is enough to avoid typing two Rs.
I would remove RunONNXModel, but it seems not straightforward unless we reorganize RunONNXModel.py.

Anyway, we have several python utilities, perhaps it's time to reorganize them into a single python package.

Signed-off-by: chentong319 <[email protected]>

chentong319 · 2024-07-18T19:55:01Z

I changed the init.py for the package to simplify onnxmlirrun.RunONNXModel.onnxmlirrun to onnxmlir.onnxmlir.
I am thinking to add the code for package into onnx-mlir in future.

AlexandreEichenberger · 2024-07-18T19:57:53Z

Can you copy/link a description of the ORT runtime interface and update ours with the new names, so that we can see how they relate?

Is it something on this page? https://onnxruntime.ai/docs/get-started/with-python.html

chentong319 · 2024-07-20T19:40:36Z

The Ort code is like

import onnxruntime as ort
import numpy as np
x, y = test_data[0][0], test_data[0][1]
ort_sess = ort.InferenceSession('fashion_mnist_model.onnx')
outputs = ort_sess.run(None, {'input': x.numpy()})

Since the current RunModelPython does not separate the session creation and run, our code will be like:

import onnxmlir
import numpy as np
x, y = test_data[0][0], test_data[0][1]
outputs = onnxmlir.run(onnx_model='fashion_mnist_model.onnx', inputs = [a, b])

It is doable to totally mimic ORT in separating the session creation and run, and using dictionary for input.

AlexandreEichenberger · 2024-07-23T15:38:23Z

Ideally

import onnxmlir as ort
import numpy as np
x, y = test_data[0][0], test_data[0][1]
ort_sess = ort.InferenceSession('fashion_mnist_model.onnx')
outputs = ort_sess.run(None, {'input': x.numpy()})

with additional parameters ok

import onnxruntime as ort
import numpy as np
x, y = test_data[0][0], test_data[0][1]
ort_sess = ort.InferenceSession('fashion_mnist_model.onnx', compile_flags="-O3")
outputs = ort_sess.run(None, {'input': x.numpy()})

reusing flags that already exists, if any.

Signed-off-by: chentong319 <[email protected]>

tungld · 2024-08-06T06:59:11Z

utils/RunONNXModel.py

+
+
+def inferenceSession(model_name, **kwargs):
+    global args


Can we move this code into the __init__ of InferenceSession class and remove inferenceSession method? The reason is it is a bit confusing about InferenceSession class and inferenceSession method. Other parts looks good to me.

Fixed. Thanks for the suggestion.

Signed-off-by: chentong319 <[email protected]>

AlexandreEichenberger · 2024-08-06T15:13:47Z

utils/onnxmlir/VERSION_NUMBER

@@ -0,0 +1 @@
+1.0.0


Version number here is different than onnx-mlir/VERSION_NUMBER. Is that on purpose?
Also the number here is different than the version listed in the file below.

Fixed. I used a version number different from the onnx-mlir. The version for onnxmlir is for upstream onnxmlir to pip repo. It could be viewed as a pre-required package.

May I suggest a 0.1.0 and move to a 1.0.0 once we have the installation of the compiler with it at least one z?

AlexandreEichenberger · 2024-08-06T15:33:43Z

utils/onnxmlir/src/onnxmlir/RunONNXModel.py

@@ -0,0 +1,959 @@
+#!/usr/bin/env python3


Is this an identical copy? If so, at a minimum there should be a comment stating that they should stay in sync. Or maybe we could have a single version and we could "build" the pip structure by copying files in the right place. That would be my preference.

Added a target in CMake to check the files.

AlexandreEichenberger

Check error message if file does not exist.

AlexandreEichenberger

TODO: let's try to have parallel default and/or as set in onnxruntime.

AlexandreEichenberger

TODO: sync also with #2899 with respect to catching errors/crasshes

Signed-off-by: chentong319 <[email protected]>

chentong319 · 2024-08-06T18:40:12Z

Check error message if file does not exist.

onnx-mlir will report the error if file does not exist. Leave this to the TODO list to catch all the errors gracefully.

Signed-off-by: chentong319 <[email protected]>

AlexandreEichenberger · 2024-08-07T14:18:53Z

utils/onnxmlir/VERSION_NUMBER

@@ -0,0 +1 @@
+1.0.0


May I suggest a 0.1.0 and move to a 1.0.0 once we have the installation of the compiler with it at least one z?

AlexandreEichenberger · 2024-08-07T14:20:10Z

utils/onnxmlir/pyproject.toml

+build-backend = "hatchling.build"
+[project]
+name = "onnxmlir"
+version = "1.0.0"


Thanks for having it in sync.

Signed-off-by: chentong319 <[email protected]>

chentong319 added 20 commits February 5, 2024 21:08

parallel for

569a4d4

Signed-off-by: Chen Tong <[email protected]>

run through

9f8d503

Signed-off-by: Tong Chen <[email protected]>

Merge remote-tracking branch 'upstream/main' into parallel-matmul

554edb3

flag

432e4be

Signed-off-by: Tong Chen <[email protected]>

format

ebd27e2

Signed-off-by: chentong319 <[email protected]>

Revert "format"

7a4cea3

This reverts commit ebd27e2.

Revert "flag"

38c02d4

This reverts commit 432e4be.

Merge remote-tracking branch 'upstream/main'

df3a68f

fix merge error

42d0890

Signed-off-by: chentong319 <[email protected]>

Merge remote-tracking branch 'upstream/main'

7c88851

Merge remote-tracking branch 'upstream/main'

ddefa90

Merge remote-tracking branch 'upstream/main'

e20ce14

Merge remote-tracking branch 'upstream/main'

67cf6fe

Merge remote-tracking branch 'upstream/main'

88a0358

Merge remote-tracking branch 'upstream/main'

9d183a2

Merge remote-tracking branch 'upstream/main'

60cfc2d

Merge remote-tracking branch 'upstream/main'

1820af3

Merge remote-tracking branch 'upstream/main'

a6d7a28

Merge remote-tracking branch 'upstream/main'

bea2e7d

implementation

5f00fa1

Signed-off-by: chentong319 <[email protected]>

tungld reviewed Jul 1, 2024

View reviewed changes

chentong319 added 5 commits July 16, 2024 13:39

Merge remote-tracking branch 'upstream/main' into pip-1

a80e816

add check

42fde3c

Signed-off-by: chentong319 <[email protected]>

Merge remote-tracking branch 'upstream/main' into pip-1

87d1b55

format

2cf740c

Signed-off-by: chentong319 <[email protected]>

interface

1012ce3

Signed-off-by: chentong319 <[email protected]>

chentong319 added 5 commits July 23, 2024 13:17

Merge remote-tracking branch 'upstream/main' into pip-1

96f1c5b

add package

711bd02

Signed-off-by: chentong319 <[email protected]>

format

344fecf

Signed-off-by: chentong319 <[email protected]>

Merge remote-tracking branch 'upstream/main' into pip-1

367c4a1

authors

863a05e

Signed-off-by: chentong319 <[email protected]>

chentong319 requested review from tungld and AlexandreEichenberger August 6, 2024 00:21

tungld reviewed Aug 6, 2024

View reviewed changes

tungld and others added 2 commits August 6, 2024 16:40

Merge branch 'main' into pip-1

54fb499

fix

ad29d90

Signed-off-by: chentong319 <[email protected]>

AlexandreEichenberger reviewed Aug 6, 2024

View reviewed changes

fix

bd03582

Signed-off-by: chentong319 <[email protected]>

chentong319 added 2 commits August 6, 2024 19:44

Merge remote-tracking branch 'upstream/main' into pip-1

3fbf63b

fix typo

8478f5a

Signed-off-by: chentong319 <[email protected]>

AlexandreEichenberger reviewed Aug 7, 2024

View reviewed changes

chentong319 added 3 commits August 7, 2024 10:59

check input name

20a8abd

Signed-off-by: chentong319 <[email protected]>

Merge remote-tracking branch 'upstream/main' into pip-1

afdf7de

format

b13e7bc

Signed-off-by: chentong319 <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modify RunONNXModel.py for pip install #2857

Modify RunONNXModel.py for pip install #2857

chentong319 commented Jun 27, 2024 •

edited

Loading

AlexandreEichenberger commented Jun 27, 2024

chentong319 commented Jun 27, 2024

AlexandreEichenberger commented Jun 27, 2024

tungld Jul 1, 2024

AlexandreEichenberger Jul 1, 2024

chentong319 Jul 18, 2024

tungld commented Jul 1, 2024

chentong319 commented Jul 18, 2024

AlexandreEichenberger commented Jul 18, 2024 •

edited

Loading

chentong319 commented Jul 20, 2024

AlexandreEichenberger commented Jul 23, 2024

tungld Aug 6, 2024

chentong319 Aug 6, 2024

AlexandreEichenberger Aug 6, 2024

chentong319 Aug 6, 2024

AlexandreEichenberger Aug 7, 2024

AlexandreEichenberger Aug 6, 2024

chentong319 Aug 6, 2024

AlexandreEichenberger left a comment

AlexandreEichenberger left a comment

AlexandreEichenberger left a comment

chentong319 commented Aug 6, 2024

AlexandreEichenberger Aug 7, 2024

AlexandreEichenberger Aug 7, 2024

		@@ -0,0 +1 @@
		1.0.0

		@@ -0,0 +1 @@
		1.0.0

Modify RunONNXModel.py for pip install #2857

Are you sure you want to change the base?

Modify RunONNXModel.py for pip install #2857

Conversation

chentong319 commented Jun 27, 2024 • edited Loading

AlexandreEichenberger commented Jun 27, 2024

chentong319 commented Jun 27, 2024

AlexandreEichenberger commented Jun 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tungld commented Jul 1, 2024

chentong319 commented Jul 18, 2024

AlexandreEichenberger commented Jul 18, 2024 • edited Loading

chentong319 commented Jul 20, 2024

AlexandreEichenberger commented Jul 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexandreEichenberger left a comment

Choose a reason for hiding this comment

AlexandreEichenberger left a comment

Choose a reason for hiding this comment

AlexandreEichenberger left a comment

Choose a reason for hiding this comment

chentong319 commented Aug 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chentong319 commented Jun 27, 2024 •

edited

Loading

AlexandreEichenberger commented Jul 18, 2024 •

edited

Loading