[WIP] Out-Tree EP feature #21450

jslhcl · 2024-07-23T00:36:08Z

Description

Out-Tree EP feature.

Motivation and Context

Decouple ExecutionProvider from ONNXRuntime and make it a standalone class which will benefit 3rd party EP authors to write their own EP.
Binary compatibility is required for this feature.

samples/outTreeEp/out_tree_ep.cc

samples/c_test/test.cpp

samples/outTreeEp/out_tree_ep.h

@@ -0,0 +1,35 @@
+#pragma once


onnxruntime/core/framework/provider_adapter.h

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/c_test/test.cpp

onnxruntime/core/session/inference_session.cc

jslhcl · 2024-07-23T22:11:31Z

samples/outTreeEp/out_tree_ep.cc

+#ifdef __cplusplus
+extern "C" {
+#endif
+OrtExecutionProviderFactory* RegisterCustomEp() {


return Status instead #Resolved

Do we have to do this? This function will new a factory object by invoking its constructor which has no return type

… EP as graph API is not exported by ORT. Need to put these graph API into ortapi structure

samples/outTreeEp/out_tree_ep.cc

samples/c_test/test.cpp

…roviderAdapter::Compile()

adrianlizarraga · 2024-07-30T21:24:49Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+} OrtMetaDef;
+
+typedef struct OrtIndexedSubGraph {
+  OrtMetaDef* meta_def; // TODO(leca): how to define a nested structure pointer?


Does this have to be a pointer to an OrtMetaDef? It may be simpler if this meta_def is contained by value instead. #Resolved

It looks we will check the pointer is null or not to distinguish between single node mode and fused node mode (See base class IExecutionProvider::GetCapability() which does not set this pointer and TryAssignSingleNode() which will check this pointer)

adrianlizarraga · 2024-07-30T21:34:37Z

samples/outTreeEp/out_tree_ep.cc

+
+OutTreeEp::OutTreeEp(const char* ep_type, const OutTreeEpInfo& ep_info) : info(ep_info) {
+    type = ep_type;
+    OrtExecutionProvider::GetCapability = [](const OrtExecutionProvider* this_, const OrtGraphViewer* graph, size_t* cnt, OrtIndexedSubGraph*** indexed_sub_graph) {


If I'm understanding correctly, the type of the OrtIndexedSubGraph*** indexed_sub_graph parameter is essentially asking the EP to fill out an array of pointers to OrtIndexedSubGraph objects.

Would it be simpler to change this to OrtIndexedSubgraph** indexed_sub_graph so that the EP fills out an array of OrtIndexedSubGraph objects directly? Each OrtIndexedSubgraph struct is a simple POD that can be created on the stack and copied around. It seems like it would result in less pointer tracking. #Resolved

Also, who is responsible for freeing this memory and when? If the EP allocates an array, then the EP should free it. The currently example leaks the allocations.

Edit: one possibility is to have onnxruntime call a new EP function (e.g., ReleaseOrtIndexedSubGraph()) so the the EP can free the memory. onnxruntime would call this once it is done using the indexed_sub_graph.

The problem is that we don't know how many OrtIndexedSubGraph would be before we call GetCapability() function. I will fix the leak issue in the coming commits

samples/outTreeEp/out_tree_ep.cc

onnxruntime/core/framework/provider_factory_adapter.h

@@ -0,0 +1,21 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/outTreeEp_kernel/kernel_ep.cc

@@ -0,0 +1,89 @@
+#include "kernel_ep.h"


samples/outTreeEp_kernel/kernel_ep.h

@@ -0,0 +1,36 @@
+#pragma once


include/onnxruntime/core/framework/ort_type_constraints.h

@@ -0,0 +1,15 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


onnxruntime/core/framework/ort_type_constraints.cc

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_model_wrapper.cc

@@ -0,0 +1,627 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_model_wrapper.h

@@ -0,0 +1,285 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_quant_params_wrapper.cc

@@ -0,0 +1,266 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_quant_params_wrapper.h

@@ -0,0 +1,147 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_utils.cc

@@ -0,0 +1,557 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_utils.h

@@ -0,0 +1,110 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/qnn_execution_provider.cc

@@ -0,0 +1,54 @@
+#include "qnn_execution_provider.h"


samples/qnnEp/qnn_execution_provider.h

@@ -0,0 +1,33 @@
+#pragma once


…22481)

Add some utility files for plugin ep to include and compile. - provider option map -> `provider_option`.h - provider option parser -> `provider_option_utils`.h - some macro define, classes and functions from include/onnxruntime/core/common

…untime into leca/outOfTreeEP

Hook TRT EP plugin to run the existing unit test in CI - Migrate from `onnxruntime/test/providers/tensorrt/tensorrt_basic_test.cc` - Replace internal APIs with new EP APIs - Add unit test in `onnxruntime_shared_lib_test` (which links against onnxruntime dll) - Build ORT with `--test_tensorrt_ep_plugin` to run `onnxruntime_shared_lib_test` Note: The unit test doesn't cover all the cases since current TRT EP plugin hasn't added all the features yet, will update later.

…untime into HEAD

…e attribute might contain null character (#22769) When running EP Context model, EP might call `OrtGraphApis::OrtNode_GetAttributeStr` to get the string-based content of the attribute. However, the API returns the c_str() of the string, and it's possible that the cache context contains null character, so the string might be cut off and caller ends up getting the wrong string. Add a new OrtGraphApis::OrtNode_GetAttributeStrWithSize to return const char* pointer and string size.

This PR support several features: - Add new graph API to create and update EP Context graph, and dump EP Context model. 1. OrtGraph_CreateOrUpdateEpCtxGraph 2. OrtGraph_DumpOnnxModel 3. OrtGraph_ReleaseGraph - Add new graph API to dump onnx model - The APIs provided by this PR can dump EP Context model when the whole model can be run by one EP, the APIs also aim to support the case where the whole model is partitioned into multiple EP's subgraphs. (Note: i haven't fully tested the partitioning case, please help review it) - Modify TRT EP plugin to use those APIs.

…ed case

TRT 8 doesn't support INT64 and DOUBLE data type. TRT 10 doesn't support DOUBLE data type. Therefore, TRT EP internally needs to convert INT64 to INT32, and DOUBLE to FLOAT, which needs the cuda::Impl_Cast function. The implementation is copied from CUDA EP.

jslhcl added 2 commits July 17, 2024 20:39

opaque pointer for graph

0e6a80c

ORT C API RegisterOrtExecutionProviderLibrary work

c30a639

jslhcl requested review from souptc, adrianlizarraga, jywu-msft and chilo-ms July 23, 2024 00:36

github-advanced-security bot found potential problems Jul 23, 2024

View reviewed changes

ORT C-API SessionOptionsAppendOrtExecutionProvider work

7bfe57e

github-advanced-security bot found potential problems Jul 23, 2024

View reviewed changes

jslhcl commented Jul 23, 2024

View reviewed changes

onnxruntime/core/session/inference_session.cc Outdated Show resolved Hide resolved

jslhcl commented Jul 23, 2024

View reviewed changes

Test Relu with compile based EP, build work, runtime error of loading…

8e7d28d

… EP as graph API is not exported by ORT. Need to put these graph API into ortapi structure

github-advanced-security bot found potential problems Jul 26, 2024

View reviewed changes

samples/outTreeEp/out_tree_ep.cc Fixed Show fixed Hide fixed

samples/c_test/test.cpp Fixed Show fixed Hide fixed

jslhcl added 2 commits July 29, 2024 17:49

prototype works with hardcode node_compute_info's index in ExecutionP…

808bfc3

…roviderAdapter::Compile()

prototype works without hardcode

49e396c

adrianlizarraga reviewed Jul 30, 2024

View reviewed changes

samples/outTreeEp/out_tree_ep.cc Outdated Show resolved Hide resolved

jslhcl added 2 commits July 31, 2024 20:38

fix comments for Compile function

e790105

add provider_factory_adapter.h

92f529d

github-advanced-security bot found potential problems Aug 1, 2024

View reviewed changes

onnxruntime/core/framework/provider_factory_adapter.h

@@ -0,0 +1,21 @@

// Copyright (c) Microsoft Corporation. All rights reserved.

Check warning

Code scanning / lintrunner

CLANGFORMAT/format Warning

See https://clang.llvm.org/docs/ClangFormat.html.
Run lintrunner -a to apply this patch.

jslhcl added 2 commits August 5, 2024 19:27

fix crash after introducing kernel based EP

3d83ed1

kernel based EP work with type constraint check commented out

e29499a

github-advanced-security bot found potential problems Aug 6, 2024

View reviewed changes

add kernel type constraints from out tree EP

f3678c4

github-advanced-security bot found potential problems Aug 7, 2024

View reviewed changes

jslhcl added 2 commits August 7, 2024 16:27

add API ReleaseOrtTypeConstraints

ac5ae0a

introduce qnn ep

0cc78e8

github-advanced-security bot found potential problems Aug 12, 2024

View reviewed changes

more graph/node C API

740a687

jslhcl and others added 30 commits October 14, 2024 22:15

Python API

b1f8e2a

fix memory leak (#22444)

7acaaab

refactor all functions in onnxruntime_c_api_ep with status as return (#…

d150a03

…22481)

resolve comments

da5b6eb

add documents for all functions in c_api_ep (#22502)

d280e59

fix comments

cbe98e7

fix memory leak (#22522)

1529059

add mutex to plugin trt ep (#22581)

fa549f8

use std::mutex instead of OrtMutex and fix build error in Windows

a28ad38

openvino

aa49805

openvino, GetCapability almost ready

bc65613

openvino GetCapacity() is done. UnregisterPluginExecutionProviderLibrary

a1a3eea

refine compile of openvino ep (#22689)

0fe5f01

Add utility files (#22650)

6bae1b9

Add some utility files for plugin ep to include and compile. - provider option map -> `provider_option`.h - provider option parser -> `provider_option_utils`.h - some macro define, classes and functions from include/onnxruntime/core/common

OpenVino, compile() is done

ab75d98

Merge branch 'leca/outOfTreeEP' of https://github.com/microsoft/onnxr…

c5510f2

…untime into leca/outOfTreeEP

add test for openvino plugin ep and fix bugs (#22734)

b0b3123

add missing mutex to plugin trt ep

9dbb0b1

merge code

5a59803

Merge branch 'leca/outOfTreeEP' of https://github.com/microsoft/onnxr…

999e7fd

…untime into HEAD

fix bugs (#22744)

084f735

relu and resnet works in OpenVINO plugin

2b1cfdf

[TensorRT EP Plugin] use new graph api for ep context model generation

63f8774

use cuda's preferred allocator for plugin trt and builtin cuda combin…

bf359a1

…ed case

fix build/compiler error for nvcc 11.8

72afdc4

Do not expose OrtGraph

6822206

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Out-Tree EP feature #21450

[WIP] Out-Tree EP feature #21450

jslhcl commented Jul 23, 2024 •

edited

Loading

jslhcl Jul 23, 2024 •

edited

Loading

jslhcl Oct 18, 2024

adrianlizarraga Jul 30, 2024 •

edited by jslhcl

Loading

jslhcl Jul 30, 2024

adrianlizarraga Jul 30, 2024 •

edited by jslhcl

Loading

adrianlizarraga Jul 30, 2024 •

edited

Loading

jslhcl Jul 31, 2024 •

edited

Loading

		@@ -0,0 +1,14 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,21 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,15 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,627 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,285 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,266 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,147 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,557 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,110 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

[WIP] Out-Tree EP feature #21450

Are you sure you want to change the base?

[WIP] Out-Tree EP feature #21450

Conversation

jslhcl commented Jul 23, 2024 • edited Loading

Description

Motivation and Context

jslhcl Jul 23, 2024 • edited Loading

Choose a reason for hiding this comment

jslhcl Oct 18, 2024

Choose a reason for hiding this comment

adrianlizarraga Jul 30, 2024 • edited by jslhcl Loading

Choose a reason for hiding this comment

jslhcl Jul 30, 2024

Choose a reason for hiding this comment

adrianlizarraga Jul 30, 2024 • edited by jslhcl Loading

Choose a reason for hiding this comment

adrianlizarraga Jul 30, 2024 • edited Loading

Choose a reason for hiding this comment

jslhcl Jul 31, 2024 • edited Loading

Choose a reason for hiding this comment

jslhcl commented Jul 23, 2024 •

edited

Loading

jslhcl Jul 23, 2024 •

edited

Loading

adrianlizarraga Jul 30, 2024 •

edited by jslhcl

Loading

adrianlizarraga Jul 30, 2024 •

edited by jslhcl

Loading

adrianlizarraga Jul 30, 2024 •

edited

Loading

jslhcl Jul 31, 2024 •

edited

Loading