[EP ABI] Add CreateCustomOpDomains() API for plugin EP to register custom ops #26759

chilo-ms · 2025-12-09T06:42:03Z

Description

The newly added two APIs, CreateCustomOpDomains() and GetNumCustomOpDomains, are used when running inference on a model that contains EP-specific custom operations.

Workflow:

The EP implements these functions to supply a list of OrtCustomOpDomain instances.
The application calls SessionOptionsAppendExecutionProvider_V2() with the OrtEpDevice containing
the plugin EP's factory.
SessionOptionsAppendExecutionProvider_V2() internally appends the provided OrtCustomOpDomain list to the
session options.

As a result, any session created from these session options will have these custom op domains registered
in ORT, ensuring that the custom ops are properly recognized and validated when the model is loaded.

Plugin EPs can provide two types of custom ops:

A full OrtCustomOp with a concrete kernel implementation
- This Example EP demonstrates this approach.
- In GetCapability(), it calls EpGraphSupportInfo_AddSingleNode() to inform ORT
  that the custom node should NOT be fused or compiled. Instead, ORT should invoke
  the custom node's Compute() function at runtime.
A "placeholder" OrtCustomOp with an empty kernel implementation
- A compile-based Plugin EP can supply an OrtCustomOp whose CustomKernel::Compute()
  does nothing. The purpose is to satisfy model validation during model loading by
  registering the custom op as a valid operator in the session.
- In GetCapability(), the EP should call EpGraphSupportInfo_AddNodesToFuse() to
  notify ORT that this custom node should be fused and compiled by the EP.
- In Compile(), the EP executes its compiled bits to perform inference for
  the fused custom node.

Motivation and Context

Currently, the provider-bridge TRT RTX EP and TRT EP supports registering custom op domain list in session option so
that it can run model contains TRT specific custom ops.

This PR adds the same feature for plugin EP.

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

adrianlizarraga · 2025-12-09T19:51:24Z

onnxruntime/core/session/utils.cc

+        return false;
+      };
+
+      if (ep_factory->CreateCustomOpDomains == nullptr) {


ep_factory->CreateCustomOpDomains does not exist for ORT 1.23. Think we need a check here that skips a factory with version older than 24.

Added the check.

onnxruntime/core/session/utils.cc

adrianlizarraga · 2025-12-09T20:00:59Z

onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.cc

+
+/*static*/
+OrtStatus* ORT_API_CALL ExampleEpFactory::CreateCustomOpDomainsImpl(OrtEpFactory* this_ptr,
+                                                                    _Outptr_result_maybenull_ OrtCustomOpDomain** out,


I don't think a OrtCustomOpDomain** parameter allows this function to return more than 1 domain, right?

Maybe it would be easier to have a separate API that returns the number of domains (e.g., OrtEpFactory::GetNumberOfCustomOpDomains()), then ORT allocates an array of the right size, ORT calls OrtEpFactory::CreateCustomOpDomains with the array it allocated, and then the EP factory just fills the array.

The EP is responsible for calling ReleaseCustomOpDomain (happens automatically if using C++ api). ORT is responsible for managing the memory for the array of domains (can just be a std::vector<OrtCustomOpDomain*>)

Good catch!
I update the CreateCustomOpDomains() and added GetNumCustomOpDomains() so that it now can support more than one domain.

adrianlizarraga · 2025-12-09T20:02:59Z

onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.h

+                                                                          compute_stream_(compute_stream) {
+  }
+
+  void* CreateKernel(const OrtApi& /* api */, const OrtKernelInfo* info) const {


where is this called?

I updated the unit test, so now the CreateKernel will be called by ORT.

adrianlizarraga · 2025-12-09T20:03:18Z

onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.h

+  }
+
+  void* CreateKernel(const OrtApi& /* api */, const OrtKernelInfo* info) const {
+    return new PluginEpCustomKernel(info, compute_stream_);


Does this lead to a memory leak?

It won't leak.
ExampleEpCustomOp derives from CustomOpBase so the KernelDestroy will be called during clean up.

struct CustomOpBase : OrtCustomOp { ... OrtCustomOp::KernelDestroy = [](void* op_kernel) { delete static_cast<TKernel*>(op_kernel); }; ...

onnxruntime/test/testdata/custom_mul.py

github-actions

You can commit the suggested changes from lintrunner.

onnxruntime/test/testdata/custom_mul.py

ishwar-raut1 · 2025-12-11T17:34:48Z

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

+   * \since Version 1.24.
+   */
+  ORT_API2_STATUS(CreateCustomOpDomains, _In_ OrtEpFactory* this_ptr,
+                  _Outptr_result_maybenull_ OrtCustomOpDomain** domains, _Out_ size_t num_domains);


OrtCustomOpDomain!
This structure is not ABI safe or stable across boundaries.

The OrtCustomOpDomain is an opaque struct to EP, and EP can create it by calling C API CreateCustomOpDomain() to get a pointer of an OrtCustomOpDomain instance and then add OrtCustomOp instances to it.
I think it's ABI stable as it's just a C pointer.

Please see the implementation in Example EP in the unit test as a reference.

ynankani · 2025-12-12T05:35:13Z

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

+   * \since Version 1.24.
+   */
+  ORT_API2_STATUS(CreateCustomOpDomains, _In_ OrtEpFactory* this_ptr,
+                  _Outptr_result_maybenull_ OrtCustomOpDomain** domains, _Out_ size_t num_domains);


What is the expectation for OrtCustomOpDomain** domains this will contain a deep copy or shallow copy of the OrtCustomOp that ep provides?

The OrtCustomOpDomain contains a shallow copy (a pointer) of the OrtCustomOp.

The reason is:

EP should call OrtApis::CustomOpDomain_Add() to add OrtCustomOp to OrtCustomOpDomain.
As per api implementation, it simply stores the pointer of the OrtCustomOp in the vector, not constructing a new OrtCustomOp

ORT_API_STATUS_IMPL(OrtApis::CustomOpDomain_Add, _Inout_ OrtCustomOpDomain* custom_op_domain, _In_ const OrtCustomOp* op) { API_IMPL_BEGIN custom_op_domain->custom_ops_.emplace_back(op); return nullptr; API_IMPL_END }

Then, inside ORT, it simply stores the pointer of OrtCustomOpDomain returned from EP into session options.

Status AddEpOptionsToSessionOptions() { ... ORT_RETURN_IF_ERROR(ToStatusAndRelease(ep_factory->CreateCustomOpDomains(ep_factory, domains.data(), domains.size()))); const auto domains_span = gsl::span<OrtCustomOpDomain*>(domains.data(), domains.size()); for (auto domain : domains_span) { if (!is_already_in_domains(domain->domain_, ort_session_options.custom_op_domains_) && domain->custom_ops_.size() > 0) { ort_session_options.custom_op_domains_.push_back(domain); } else { LOGS_DEFAULT(WARNING) << "The custom op domain name " << domain->domain_ << " is already in the session option. Skip it."; } } ... }

adrianlizarraga

some minor things

adrianlizarraga · 2025-12-15T16:16:10Z

onnxruntime/core/session/utils.cc

+    OrtEpFactory* ep_factory = ep_device->ep_factory;
+    if (ep_factory &&
+        ep_factory->ort_version_supported >= 24 &&
+        ep_factory->CreateCustomOpDomains != nullptr) {


also need to check ep_factory->GetNumCustomOpDomains just in case?

adrianlizarraga · 2025-12-15T16:19:02Z

onnxruntime/core/session/plugin_ep/ep_factory_internal_impl.h

    return nullptr;
  }

+  virtual OrtStatus* CreateCustomOpDomains(_Outptr_result_maybenull_ OrtCustomOpDomain** out,


Do we also need GetNumCustomOpDomains in here?

adrianlizarraga · 2025-12-15T16:22:53Z

onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.h

+//
+// Note: Approach #2 is suitable for plugin TRT RTX EP to support TRT plugins.
+
+struct CustomMulKernel : MulKernel {


nit: could these custom op classes be placed in a separate file?

github-actions

You can commit the suggested changes from lintrunner.

github-actions · 2025-12-18T08:42:12Z

onnxruntime/test/autoep/library/example_plugin_ep/ep_custom_op.h

+#include "onnxruntime_c_api.h"
+#include "ep.h"
+
+


Suggested change

onnxruntime/test/autoep/library/example_plugin_ep/ep_custom_op.h

edgchen1 · 2025-12-19T23:42:30Z

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

+   * Note: EP has the responsibility to release OrtCustomOpDomain instances it creates. It happens
+   *       automatically if using ORT C++ api.


IMO, with a name like "CreateCustomOpDomains", there's some expectation of ownership transfer. E.g., how OrtApi::CreateCustomOpDomain() creates a new domain that the user is responsible for releasing.

since this function does not transfer ownership to the caller, maybe a name like GetCustomOpDomains() would be better?

It makes sense to me to have a name GetCustomOpDomains(). Changed.

edgchen1 · 2025-12-20T00:07:12Z

onnxruntime/test/autoep/library/example_plugin_ep/ep_custom_op.h

+  }
+};
+
+struct ExampleEpCustomOp : Ort::CustomOpBase<ExampleEpCustomOp, CustomMulKernel> {


CustomMulKernel::Compute() returns OrtStatus*. should this inherit from CustomOpBase<ExampleEpCustomOp, CustomMulKernel, /*WithStatus*/ true> instead so that the OrtStatus will be checked?

Good catch. I also used CreateKernelV2 and ComputeKernerV2 so that OrtStatus will be checked.

edgchen1 · 2025-12-20T00:46:59Z

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

+   * \since Version 1.24.
+   */
+  ORT_API2_STATUS(CreateCustomOpDomains, _In_ OrtEpFactory* this_ptr,
+                  _Outptr_result_maybenull_ OrtCustomOpDomain** domains, _In_ size_t num_domains);


why is _Outptr_result_maybenull_ used for domains?

I think EP can still implement this function and creates no custom op domain.
Or do you think _Outptr_result_ makes more sense?

hm, would _Out_writes_all_(num_domains) make sense here? assuming that num_domains is the value returned by GetNumCustomOpDomains() - perhaps that should also be documented. the implementation is expected to treat domains as a buffer.

ynankani · 2025-12-23T08:59:31Z

onnxruntime/core/session/utils.cc

  return Status::OK();
 }

 Status AddEpOptionsToSessionOptions(gsl::span<const OrtEpDevice* const> ep_devices,


Is this not called during auto ep selection path, when the session_options.set_provider_selection_policy(ort.OrtExecutionProviderDevicePolicy.PREFER_GPU) is set?
For auto ep path when it creates plugin ep it goes for inference_session.cc - RegisterExecutionProvider() where it calls GetCustomOpDomainList? Is this understanding correct?

In the case of EP using GetCustomOpDomains to register custom op, only this EP can run the model contains that custom op.
IMO, the application should explicitly call SessionOptionsAppendExecutionProvider_V2() and specify that ep device, rather than using auto ep selection as other devices might not be able to run that custom op.

Registering custom ops should be independent of EP , so it should be available in auto ep selectin path as well

github-actions

You can commit the suggested changes from lintrunner.

onnxruntime/test/autoep/test_execution.cc

github-actions

You can commit the suggested changes from lintrunner.

onnxruntime/test/autoep/test_execution.cc

github-actions

You can commit the suggested changes from lintrunner.

onnxruntime/core/session/plugin_ep/ep_factory_internal_impl.h

onnxruntime/core/session/utils.cc

onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.h

edgchen1 · 2026-01-09T02:49:14Z

onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.cc

+  auto* factory = static_cast<ExampleEpFactory*>(this_ptr);
+
+  // Custom Op Domains
+  factory->custom_op_domains_[0] = Ort::CustomOpDomain{"test"};


would it be better to only create the custom op domains once, e.g., in the factory ctor?

Ah, that's good suggestion. Moved to factor ctor.

edgchen1 · 2026-01-09T03:17:14Z

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

+   * \since Version 1.24.
+   */
+  ORT_API2_STATUS(CreateCustomOpDomains, _In_ OrtEpFactory* this_ptr,
+                  _Outptr_result_maybenull_ OrtCustomOpDomain** domains, _In_ size_t num_domains);


hm, would _Out_writes_all_(num_domains) make sense here? assuming that num_domains is the value returned by GetNumCustomOpDomains() - perhaps that should also be documented. the implementation is expected to treat domains as a buffer.

edgchen1 · 2026-01-09T03:26:51Z

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

+   * This function is used when running inference on a model that contains EP-specific custom operations.
+   *
+   * Workflow:
+   * 1. The EP implements this function to supply a list of OrtCustomOpDomain instances.


should clarify that the custom op domains are provided by the EP factory, not the EP.

yes, modified to EP factory

edgchen1 · 2026-01-09T03:30:55Z

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

+   *    - In Compile(), the EP executes its compiled bits to perform inference for
+   *      the fused custom node.
+   *
+   * Note: EP has the responsibility to release OrtCustomOpDomain instances it creates. It happens


it would be the EP factory and not the EP that has this responsibility, right? and the instances must be valid while any session is using them

edgchen1 · 2026-01-09T03:33:16Z

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

   */
  ORT_API2_STATUS(SetEnvironmentOptions, _In_ OrtEpFactory* this_ptr, _In_ const OrtKeyValuePairs* options);
+
+  /** \brief Returns the number of OrtCustomOpDomains that this factory creates.


since we renamed it to GetCustomOpDomains, maybe replace "creates" with "provides" or "supplies".

replaced with "provides"

edgchen1 · 2026-01-09T03:35:59Z

onnxruntime/test/autoep/library/example_plugin_ep/ep.cc


        supported_nodes.push_back(node);  // Only support a single Mul for now.
        break;
+      } else if (op_type == "Custom_Mul") {


should we also check the domain?

I think we don't need to. As when the model is loading, ORT already checks the domain.
Also, we can't really get the domain info given only an op, can we? i might be wrong

I mean the op domain, like node.GetDomain(). domain + op type identifies the op. here we are matching an op from a custom domain.

Ah, okay, i added the domain check.

github-actions

You can commit the suggested changes from lintrunner.

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

onnxruntime/core/session/plugin_ep/ep_factory_internal_impl.h

onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.cc

onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.h

chilo-ms added 4 commits December 8, 2025 14:47

update

0b5b4d7

update

80561db

add API summary

6bd316f

update

ad0a023

adrianlizarraga reviewed Dec 9, 2025

View reviewed changes

chilo-ms added 2 commits December 9, 2025 16:29

address reviewer's comments and add GetNumCustomOpDomains

5e398d4

update example ep to run Custom_Mul op

aeb2386

github-advanced-security bot found potential problems Dec 10, 2025

View reviewed changes

onnxruntime/test/testdata/custom_mul.py Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Dec 10, 2025

View reviewed changes

onnxruntime/test/testdata/custom_mul.py Fixed Show fixed Hide fixed

onnxruntime/test/testdata/custom_mul.py Fixed Show fixed Hide fixed

onnxruntime/test/testdata/custom_mul.py Fixed Show fixed Hide fixed

github-actions bot reviewed Dec 10, 2025

View reviewed changes

chilo-ms added 2 commits December 10, 2025 15:09

address reviewr's comment

3849cd3

lintrunner -a

9c987be

chilo-ms marked this pull request as ready for review December 10, 2025 23:12

github-advanced-security bot found potential problems Dec 10, 2025

View reviewed changes

onnxruntime/test/testdata/custom_mul.py Dismissed Show dismissed Hide dismissed

update example ep GetCapability()

fbe2434

ishwar-raut1 reviewed Dec 11, 2025

View reviewed changes

update Example EP

40fa8fe

ynankani reviewed Dec 12, 2025

View reviewed changes

add more comments in API summary

c7a0491

adrianlizarraga reviewed Dec 15, 2025

View reviewed changes

address reviewer's comments

4787c3f

github-actions bot reviewed Dec 18, 2025

View reviewed changes

github-advanced-security bot found potential problems Dec 18, 2025

View reviewed changes

onnxruntime/test/autoep/library/example_plugin_ep/ep_custom_op.h Fixed Show fixed Hide fixed

lintrunner -a

632ce31

edgchen1 reviewed Dec 20, 2025

View reviewed changes

ynankani reviewed Dec 23, 2025

View reviewed changes

Merge branch 'main' into chi/custom_op_for_ep

5905434

github-actions bot reviewed Jan 7, 2026

View reviewed changes

onnxruntime/test/autoep/test_execution.cc Outdated Show resolved Hide resolved

Use CreateKernelV2 and ComputeKernelV2

6017c00

github-actions bot reviewed Jan 8, 2026

View reviewed changes

onnxruntime/test/autoep/test_execution.cc Outdated Show resolved Hide resolved

address reviewer's comments

47bb4dc

github-actions bot reviewed Jan 8, 2026

View reviewed changes

onnxruntime/core/session/plugin_ep/ep_factory_internal_impl.h Outdated Show resolved Hide resolved

onnxruntime/core/session/utils.cc Outdated Show resolved Hide resolved

onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.h Outdated Show resolved Hide resolved

chilo-ms added 3 commits January 7, 2026 19:42

lintrunner -a

6721a98

update

1ab246d

Remove accidentally added file

3478732

edgchen1 reviewed Jan 9, 2026

View reviewed changes

chilo-ms added the release:1.24.0 label Jan 9, 2026

chilo-ms and others added 4 commits January 9, 2026 10:58

address reviewer's comments

a1d36af

address reviewer's comment

3065e9d

address reveiwer's comment

d340de5

Merge branch 'main' into chi/custom_op_for_ep

ee8851b

github-actions bot reviewed Jan 9, 2026

View reviewed changes

chilo-ms added 3 commits January 9, 2026 12:51

update

15f5baf

lintrunner -a

6b01e7f

fix bug when merging main

adf565e

		* Note: EP has the responsibility to release OrtCustomOpDomain instances it creates. It happens
		* automatically if using ORT C++ api.

		#include "onnxruntime_c_api.h"
		#include "ep.h"

[EP ABI] Add CreateCustomOpDomains() API for plugin EP to register custom ops #26759

Are you sure you want to change the base?

[EP ABI] Add CreateCustomOpDomains() API for plugin EP to register custom ops #26759

Conversation

chilo-ms commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chilo-ms Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chilo-ms Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chilo-ms Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adrianlizarraga left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chilo-ms Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chilo-ms commented Dec 9, 2025 •

edited

Loading

chilo-ms Dec 10, 2025 •

edited

Loading

chilo-ms Dec 11, 2025 •

edited

Loading

chilo-ms Dec 12, 2025 •

edited

Loading

chilo-ms Jan 8, 2026 •

edited

Loading

chilo-ms Jan 8, 2026 •

edited

Loading