[EP ABI] Add APIs to create control flow kernels for plugin EPs #26927

adrianlizarraga · 2026-01-07T08:26:05Z

Description

Adds APIs to enable plugin EPs to create and register kernels for control flow operators (If, Loop, and Scan). The implementation provides ORT-managed kernel implementations that handle subgraph execution while allowing EPs to provide device-specific helper functions for operations like tensor concatenation and transposition.

Key changes:

Adds four EP API functions: CreateIfKernel, CreateLoopKernel, CreateScanKernel, and ReleaseKernelImpl for creating control flow kernel implementations
Introduces public helper structures (OrtLoopKernelHelper, OrtScanKernelHelper) that EPs implement to provide device-specific operations
Updates the example kernel-based EP with kernel registrations for all control flow operators and adds corresponding test models.

Motivation and Context

…nels

…Impl returned by ORT with an OrtKernelImpl created by the EP

… along with the helpers

Copilot

Pull request overview

This pull request adds APIs to enable plugin execution providers (EPs) to create and register kernels for control flow operators (If, Loop, and Scan). The implementation provides ORT-managed kernel implementations that handle the complex subgraph execution while allowing EPs to provide device-specific helper functions for operations like tensor concatenation and transposition.

Key changes:

Adds three new OrtEpApi functions: CreateIfKernel, CreateLoopKernel, and CreateScanKernel for creating control flow kernel implementations
Introduces helper structures (OrtLoopKernelHelper, OrtScanKernelHelper) that EPs implement to provide device-specific operations
Adds comprehensive test coverage with new ONNX model test cases and execution tests for all three control flow operators

Reviewed changes

Copilot reviewed 28 out of 31 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
onnxruntime/test/testdata/scan_mul.py	Python script to generate test model for Scan operator
onnxruntime/test/testdata/scan_mul.onnx	Binary ONNX test model for Scan operator
onnxruntime/test/testdata/loop_sub_one.py	Python script to generate test model for Loop operator
onnxruntime/test/testdata/loop_sub_one.onnx	Binary ONNX test model for Loop operator
onnxruntime/test/testdata/if_mul.py	Python script to generate test model for If operator
onnxruntime/test/testdata/if_mul.onnx	Binary ONNX test model for If operator
onnxruntime/test/autoep/test_execution.cc	Adds execution tests for If, Loop, and Scan operators with the plugin EP
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/kernels/utils.h	Updates kernel creation macro to match new Create signature returning OrtKernelImpl*
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/kernels/squeeze.h/cc	Updates Create method signature and adds creator field initialization
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/kernels/relu.h/cc	Updates Create method signature and adds creator field initialization
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/kernels/binary_op.h/cc	Updates Create method signature and adds creator field initialization
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/kernels/scan.h/cc	Implements ScanHelper for Scan operator with transpose helper function
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/kernels/loop.h/cc	Implements LoopHelper for Loop operator with concatenation helper function
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/kernels/if.h/cc	Implements IfHelper for If operator
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/ep_kernel_registration.cc	Registers If, Loop, and Scan kernels with the plugin EP
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/ep.cc	Updates GetCapability to claim If, Loop, and Scan operators
onnxruntime/core/session/plugin_ep/ep_kernel_registration.cc	Updates PluginEpOpKernel to derive from IControlFlowKernel and implement SetupSubgraphExecutionInfo
onnxruntime/core/session/plugin_ep/ep_control_flow_kernel_impls.h	Defines control flow kernel implementation classes that wrap CPU EP kernels
onnxruntime/core/session/plugin_ep/ep_control_flow_kernel_impls.cc	Implements control flow kernel wrappers and helper function adapters
onnxruntime/core/session/plugin_ep/ep_api.h	Declares new control flow kernel creation functions
onnxruntime/core/session/plugin_ep/ep_api.cc	Implements control flow kernel creation APIs with validation
onnxruntime/core/providers/cpu/controlflow/loop.h	Changes SetConcatOutputFunc visibility to public
onnxruntime/core/framework/session_state.cc	Updates CreateSubgraphSessionState to handle plugin EPs with control flow operators
include/onnxruntime/core/session/onnxruntime_ep_c_api.h	Adds OrtKernelImplCreator enum, helper structures, and new API function declarations
cmake/onnxruntime_unittests.cmake	Adds new control flow kernel source files to the test build

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

onnxruntime/core/session/plugin_ep/ep_control_flow_kernel_impls.h

onnxruntime/core/session/plugin_ep/ep_api.cc

onnxruntime/core/framework/session_state.cc

onnxruntime/core/session/plugin_ep/ep_api.cc

onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/kernels/scan.cc

onnxruntime/core/session/plugin_ep/ep_kernel_registration.cc

Copilot

Pull request overview

Copilot reviewed 28 out of 31 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

onnxruntime/core/session/plugin_ep/ep_api.cc

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

onnxruntime/core/framework/session_state.cc

…dates

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

onnxruntime/core/session/plugin_ep/ep_control_flow_kernel_impls.cc

…, catch exceptions and propagate as OrtStatus* for internal OrtKernelImpl implementations

…ullptr

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

…osoft#26927) ### Description Adds APIs to enable plugin EPs to create and register kernels for control flow operators (If, Loop, and Scan). The implementation provides ORT-managed kernel implementations that handle subgraph execution while allowing EPs to provide device-specific helper functions for operations like tensor concatenation and transposition. Key changes: - Adds four EP API functions: `CreateIfKernel`, `CreateLoopKernel`, `CreateScanKernel`, and `ReleaseKernelImpl` for creating control flow kernel implementations - Introduces public helper structures (`OrtLoopKernelHelper`, `OrtScanKernelHelper`) that EPs implement to provide device-specific operations - Updates the example kernel-based EP with kernel registrations for all control flow operators and adds corresponding test models. ### Motivation and Context

Fixes multiple Doxygen errors introduced by recent API additions that cause the nightly documentation build to fail with WARN_AS_ERROR=YES. Root causes by PR: PR #26828 (c54be3c - OrtExternalResourceImporter API for D3D12): - Incorrect \param names for ORT_CLASS_RELEASE macros: the macro expands to use 'input' as the parameter name, but docs used 'importer' and 'handle' PR #26927 (1ed8fd9 - Control flow kernels API): - Used ::CreateLoopKernel() and ::CreateScanKernel() syntax which Doxygen cannot resolve as links PR #26971 (3874516 - CreateEnvWithOptions API): - Used ::SetSessionLogSeverityLevel() and ::RunOptionsSetRunLogSeverityLevel() syntax which Doxygen cannot resolve - Used <ep_name> in documentation which Doxygen interprets as HTML tag - Used incorrect \param[out] name 'out' instead of 'config_entries' Fixes: - Qualify method references with struct name (e.g., OrtApi::Method) - Escape angle brackets as \<ep_name\> - Use 'input' as param name for ORT_CLASS_RELEASE documentation - Use actual param name 'config_entries' for GetEnvConfigEntries Verified locally with Doxygen 1.9.8 (matches CI configuration).

# Fix Doxygen documentation build errors from recent PRs Fixes multiple Doxygen errors introduced by recent API additions that cause the nightly documentation build to fail (`WARN_AS_ERROR=YES`). ## Root Cause Analysis | Error | File | Line | Introduced By | Commit | Fix | |-------|------|------|---------------|--------|-----| | Duplicate `\addtogroup Global` | onnxruntime_c_api.h | 973 | PR #26828 - OrtExternalResourceImporter API | c54be3c | Remove redundant group markers | | Unresolved `::SetSessionLogSeverityLevel()` | onnxruntime_c_api.h | 1065 | PR #26971 - CreateEnvWithOptions API | 3874516 | Use `OrtApi::SetSessionLogSeverityLevel` | | Unresolved `::RunOptionsSetRunLogSeverityLevel()` | onnxruntime_c_api.h | 1066 | PR #26971 - CreateEnvWithOptions API | 3874516 | Use `OrtApi::RunOptionsSetRunLogSeverityLevel` | | `<ep_name>` interpreted as HTML | onnxruntime_c_api.h | 1119 | PR #26971 - CreateEnvWithOptions API | 3874516 | Escape as `\<ep_name\>` | | `\param[in] importer` not found | onnxruntime_c_api.h | 7982 | PR #26828 - OrtExternalResourceImporter API | c54be3c | Use `\param[in] input` (macro expands to `input`) | | `\param[in] handle` not found | onnxruntime_c_api.h | 8025 | PR #26828 - OrtExternalResourceImporter API | c54be3c | Use `\param[in] input` | | `\param[in] handle` not found | onnxruntime_c_api.h | 8091 | PR #26828 - OrtExternalResourceImporter API | c54be3c | Use `\param[in] input` | | Unresolved `::CreateLoopKernel()` | onnxruntime_ep_c_api.h | 667 | PR #26927 - Control flow kernels API | 1ed8fd9 | Use `OrtEpApi::CreateLoopKernel` | | Unresolved `::CreateScanKernel()` | onnxruntime_ep_c_api.h | 710 | PR #26927 - Control flow kernels API | 1ed8fd9 | Use `OrtEpApi::CreateScanKernel` | | `<ep_name>` interpreted as HTML | onnxruntime_ep_c_api.h | 1434 | PR #26971 - CreateEnvWithOptions API | 3874516 | Escape as `\<ep_name\>` | | `\param[out] out` not found | onnxruntime_ep_c_api.h | 1440 | PR #26971 - CreateEnvWithOptions API | 3874516 | Use `\param[out] config_entries` | ## Summary by PR | PR | Issues | |----|--------| | **#26828** (c54be3c) - OrtExternalResourceImporter API for D3D12 | Duplicate Doxygen group, incorrect `\param` names for `ORT_CLASS_RELEASE` macros | | **#26927** (1ed8fd9) - Control flow kernels API | `::Method()` syntax unresolvable by Doxygen | | **#26971** (3874516) - CreateEnvWithOptions API | `::Method()` syntax, `<ep_name>` HTML interpretation, incorrect param name | ## Technical Details ### `ORT_CLASS_RELEASE` Macro Issue The `ORT_CLASS_RELEASE(X)` macro at line 164 expands to: ```cpp void(ORT_API_CALL * Release##X)(_Frees_ptr_opt_ Ort##X * input) ``` The parameter is always named `input`, but the documentation in PR #26828 used semantic names like `importer` and `handle`. Doxygen validates `\param` names against actual parameter names in the expanded code. ### Doxygen Link Resolution Doxygen 1.9.8 cannot resolve `::MethodName()` as a link to a method. The correct syntax is to qualify with the struct name: `OrtApi::MethodName`. ## Testing Verified locally with Doxygen 1.9.8 (matches CI configuration).

# Fix Doxygen documentation build errors from recent PRs Fixes multiple Doxygen errors introduced by recent API additions that cause the nightly documentation build to fail (`WARN_AS_ERROR=YES`). ## Root Cause Analysis | Error | File | Line | Introduced By | Commit | Fix | |-------|------|------|---------------|--------|-----| | Duplicate `\addtogroup Global` | onnxruntime_c_api.h | 973 | PR #26828 - OrtExternalResourceImporter API | c54be3c | Remove redundant group markers | | Unresolved `::SetSessionLogSeverityLevel()` | onnxruntime_c_api.h | 1065 | PR #26971 - CreateEnvWithOptions API | 3874516 | Use `OrtApi::SetSessionLogSeverityLevel` | | Unresolved `::RunOptionsSetRunLogSeverityLevel()` | onnxruntime_c_api.h | 1066 | PR #26971 - CreateEnvWithOptions API | 3874516 | Use `OrtApi::RunOptionsSetRunLogSeverityLevel` | | `<ep_name>` interpreted as HTML | onnxruntime_c_api.h | 1119 | PR #26971 - CreateEnvWithOptions API | 3874516 | Escape as `\<ep_name\>` | | `\param[in] importer` not found | onnxruntime_c_api.h | 7982 | PR #26828 - OrtExternalResourceImporter API | c54be3c | Use `\param[in] input` (macro expands to `input`) | | `\param[in] handle` not found | onnxruntime_c_api.h | 8025 | PR #26828 - OrtExternalResourceImporter API | c54be3c | Use `\param[in] input` | | `\param[in] handle` not found | onnxruntime_c_api.h | 8091 | PR #26828 - OrtExternalResourceImporter API | c54be3c | Use `\param[in] input` | | Unresolved `::CreateLoopKernel()` | onnxruntime_ep_c_api.h | 667 | PR #26927 - Control flow kernels API | 1ed8fd9 | Use `OrtEpApi::CreateLoopKernel` | | Unresolved `::CreateScanKernel()` | onnxruntime_ep_c_api.h | 710 | PR #26927 - Control flow kernels API | 1ed8fd9 | Use `OrtEpApi::CreateScanKernel` | | `<ep_name>` interpreted as HTML | onnxruntime_ep_c_api.h | 1434 | PR #26971 - CreateEnvWithOptions API | 3874516 | Escape as `\<ep_name\>` | | `\param[out] out` not found | onnxruntime_ep_c_api.h | 1440 | PR #26971 - CreateEnvWithOptions API | 3874516 | Use `\param[out] config_entries` | ## Summary by PR | PR | Issues | |----|--------| | **#26828** (c54be3c) - OrtExternalResourceImporter API for D3D12 | Duplicate Doxygen group, incorrect `\param` names for `ORT_CLASS_RELEASE` macros | | **#26927** (1ed8fd9) - Control flow kernels API | `::Method()` syntax unresolvable by Doxygen | | **#26971** (3874516) - CreateEnvWithOptions API | `::Method()` syntax, `<ep_name>` HTML interpretation, incorrect param name | ## Technical Details ### `ORT_CLASS_RELEASE` Macro Issue The `ORT_CLASS_RELEASE(X)` macro at line 164 expands to: ```cpp void(ORT_API_CALL * Release##X)(_Frees_ptr_opt_ Ort##X * input) ``` The parameter is always named `input`, but the documentation in PR #26828 used semantic names like `importer` and `handle`. Doxygen validates `\param` names against actual parameter names in the expanded code. ### Doxygen Link Resolution Doxygen 1.9.8 cannot resolve `::MethodName()` as a link to a method. The correct syntax is to qualify with the struct name: `OrtApi::MethodName`. ## Testing Verified locally with Doxygen 1.9.8 (matches CI configuration). (cherry picked from commit 39f966e)

adrianlizarraga added 11 commits January 7, 2026 00:23

Add basic (untested) C API functions for creating control flow op ker…

2991ff8

…nels

Fix compilation error on mac

98a5bb8

Try again

5106084

Get If kernel working for example EP

c0c3eae

Add OrtKernelImpl::GetControlFlowKernel to allow wrapping a OrtKernel…

74b0580

…Impl returned by ORT with an OrtKernelImpl created by the EP

Mark classes as final

50a21f9

Add Loop kernel to example EP

6397544

Move kernel impls to separate files

fde7b3b

Add Scan kernel to example EP (and test)

9ffd692

Simplify

c5849ac

Add public kernel helper structs to the API. This allow storing state…

f0018a1

… along with the helpers

adrianlizarraga requested a review from Copilot January 9, 2026 09:11

Copilot started reviewing on behalf of adrianlizarraga January 9, 2026 09:11 View session

Copilot AI reviewed Jan 9, 2026

View reviewed changes

adrianlizarraga added 2 commits January 9, 2026 01:45

Address review comments

aa9b970

Merge branch 'main' into adrianl/KernelPluginEp_ControlFlowKernelApis

ccba755

adrianlizarraga requested a review from Copilot January 9, 2026 09:47

Copilot started reviewing on behalf of adrianlizarraga January 9, 2026 09:48 View session

adrianlizarraga marked this pull request as ready for review January 9, 2026 09:48

Copilot AI reviewed Jan 9, 2026

View reviewed changes

Rephrase doc comments

e4b5fb4

adrianlizarraga requested review from edgchen1, fs-eire and skottmckay January 9, 2026 10:02

skottmckay reviewed Jan 9, 2026

View reviewed changes

Address most comments: flags field in OrtKernelImpl, documentation up…

b5efde0

…dates

skottmckay reviewed Jan 9, 2026

View reviewed changes

include/onnxruntime/core/session/onnxruntime_ep_c_api.h Show resolved Hide resolved

skottmckay previously approved these changes Jan 9, 2026

View reviewed changes

edgchen1 reviewed Jan 9, 2026

View reviewed changes

Added validation checks for OrtKernelImpl provided by EP

af26588

adrianlizarraga dismissed skottmckay’s stale review via af26588 January 9, 2026 22:55

adrianlizarraga added 2 commits January 9, 2026 15:38

Address comments: doc updates, const OrtValue inputs in concat helper…

2f193b9

…, catch exceptions and propagate as OrtStatus* for internal OrtKernelImpl implementations

Enforce that IExecutionProvider* retrieved from OpKernelInfo is not n…

642e7f1

…ullptr

adrianlizarraga requested a review from edgchen1 January 10, 2026 00:01

edgchen1 approved these changes Jan 10, 2026

View reviewed changes

include/onnxruntime/core/session/onnxruntime_ep_c_api.h Show resolved Hide resolved

include/onnxruntime/core/session/onnxruntime_ep_c_api.h Show resolved Hide resolved

adrianlizarraga enabled auto-merge (squash) January 10, 2026 08:43

adrianlizarraga merged commit 1ed8fd9 into main Jan 10, 2026
102 of 110 checks passed

adrianlizarraga deleted the adrianl/KernelPluginEp_ControlFlowKernelApis branch January 10, 2026 11:55

nieubank mentioned this pull request Jan 20, 2026

Fix Doxygen documentation build error in onnxruntime_c_api.h #27083

Merged

[EP ABI] Add APIs to create control flow kernels for plugin EPs #26927

[EP ABI] Add APIs to create control flow kernels for plugin EPs #26927

Uh oh!

Conversation

adrianlizarraga commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

adrianlizarraga commented Jan 7, 2026 •

edited

Loading