Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Out-Tree EP feature #21450

Draft
wants to merge 79 commits into
base: main
Choose a base branch
from
Draft

[WIP] Out-Tree EP feature #21450

wants to merge 79 commits into from

Commits on Jul 17, 2024

  1. opaque pointer for graph

    jslhcl committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    0e6a80c View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. Configuration menu
    Copy the full SHA
    c30a639 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7bfe57e View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. Test Relu with compile based EP, build work, runtime error of loading…

    … EP as graph API is not exported by ORT. Need to put these graph API into ortapi structure
    jslhcl committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    8e7d28d View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. prototype works with hardcode node_compute_info's index in ExecutionP…

    …roviderAdapter::Compile()
    jslhcl committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    808bfc3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    49e396c View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. Configuration menu
    Copy the full SHA
    e790105 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. Configuration menu
    Copy the full SHA
    92f529d View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Configuration menu
    Copy the full SHA
    3d83ed1 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Configuration menu
    Copy the full SHA
    e29499a View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. Configuration menu
    Copy the full SHA
    f3678c4 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ac5ae0a View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2024

  1. introduce qnn ep

    jslhcl committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    0cc78e8 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. more graph/node C API

    jslhcl committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    740a687 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. stream support

    jslhcl committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    dad6397 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2024

  1. Configuration menu
    Copy the full SHA
    94e9cf7 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Configuration menu
    Copy the full SHA
    8698517 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3d5d2bf View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. Add extra parameters in OrtExecutionProvider to avoid capture variabl…

    …es in Compile function's lambda
    jslhcl committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    1f10c28 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. add OrtGraph_SerializeToArray

    jslhcl committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    5e46d0f View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2024

  1. finish Compile function

    jslhcl committed Aug 24, 2024
    Configuration menu
    Copy the full SHA
    85c168d View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. Configuration menu
    Copy the full SHA
    7bdb36a View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. add outOfTree tensorrt ep.1 (#21830)

    add GetCapability for tensorrt ep
    guyang3532 authored Aug 27, 2024
    Configuration menu
    Copy the full SHA
    7d915b7 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2024

  1. GetSupportedList

    jslhcl committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    4aea94b View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. Configuration menu
    Copy the full SHA
    865a17f View commit details
    Browse the repository at this point in the history
  2. Add simple CUDA allocators for TRT EP (#21901)

    Only implemented two simple CUDA allocators (without BFC) for now.
    Note: several TODO.
    chilo-ms authored Aug 29, 2024
    Configuration menu
    Copy the full SHA
    2811541 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c97b19f View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    36f97b5 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2024

  1. Configuration menu
    Copy the full SHA
    2fc7aac View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2024

  1. Configuration menu
    Copy the full SHA
    4ad6993 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. Configuration menu
    Copy the full SHA
    53c736f View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. Configuration menu
    Copy the full SHA
    5fcb972 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Configuration menu
    Copy the full SHA
    c3bb437 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d1c657c View commit details
    Browse the repository at this point in the history
  3. resnet18-v1-7 works for TRT EP, with next_nodes_list assignment comme…

    …nted out in GetSupportedList()
    jslhcl committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    3efac97 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. test cases for decoder and fast_rcnn, delete dynamic_cast in ShouldPo…

    …stPoneRegisterResourceFor()
    jslhcl committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    766fec9 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. add tensorrt home in CMakeLists, add trt and CUDA ep for test, change…

    … trt type to tensorrtEp
    jslhcl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    ea2465c View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. [WIP, DONT REVIEW] add initializer to graph proto (#22085)

    ### Description
    <!-- Describe your changes. -->
    
    
    
    ### Motivation and Context
    <!-- - Why is this change required? What problem does it solve?
    - If it fixes an open issue, please link to the issue here. -->
    
    ---------
    
    Co-authored-by: guyang3532 <[email protected]>
    jslhcl and guyang3532 authored Sep 18, 2024
    Configuration menu
    Copy the full SHA
    76a9305 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. use parameter ExecutionOrder::PRIORITY_BASED for GraphViewerToProto()…

    … to make graph partition the same, and improve test
    jslhcl committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    330cdb6 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. can create session with out tree trt ep now. Error:Name:'tensorrtEp_T…

    …RTKernel_graph_torch-jit-export10497453988321570186_10_10' Status Message: TensorRT EP failed to create engine from network
    jslhcl committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    6fd50f0 View commit details
    Browse the repository at this point in the history
  2. make trt_node_name_with_precision_ from string to map, to capture the…

    … corresponding value for compute function
    jslhcl committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    681585f View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. Configuration menu
    Copy the full SHA
    7db20cb View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. RunTinyYolov3()

    jslhcl committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    ff782e0 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. Configuration menu
    Copy the full SHA
    1d7b2df View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a407944 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. new test control_flow, error: ErrorMessage:Failed to find kernel for …

    …MemcpyFromHost(1) (node:'Memcpy' ep:'tensorrtEp'). Kernel not found
    jslhcl committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    f871b25 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. control flow model works

    jslhcl committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    e84f00c View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. API refactor

    jslhcl committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    5b2de22 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. Python API

    jslhcl committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    b1f8e2a View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. fix memory leak (#22444)

    guyang3532 authored Oct 15, 2024
    Configuration menu
    Copy the full SHA
    7acaaab View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. Configuration menu
    Copy the full SHA
    d150a03 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2024

  1. resolve comments

    jslhcl committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    da5b6eb View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d280e59 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2024

  1. fix comments

    jslhcl committed Oct 19, 2024
    Configuration menu
    Copy the full SHA
    cbe98e7 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. fix memory leak (#22522)

    guyang3532 authored Oct 21, 2024
    Configuration menu
    Copy the full SHA
    1529059 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. Configuration menu
    Copy the full SHA
    fa549f8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a28ad38 View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2024

  1. openvino

    jslhcl committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    aa49805 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2024

  1. Configuration menu
    Copy the full SHA
    bc65613 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2024

  1. Configuration menu
    Copy the full SHA
    a1a3eea View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0fe5f01 View commit details
    Browse the repository at this point in the history
  3. Add utility files (#22650)

    Add some utility files for plugin ep to include and compile.
    
    - provider option map -> `provider_option`.h
    - provider option parser -> `provider_option_utils`.h
    - some macro define, classes and functions from
    include/onnxruntime/core/common
    chilo-ms authored Nov 1, 2024
    Configuration menu
    Copy the full SHA
    6bae1b9 View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2024

  1. OpenVino, compile() is done

    jslhcl committed Nov 2, 2024
    Configuration menu
    Copy the full SHA
    ab75d98 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c5510f2 View commit details
    Browse the repository at this point in the history
  3. Add unit test for TRT EP plugin (#22548)

    Hook TRT EP plugin to run the existing unit test in CI
    
    - Migrate from
    `onnxruntime/test/providers/tensorrt/tensorrt_basic_test.cc`
    - Replace internal APIs with new EP APIs
    - Add unit test in `onnxruntime_shared_lib_test` (which links against
    onnxruntime dll)
    - Build ORT with `--test_tensorrt_ep_plugin` to run
    `onnxruntime_shared_lib_test`
    
    Note: The unit test doesn't cover all the cases since current TRT EP
    plugin hasn't added all the features yet, will update later.
    chilo-ms authored Nov 2, 2024
    Configuration menu
    Copy the full SHA
    08e3f20 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2024

  1. Configuration menu
    Copy the full SHA
    b0b3123 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2024

  1. Configuration menu
    Copy the full SHA
    9dbb0b1 View commit details
    Browse the repository at this point in the history
  2. merge code

    jslhcl committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    5a59803 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    999e7fd View commit details
    Browse the repository at this point in the history
  4. fix bugs (#22744)

    guyang3532 authored Nov 6, 2024
    Configuration menu
    Copy the full SHA
    084f735 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2024

  1. Configuration menu
    Copy the full SHA
    2b1cfdf View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2024

  1. Add OrtGraphApis::OrtNode_GetAttributeStrWithSize to handle case wher…

    …e attribute might contain null character (#22769)
    
    When running EP Context model, EP might call
    `OrtGraphApis::OrtNode_GetAttributeStr` to get the string-based content
    of the attribute.
    However, the API returns the c_str() of the string, and it's possible
    that the cache context contains null character, so the string might be
    cut off and caller ends up getting the wrong string.
    
    Add a new OrtGraphApis::OrtNode_GetAttributeStrWithSize to return const
    char* pointer and string size.
    chilo-ms authored Nov 13, 2024
    Configuration menu
    Copy the full SHA
    e337d8f View commit details
    Browse the repository at this point in the history
  2. Make EP plugin be able to create and update EP Context graph (#22740)

    This PR support several features:
    
    - Add new graph API to create and update EP Context graph, and dump EP
    Context model.
    
    1.  OrtGraph_CreateOrUpdateEpCtxGraph
    2. OrtGraph_DumpOnnxModel
    3. OrtGraph_ReleaseGraph 
    
    - Add new graph API to dump onnx model
    - The APIs provided by this PR can dump EP Context model when the whole
    model can be run by one EP, the APIs also aim to support the case where
    the whole model is partitioned into multiple EP's subgraphs. (Note: i
    haven't fully tested the partitioning case, please help review it)
    - Modify TRT EP plugin to use those APIs.
    chilo-ms authored Nov 13, 2024
    Configuration menu
    Copy the full SHA
    afe92e1 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. Configuration menu
    Copy the full SHA
    63f8774 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2024

  1. Configuration menu
    Copy the full SHA
    bf359a1 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2024

  1. [TensorRT EP Plugin] Add cuda::Impl_Cast (#22908)

    TRT 8 doesn't support INT64 and DOUBLE data type.
    TRT 10 doesn't support DOUBLE data type.
    
    Therefore, TRT EP internally needs to convert INT64 to INT32, and DOUBLE
    to FLOAT, which needs the cuda::Impl_Cast function.
    The implementation is copied from CUDA EP.
    chilo-ms authored Nov 20, 2024
    Configuration menu
    Copy the full SHA
    c267ea5 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2024

  1. Configuration menu
    Copy the full SHA
    72afdc4 View commit details
    Browse the repository at this point in the history

Commits on Dec 3, 2024

  1. Do not expose OrtGraph

    jslhcl committed Dec 3, 2024
    Configuration menu
    Copy the full SHA
    6822206 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c8ddc73 View commit details
    Browse the repository at this point in the history