Sharing model across threads

[patcollis34]

Are there any good example codebases that share ORT session or environments across threads? I'm guessing I want to load the session or environment into struct on startup and then pass that struct to the threads. seems like I may need to do something with threadpool configs on the session as well?

[decahedron1]

There are 2 main ways to do this:

1. Wrap Session in an Arc. Session is Send + Sync so you can just wrap it in an Arc and send that to different threads.
   Though if you plan to use GPU acceleration, I've had issues where using a session in multiple threads with CUDA & DirectML on Windows would cause a segfault. This might have been fixed by fix: do not lock api pointer #160 (which you'll need to add ort as a git dependency to use) but I haven't tested it.
2. Create a dedicated thread to run inference. The thread receives data from an mpsc channel, runs inference on the session, and sends the results back through oneshot channels.

I don't think there's a need for threadpool if you're only ever using one session (shared between threads or not) at a time.

[patcollis34]

cool thank you

[BinaryWarlock]

I don't think there's a need for threadpool if you're only ever using one session (shared between threads or not) at a time.

@decahedron1 Why is that? If the model doesn't support batching, won't running N threads with the same session allow you to run N inputs in parallel (on CPU executor)? Or is there another way to do that?

Asking because I'm wondering what to do in this situation.