are the two java dependencies mutually exclusive?

[siddhsql]

when using the onnx runtime from java are the two dependencies onnxruntime and onnxruntime_gpu mutually exclusive? i ask because this page says to include only one of them. if i want to use gpu if available but fallback to cpu if it is not available then should i include both dependencies or only one (i am guessing onnxruntime_gpu)?

[Craigacp]

The CPU package includes native binaries for more platforms (e.g. macOS ARM64), and the GPU package only includes binaries for Windows & Linux x86_64. The GPU package will run on CPU for the included platforms just fine. The Java code in both is the same, but if you put both of them on your classpath it's not clear which native libraries will load and that can cause trouble.

[siddhsql]

thanks. Is it possible to revamp the code so that it dynamically downloads only the necessary native dlls for the specific platform at runtime? i.e., at runtime it should detect the platform and download the bare minimum necessary dlls for that specific platform. I think this is what DJL does for example. When I ran it on a mac, it downloaded these libraries at runtime: $ tree ~/.djl.ai/pytorch/2.1.1-cpu-osx-aarch64 /Users/xxx/.djl.ai/pytorch/2.1.1-cpu-osx-aarch64 ├── 0.27.0-libdjl_torch.dylib ├── libc10.dylib ├── libtorch.dylib └── libtorch_cpu.dylib and when I ran it on Linux it downloaded a different set of dlls. In both cases,* the maven dependency was the same*: <!-- https://mvnrepository.com/artifact/ai.djl.pytorch/pytorch-engine --> <dependency> <groupId>ai.djl.pytorch</groupId> <artifactId>pytorch-engine</artifactId> </dependency> I would expect this to be the smart thing to do. 1. It avoids unnecessary bloat. why should I be downloading a massive jar with windows and linux dlls in it when I am running on a mac? 2. more importantly, developer does not have to add different dependencies to their application. developer does not know what platform their application will be deployed on. the other piece of feedback is that when I unzipped the jars (both using 1.17.3 version) the dlls don't match. onnxruntime has 15469240 Apr 5 19:46 onnxruntime/ai/onnxruntime/native/linux-x64/libonnxruntime.so whereas onnxruntime_gpu has 19260592 Apr 7 06:51 onnxruntime_gpu/ai/onnxruntime/native/linux-x64/libonnxruntime.so

…

On Fri, May 10, 2024 at 5:47 PM Adam Pocock ***@***.***> wrote: The CPU package includes native binaries for more platforms (e.g. macOS ARM64), and the GPU package only includes binaries for Windows & Linux x86_64. The GPU package will run on CPU for the included platforms just fine. The Java code in both is the same, but if you put both of them on your classpath it's not clear which native libraries will load and that can cause trouble. — Reply to this email directly, view it on GitHub <#20649 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A6NWEKYY22M56I7VPMCM3M3ZBVTBJAVCNFSM6AAAAABHRJXQG2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGOJQHA2TQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Craigacp]

No, we're not going to rewrite it to dynamically download binaries. That would cause problems in a bunch of places where it's deployed which don't necessarily have access to the public internet or aren't allowed to run externally created binaries. Splitting it out into separate binaries and having the developer pick which ones they need is potentially possible, but doesn't seem worth the additional effort and complexity for developers.

The binaries don't match as they are compiled with different flags.