Plans for EP for GPU acceleration under macOS?

[j-paulus]

ORT seems to be quite a capable inference backend, but in the domain of EPs, there is one gap: GPU acceleration under macOS. I am fully aware that a Mac is not the best idea when working with DNNs, but that doesn't still mean that the development / training couldn't happen on a different platform, and the final application with the inference also under macOS. Sure, with eGPU hacks an nVidia GPU is possible, but I'm thinking now in the lines of the off-the-shelf setups.

The CPUs go already quite a long way, but it would be interesting to make use of the GPUs available in some devices, e.g., the AMD Radeons built into the 16" MacBooks. They are no CUDA devices, but should still be better than nothing.

ROCm + MlGraphX doesn't seem like an option, since they are supported only under Linux.

PlaidML has / had support for some AMD GPUs and it was merged into nGraph in the Intel acquisition and apparently a short time later it was dropped from nGraph. Also nGraph has been deprecated and replaced by OpenVINO, which doesn't have any traces of AMD GPU support left.

It would seem to me (a non-expert in the matter), that ORT would need an EP for Metal to make a better use of the HW in Macs, but it is unknown if that is ever going to happen. There is of course Smelter, but could that evolve into a transparent ORT EP?

The announcement of M1 imposes a time limit on this discussion in the form that the Mac hardware with AMD GPUs will probably start slowly disappearing in the next few years. A few years is still a long time in NN world. And this opens the possibilities of using the M1 CPU / GPU / neural engine.