[Web] BiRefNet_T not working on webgpu

[guschmue]

Describe the issue

https://huggingface.co/onnx-community/BiRefNet_T does not work on webgpu

To reproduce

See https://huggingface.co/onnx-community/BiRefNet_T

Urgency

No response

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.19

Execution Provider

'webgpu' (WebGPU)

[guschmue]

I looked briefly at this: the model takes a lot of memory for activations and will not work with wasm32.
If I run it on webgpu I see it using close to 7GB gpu memory. In theory if your gpu has that kind of memory it should work BUT the model uses GatherND and ScatterND which we have not implemented for webgpu and falls back to wasm just to run out of memory on the wasm end.
We could implement GatherND and ScatterND what should make it work on high end gpu's but most won't have those high end gpu's.

[tidus2102]

hi @guschmue, thanks for your support. It's great if you can make it works on webgpu (at least with high-end gpus). About the memory, you means we can only run this model with wasm64 right?

[fs-eire]

If we can make GatherND and ScatterND working on WebGPU, by using external data side-loading it is possible to make it work in wasm32. And there is an ongoing effort to support wasm64 build.

[xenova]

Possibly related: https://huggingface.co/onnx-community/DepthPro-ONNX also throws an error for me:

An uncaught WebGPU validation error was raised: The number of storage buffers (36) in the Compute stage exceeds the maximum per-stage limit (8).

• While validating binding counts
• While validating [BindGroupLayoutDescriptor]
• While calling [Device].CreateComputePipeline([ComputePipelineDescriptor "Concat"]).

Another error:

Uncaught (in promise) Error: [WebGPU] Kernel "[Concat] /encoder/Concat_23" failed. Error: non concat dimensions must match

---

The model works correctly in Node.js (CPU)

[xenova]

https://huggingface.co/briaai/RMBG-2.0 is a new birefnet-based model for state-of-the-art background removal. Would be useful to test too!

[tidus2102]

Hi, just a kindly check if there have been any updates or progress regarding this issue? Thank you

[fs-eire]

The The number of storage buffers exceeds the maximum per-stage limit. is a known issue for Concat. this issue is being tracked.

[tidus2102]

If we can make GatherND and ScatterND working on WebGPU, by using external data side-loading it is possible to make it work in wasm32. And there is an ongoing effort to support wasm64 build.

hi, how about this?

[fs-eire]

#22847 and #22755 introduced implementation of GatherND and ScatterND. Please allow one day or 2 for the pipeline to publish a nightly package.

[tidus2102]

Hi, I've just tested the latest ORT nightly dev build but still got the memory error when inference BiRefNet_lite onnx model on Chrome 131.0.6778.205 arm64 - macOS 15.2 (24C101).

Here is the sample code.

Please help to check again. Thank you!

[xenova]

the model uses GatherND and ScatterND which we have not implemented for webgpu and falls back to wasm just to run out of memory on the wasm end.
We could implement GatherND and ScatterND what should make it work on high end gpu's but most won't have those high end gpu's.

Now that we have GatherND and ScatterND ops implemented, should this be working?

[xenova]

I've released a few more birefnet models we can use for testing: https://huggingface.co/models?library=transformers.js&other=birefnet. Unfortunately, it's still an issue today 😅