XR integration

[zmerp]

Hi, I wanted to help with XR integration in bevy. @blaind is currently working on it but I wanted to take a step back and discuss some details about how this integration should be implemented (I currently can't actively work on it but I know this can help any contributor).
First of all, we may want a unified system for handling VR, AR and XR devices. A single crate bevy_xr can be written as a unified interface for OpenXR, WebXR, ARCore and ARKit backends, but much work needs to be done in other bevy crates for a proper integration.

For now I will discuss only the rendering code paths.

1. Currently the default rendering graph includes a 2D and 3D cameras, should the XR view be added as a separate camera or should it replace the 3D camera? While for AR the current camera should suffice, for VR we need a new stereo camera. Should we force a XR camera into the node graph and control is parameters or should we let the user construct it by querying raw data from bevy_xr? If we want full control in the backend we may need a new XR_CAMERA node.
2. The primary swapchain is a WindowSwapChainNode, but OpenXR uses an offscreen ad-hoc swapchain. if XR mode is enabled, should the primary swapchain be a XrSwapChainNode or should we add a new XR_SWAP_CHAIN node? A point in favor to XR_SWAP_CHAIN is that bevy_ui uses the PRIMARY_SWAP_CHAIN and expects it to be linked to a window.
3. Swapchain manipulation in RenderResourceContext requires a Window handle. Should we add a create_swap_chain_xr() and next_swap_chain_texture_xr() methods or should we abstract Window and a (new) XrScreen into a new (enum) type?

To house the most of the XR logic I though of adding a bevy_xr crate, which should handle input and should be renderer agnostic. bevy_xr plugin struct can be used by bevy_wgpu to construct wgpu context objects and swapchains (bevy_wgpu will receive just the backend-specific handles).

About wgpu and gfx-hal, @blaind is working on wgpu and it seems on the right path. I already submitted a PR on gfx-hal for constructing objects from raw handles but it still needs some tweaks to allow gfx-hal to take ownership of these handles (which should simplify the implementation of bevy_xr). Currently there is no support for multiview in gfx-hal and wgpu, this can be added later, and bevy integration can take this into account for future support.

[cart]

A single crate bevy_xr can be written as a unified interface for OpenXR, WebXR, ARCore and ARKit backends, but much work needs to be done in other bevy crates for a proper integration.

I think ultimately this is a good idea, but I'd rather focus on getting direct 1:1 apis implemented first. Then we can identify commonality + start building abstractions (which should definitely go through the RFC process). Godot built a high level XR interface, so we can probably learn from that.

1. Currently the default rendering graph includes a 2D and 3D cameras, should the XR view be added as a separate camera or should it replace the 3D camera? While for AR the current camera should suffice, for VR we need a new stereo camera. Should we force a XR camera into the node graph and control is parameters or should we let the user construct it by querying raw data from bevy_xr? If we want full control in the backend we may need a new XR_CAMERA node.

Hmm this is a tough call. I don't have an opinion off the top of my head. We should probably build a separate path first, then try to unify if possible. Ideally users don't have to build entirely new render logic when adding vr.

2. The primary swapchain is a WindowSwapChainNode, but OpenXR uses an offscreen ad-hoc swapchain. if XR mode is enabled, should the primary swapchain be a XrSwapChainNode or should we add a new XR_SWAP_CHAIN node? A point in favor to XR_SWAP_CHAIN is that bevy_ui uses the PRIMARY_SWAP_CHAIN and expects it to be linked to a window.

We'll probably need to change the graph structure according to whether or not XR is enabled (ex: replace the window swapchain node with the xr swapchain node). This will definitely require some thought / design work and like (1), ideally users don't need to build entirely new render graphs to account for this.

3. Swapchain manipulation in RenderResourceContext requires a Window handle. Should we add a create_swap_chain_xr() and next_swap_chain_texture_xr() methods or should we abstract Window and a (new) XrScreen into a new (enum) type?

I've already removed Window from next_swap_chain_texture in my renderer experiments. But yeah we'll probably need a new api for creating the xr swapchain.

[zmerp]

Thank you for your answer.

I think ultimately this is a good idea, but I'd rather focus on getting direct 1:1 apis implemented first. Then we can identify commonality + start building abstractions (which should definitely go through the RFC process). Godot built a high level XR interface, so we can probably learn from that.

My idea was to base the XR API on OpenXR, since this is the standard that should unify other VR and AR APIs. I will definitely look into Godot.

We should probably build a separate path first, then try to unify if possible. Ideally users don't have to build entirely new render logic when adding vr.

That's a good idea. I'm just starting to learn bevy and I should be more familiar with it before I can identify the best XR implementation.

We'll probably need to change the graph structure according to whether or not XR is enabled (ex: replace the window swapchain node with the xr swapchain node).

We still may want to allow a desktop window even in VR mode, but arguably this should be optional and not enabled by default.

[zmerp]

Reiterating on (2), I was tempted to replace PRIMARY_SWAP_CHAIN from WindowSwapChainNode to an XR specific swapchain, but since node graphs can be serialized, we don't want the same node name to correspond to different behaviors (depending on a xr flag). So to be more correct we could rename PRIMARY_SWAP_CHAIN to PRIMARY_WINDOW_SWAP_CHAIN.

[JeffGillin]

I'd personally like to see bevy_arcore.
quest will be main target for many.
start small. I know it's not small to write the tooling tho.