OpenXR: Add support for Foveated Inset

[BastiaanOlij]

This PR implements support for OpenXR Foveated inset rendering if supported by the hardware.

When using foveated inset rendering, the main viewport output renders a high FOV, lower resolution render which is augmented with a separate high resolution render pass at the users focal point.

A good overview of the technique is explained on Varjos website here.

In order for the user to have quality control and ensure the camera settings can be configured, this PR introduces a new viewport node that controls the rendering of the foveated inset:

An extra XROrigin3D node is required with a few lines of code so it is placed in the same location as the main XROrigin3D node.
We also have a separate XRCamera3D node with it's tracker' set to 'inset (note that I am still working on a nice solution for the dropdown here).

Todos:

• Loads of testing (few runtimes support this at the moment so this will be tricky).
• Make sure we have proper feedback if inset rendering is enabled but our viewport is not supplied or incorrectly configured.
• Figure out whether we need to worry about this being used in conjunction with frame synthesis and/or VRS.
• Improve tracker selection dropdown (might become a separate PR).

This logic is all based on #115799 (merged) and #116424

A sample project can be found here.

This PR supersedes #81505 and #108156

[dsnopek]

Thanks!

In order for the user to have quality control and ensure the camera settings can be configured, this PR introduces a new viewport node that controls the rendering of the foveated inset:

Requiring the user to add these nodes isn't ideal. You mention that it's needed for the user to have control - what exactly do they need to control? The near and far camera plane is one thing, but I'm drawing a blank on anything else.

Could we have project settings for the configurable things? Or, if that's too much, maybe an "automatic" setting that sets up a reasonable default (just copying from the current camera and XR origin), which folks can disable, and make their own nodes if they need more control?

Also, I worry a bit about having the 2nd XROrigin3D node. When unloading an old scene (which has the main XROrigin3D node) and loading another, I think it might be possible for the foveated inset one to become "current" because I don't think we are segregating XROrigin3D nodes by viewport (like we do for cameras).

Perhaps we should be segregating XROrigin3D nodes by viewport?

[dsnopek]

Falling back to the first supported view configuration worries me a little bit. If the user picked "Stereo with Foveated Inset", this should fall back on stereo. If the device supports mono and stereo, and mono happens to be first, we shouldn't fallback on mono.

This will probably be OK, since the spec says "The returned list of primary view configurations should be in order from what the runtime considered highest to lowest user preference. Thus the first enumerated view configuration type should be the one the runtime prefers the application to use if possible."

But it still worries me a bit - an explicit fallback from "Stereo with Foveated Inset" -> "Stereo" would be better

[dsnopek]

I did a little bit of testing with the Samsung Galaxy XR using your test project.

In order for the headset to report that it supports XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO_WITH_FOVEATED_INSET, it appears you need to enable "Use Experimental Features" in the "Android XR Features" section of Export Settings

For me, rendering is pretty broken, but it is moving an inset area around based on where I am looking. So, it seems like the headset has switched into the right mode.

Here's a screenshot:

And I'm seeing this error repeatedly in the log:

E 0:04:15:806   render_camera: Unsupported camera setup.
  <C++ Error>   Method/function failed.
  <C++ Source>  servers/rendering/renderer_scene_cull.cpp:2731 @ render_camera()

[BastiaanOlij]

Requiring the user to add these nodes isn't ideal. You mention that it's needed for the user to have control - what exactly do they need to control? The near and far camera plane is one thing, but I'm drawing a blank on anything else.

This is one of the things I got stuck with last time. We don't have access to the camera used for the main view, so any camera attributes or environment overrides applied to the main camera, cant' be automatically duplicated on the inset camera.
Even if we could, we can't make the assumption. With eye tracked foveated insets, we could actually lower the quality settings on the main camera, and increase the quality settings on the inset camera.

One idea I'm toying with is to create the XROrigin3D and XRCamera3D nodes within the viewport node, but exposing the additional environment and camera attributes setting on this viewport.

Because I do agree with you, demanding this is troubled. Still, this was the quickest way I could get to testing the heart of it all.

[BastiaanOlij]

I did a little bit of testing with the Samsung Galaxy XR using your test project.

In order for the headset to report that it supports XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO_WITH_FOVEATED_INSET, it appears you need to enable "Use Experimental Features" in the "Android XR Features" section of Export Settings

For me, rendering is pretty broken, but it is moving an inset area around based on where I am looking. So, it seems like the headset has switched into the right mode.

Here's a screenshot:

And I'm seeing this error repeatedly in the log:

E 0:04:15:806   render_camera: Unsupported camera setup.
  <C++ Error>   Method/function failed.
  <C++ Source>  servers/rendering/renderer_scene_cull.cpp:2731 @ render_camera()

Lol, you got further then me already :) I thought Valve supported the extension but seems not, so I need to dust off my Varjo XR3 before I can actually run through things.

The Unsupported camera setup. sounds like Samsung is encoding the eye tracked offset into the projection matrix, something that would break lighting and that we are explicitly testing for. Varjo properly encodes that into the view matrices. I could be wrong though, I'll need to test on AndroidXR as well but seeing the whole Android push to test thing, I don't plan to do that until I have everything working on Vario.

Do you know if AndroidXR supports this through their streaming app? I need to try that one out too.

[dsnopek]

@BastiaanOlij:

This is one of the things I got stuck with last time. We don't have access to the camera used for the main view, so any camera attributes or environment overrides applied to the main camera, cant' be automatically duplicated on the inset camera.

Maybe we could have a property on XRCamera3D which allows developers to mark it as the primary camera? Then in whatever automatic setup we have could find that camera and copy settings from it? And the first XRCamera3D added to the scene could automatically be marked as primary, and the one used for the foveated inset could be explicitly marked as not primary

One idea I'm toying with is to create the XROrigin3D and XRCamera3D nodes within the viewport node, but exposing the additional environment and camera attributes setting on this viewport.

Ooo, yeah, having the OpenXRFoveatedInsetViewport be responsible for creating these nodes is a good idea. And if we didn't want to do my idea above about marking a XRCamera3D as primary, then we could maybe have a property on the OpenXRFoveatedInsetViewport to point to the primary camera?

Although, I'd really prefer if the user didn't even need to create the OpenXRFoveatedInsetViewport manually in the case that they don't want to customize anything. Ideally, we'd just be able to configure that we want to use this in Project Settings, and then it would just work for 90% of use cases, similar to how it works with the XR_FB_foveation extension.

Do you know if AndroidXR supports this through their streaming app?

Unfortunately, I don't know. I haven't had a chance to switch to Windows recently, so I've just been testing by deploying to the headset

[BastiaanOlij]

Although, I'd really prefer if the user didn't even need to create the OpenXRFoveatedInsetViewport manually in the case that they don't want to customize anything. Ideally, we'd just be able to configure that we want to use this in Project Settings, and then it would just work for 90% of use cases, similar to how it works with the XR_FB_foveation extension.

It's a real PITA, it's something I really got stuck on in my other PR, you can have a look at what sort of uglyness I had to come up with. That's where I just created everything directly on the rendering server.

The problem is that we're storing too much info on the Viewport node that is leading, and we have no access to this information in the XRServer nor rendering server.

So I figured that what we're doing with composition layers, made more sense here, just add a node so people can configure what needs to be configured.

[BastiaanOlij]

@dsnopek I think I've found all the problems that were keeping it from working correctly, so you can try the updated build and see if it works. I've been testing on the XR3, I'll do some testing on AndroidXR tomorrow.

Also the test project has a few updates so be sure to pull that. I've added a transparent plane so there is a color difference for the inset render, just so we can see if it works. From your earlier testing I have a feeling AndroidXR has the inset as the first 2 views, and context as the last 2 views, which goes against the OpenXR specification and will give us problems if we do want to change quality differences between the two.

I have not changed the way the viewports work, I haven't had time for that yet. But all in all I think the heart of this is beating strong.

[dsnopek]

After making adjustments for the issue described in my comment above (re XR_VARJO_foveated_rendering depending on XR_VARJO_quad_views) this is working pretty good on Android XR! Although, it doesn't work 100% perfectly

Here's a new screenshot:

The inset is slightly darkened, so I think that means the right views are being used for the inset?

The two issues that I see in my testing:

• The rendering of the inset doesn't always look right depending on where I'm looking, for example, if I look at the lower-left corner, the lower-left edge of the inset will get kinda distorted
• Every once in a while, the left or right eye is rendering something incorrectly for the inset, which appears as flickering. I'm not sure what triggers this, it seems to happen randomly

However, it's possible these are issues on the side of Android XR's compositor - this still requires their experimental flag afterall, and may have some bugs

[BastiaanOlij]

The inset is slightly darkened, so I think that means the right views are being used for the inset?

This is on purpose for testing, note the MeshInstance child of the inset XRCamera3D. Just hide that and your darkened area goes away :)

The two issues that I see in my testing:

• The rendering of the inset doesn't always look right depending on where I'm looking, for example, if I look at the lower-left corner, the lower-left edge of the inset will get kinda distorted

That sounds like a runtime problem. We don't really control how things are blended. Sadly for some reason my eye tracking on my XR3 isn't functioning atm so I can't test if Varjo has the same issue. I'll ask Denny if he can verify.

• Every once in a while, the left or right eye is rendering something incorrectly for the inset, which appears as flickering. I'm not sure what triggers this, it seems to happen randomly

No idea, maybe something related to the swapchain not being accessible? Any output in the logs?

However, it's possible these are issues on the side of Android XR's compositor - this still requires their experimental flag afterall, and may have some bugs

Yeah I think once we have a proper test build, we should ask someone on the AndroidXR team to have a look.

[BastiaanOlij]

Ok, this is nearly done.

@dsnopek I've changed the viewport node so it creates all that is needed, and made the extension create the node and add it to the root of the scene tree.
I did run into a problem that the scene tree doesn't exist yet on OpenXR session creation, so it's moved to our session begun logic. It's a decent compromise but makes it a little more tricky to update settings.
I haven't added any project settings for the viewport, they can be set through code if needed.

I've updated the demo accordingly.

One problem we have discovered that is not a fault of this PR, but likely an issue that has been around for a long time but less noticeable for the main XR camera. The combined frustum that we use for culling is too small and can cull objects at the edge of the inset view causing very noticeable visual artifacts.
I've worked around it in the demo by increasing the AABB of the objects in the scene but the real solution is to enlarge the cull frustum. I need to do some more experimentation here.

[dsnopek]

Thanks! This is working great in my testing on the Samsung Galaxy XR, using the latest from this PR and your demo project

I've changed the viewport node so it creates all that is needed, and made the extension create the node and add it to the root of the scene tree.

This looks awesome! I think the only thing that's missing is a setting to prevent it from doing this, so that folks can do it manually if they want.

The combined frustum that we use for culling is too small and can cull objects at the edge of the inset view causing very noticeable visual artifacts.

Ah, now that you say this, I can absolutely tell this is the cause of the flickering I was seeing. It happens when I move my cube hands to the edge of the inset

[dsnopek]

I think the note should mention that we are doing this for Varjo specifically.

For Android XR, we don't need to enable either of these extensions when using OpenXR 1.1 (although, it does support them and we could use them with OpenXR 1.0).

So, we're specifically waiting for Varjo to support the OpenXR 1.1 feature and update their XR_VARJO_foveated_rendering extension for it.

[BastiaanOlij]

I was under the impression that XR_VARJO_foveated_rendering is what triggers an eye tracked foveated inset? Or is that something I misunderstood. I'm not sure what enabling this on AndroidXR adds then?

[dsnopek]

On Android XR, when using OpenXR 1.1, you don't need to enable either extension - using foveated inset will do everything, even with eye tracking.

But if you do try to enable XR_VARJO_foveated_rendering without also enabling XR_VARJO_quad_views then the OpenXR session will fail to create (because of the dependency).

On Android XR, those extension appear to only be there for OpenXR 1.0 compatibility (although, its fine to enable them in OpenXR 1.1)

[dhoverb]

Maybe mention [method OpenXRExtensionWrapper.on_state_ready] instead of "begun"?

[BastiaanOlij]

The problem is that we emit a signal called session_begun for historic reasons. So that is what the user reading these docs sees. That we internally are now following the session states as named by OpenXR is hidden to the developer.

[dsnopek]

With the latest changes, this is looking great to me!

I still think that comment should mention that we are waiting for Varjo specifically (since Android XR can do everything with just OpenXR 1.1), but that's totally in nitpicking territory :-)

[BastiaanOlij]

Note: the remaining style check CI issue is a false positive.

I think this is because the Window class is forward declared so it probably thinks that is enough. But without including window.h we will get a modules\openxr\extensions\openxr_foveated_inset_extension.cpp(109): error C2440: 'initializing': cannot convert from 'Window *' to 'Node *' compile error.

[akien-mga]

Note: the remaining style check CI issue is a false positive.

I think this is because the Window class is forward declared so it probably thinks that is enough. But without including window.h we will get a modules\openxr\extensions\openxr_foveated_inset_extension.cpp(109): error C2440: 'initializing': cannot convert from 'Window *' to 'Node *' compile error.

You can work it around by adding a pragma comment:

#include "scene/main/window.h" // IWYU pragma: keep. Used via `Node::get_window()`.

clangd should be able to notice this normally but we seem to be hitting an edge case.

[m4gr3d]

Should also memdelete xr_camera and xr_origin

[dsnopek]

Actually, we don't need to! They were added as children of this node, so they'll be freed when this node is freed