Implement multi-layer projection camera

[BastiaanOlij]

This PR implements a new projection camera mode that allows setting projections per eye/view/layer and removes the override code deep within the renderer targeted at XR use cases.
The change is meant to be transparent for the end user who configures cameras as before.
This also creates a method for developers to set any projection matrix by overriding the projection set by the built-in logic (a long outstanding wish for which many PRs have been risen before).

For XR specifically this attempts to unlock use cases such as:

• Mirrored/portal cameras that still support stereo input.
• Overriding input.
• Submitting multiple stereo layers to the compositor to enable features such as drawing hands over other composition layers.
• Likely required for foveated inset support.

Enhancements for future PRs:

• This makes storing our world origin in the XRServer obsolete as normal transform progression is now used. We should remove this in due time.
• We should deprecate RenderingServer.camera_set_perspective, RenderingServer.camera_set_orthogonal, and RenderingServer.camera_set_frustum in favour of calling RenderingServer.camera_set_projections directly from Camera3D. This does require Camera3D from receiving a signal when it's parent viewport changes size as this will change the aspect ratio. The added advantage of doing so is that we don't calculate our projection matrix every frame, and we can thus improve logic within the rendering engine to only recalculate things if the projection matrix actually changes.
• We should investigate if on the XR camera we can call RenderingServer.camera_set_projections less frequently as it's likely our projection matrices remain unchanged. Only when eye tracking is used it's possible (though still unlikely) there are adjustments that need to be made to the projection matrices per frame.
• I've had a go at this already but this change is really really big, but it would make sense to introduce a virtual BaseCamera3D class that Camera3D and XRCamera3D subclass and allows for additional custom cameras to be implemented (via GDExtension).

Important

With the removal of the camera override logic from the rendering engine, we are now determining
XR camera positioning slightly earlier in the frame render loop.
With all XR runtimes always performing a slight reprojection to correct for any tiny tracking differences
we feel that the pro's far out weigh the con's here.

Important

Deprecating get_transform_for_view and get_projection_for_view in favour of get_camera_offsets and get_camera_projections is a breaking change for XRInterfaces implemented in extensions.
These will need to be updated. While I'm thinking about adding a fallback, we still have a problem for especially TiltFive where we use a rather dirty workaround that won't work, while our changes here allow our TiltFive external to do things properly.

Implemented on top of #115799 (merged) ,#116752 (merged) and #117619
Implements part of godotengine/godot-proposals#4932

[BastiaanOlij]

Note for testing, you can use https://github.com/godotengine/godot-demo-projects/tree/master/xr/mobile_vr_interface_demo to test stereoscopic functionality.

I'm also working on this little demo project that lets you play with various projections and see whats going on:

test-projections.zip

[BastiaanOlij]

cc @huwpascoe, finally getting around to doing my version of this. I'm far from finished and have still to go through issues I flagged on your original PR. we'll see where we get to ;)

[BastiaanOlij]

I've implemented get_view_offset in MobileVRInterface and OpenXRInterface. Left the changes in a separate commit for now for easier review. The changes to OpenXR are fairly substantial BUT I think in applying this, it may have actually fixed a few things.

I also need to rebase this because I think I'm on the faulty Vulkan build where the fallback code causes issues. So for now testing with OpenXR should be done with compatibility (Mobile VR is fine).

I haven't looked into the needed changes for WebXRInterface, I'll do that next.

@dsnopek @m4gr3d might be good to already have a look at the OpenXR changes.

@celyk it would be good to see if the way I've got combined frustum working solved our shadow shaking issue as we're now working off of the actual view pose instead of calculating it back from the individual eye poses.

[celyk]

solved our shadow shaking issue

Tested with the MobileVRInterface and it does seem to fix #84510 for XR, the jitter is gone. This is consistent with our findings, but remember that I can still reproduce the issue with a Camera3D under certain conditions.

I've also tried setting the projection matrix of a Camera3D and that works nicely so far. Will get back to you with further testing of that feature, especially for XR portals and mirrors that I am a big fan of.

[BastiaanOlij]

Tested with the MobileVRInterface and it does seem to fix #84510 for XR, the jitter is gone. This is consistent with our findings, but remember that I can still reproduce the issue with a Camera3D under certain conditions.

Owh I definitely think there is more going on but this PR does fix the stability issue with how the combined frustum is calculated, which is being used to determine the center point of the shadow cascades.

In the old solution it's actually some distance behind the head where the two eye frustums meet.

[BastiaanOlij]

I took some time to improve the test project I made, which can now be found here:
https://codeberg.org/BastiaanOlij/test-projections

I purposefully added a few things that I knew are broken as we only work around them when stereo rendering is enabled:

The main one that is immediately obvious is SDFGI, where things are offset.
But there are others that are equally problematic. Less visible as I'm testing fog, the sky is wrong as it is still centered.

These are all variations around unprojecting screen coordinates through an optimisation which does not work for asymmetrical projection matrices. So the next step is to find all the effected places and instead of checking for stereo, checking for asymmetrical projections. This will likely solve many other use cases that have previously prevented us from offering more freedom around projection matrices.

[BastiaanOlij]

Spend a bunch of time investigating a number of issues that challenged a few preconceptions I've held for a while. There are a number of issues introduced with stereo rendering that I thought were related to asymmetrical projection matrices but are in fact caused by the way we handle our stereo offset. However with giving users the freedom to submit any projection matrix, we're still going to need to deal with these situations better.

Now before we look into that further, I lost a lot of time on SDFGI only to come to the conclusion that it's likely been broken for asymmetrical projection matrices for some time, unrelated to the changes I'm making here. Note here with our built-in frustum camera and zero offset, the back wall of our little hut is dark:

But move the offset by 1cm and we see that things no longer line up:

If you line up the before and after images, you'll see that screenspace wise the SDFGI data is still centered on screen.

There is another issue I ran into when the floor is metallic, first it highlights the shift even better, but also, that looks very broken (and again, not related to this PR) :)

I'll get to SDFGI later and if I can't figure it out, I'll make sure to raise some bugs.

[BastiaanOlij]

So back on the topic of asymmetrical, so I'm now testing the new logic by setting up a single projection matrix.

A projection matrix is asymmetrical when the field of view on the left is different from the right, or the top different from the bottom, which is very normal in XR scenarios, but also happens when you're dealing with things like projection matrices for screen space slanted mirrors.

Note here the "skew" applied to x and y as a factor of z (I've enabled SSAO here and cranked it up a bit):

Note that even though the camera is dead center in front of the Godot cube, our cube is to the right of the image.

Just for contrast, if we render our right eye we get the opposite effect:

I was expecting things to look broken but they actually look correct. I always made the assumption this was due to the asymmetrical projection as in various places, we take a shortcut in unprojecting.

In order to unproject a screen space coordinate, where we're only interested in the original X and Y coordinates, we only need four values from our projection matrix for basically any projection matrix:

And that is what our "shortcut" code uses.

However, we're missing an ingredient here.

In stereo rendering our eye is offset from our camera position by half our IPD.
As an optimisation we encode this offset into our projection matrix. Now we get the following situation:

These values that do effect the X and Y coordinate, aren't taken into account into the unprojection. We don't see any issues however. Our sky is projected to infinity, so a 3cm shift of the camera is not going to matter (as I previously made the mistake that the 3rd row wasn't taken into account, which does alter the results greatly, I often thought it would). The red line is right where it should be. I've still applied the fix just to be safe here.

However anything being unprojected that is in the foreground will be misplaced. I was hoping SSAO would be one of those effects but I think that is just rendered in screen space, it doesn't actually do an unproject.

The fix here btw is that we are checking if any of these values are none-zero and triggering a full unproject if that is the case:

I just wanted to document the reasoning and what we need to chase in order to fix effects that are misbehaving. Now the hunt starts for finding the actual situations where this is going wrong.

[BastiaanOlij]

The more I think about this, the more this is a problem. The unprojection in our current stereoscopic implementation works because we add our camera offset later and keep that transform.

When doing a full unproject, your vertex position is in relation to the center position of the players head, and to get the actual unprojected value we need, we add the offset back in.

If we just have a projection matrix with the eye offset (which can also include a rotation), but we don't have the offset transform, we're missing a vital ingredient.

Now the new API allows us to both provide the projection matrix and the offset, and applies the offset inside of the rendering engine,

I'm wondering if we should make a limitation that we only accept projection matrices where [0][1], [1][0], [3][0] and [3][1] are all zero and force the user to use the offset transform.

This would greatly reduce the risk of projection matrices being introduced that break.

[dsnopek]

Overall, this looks amazing! Most of my notes are on the verge of nitpicking :-)

I've tested this with OpenXR on Linux via WiVRn and on Android (Meta Quest 3 and Samsung Galaxy XR). I've also tested WebXR in the Meta browser (Quest 3). Both seemed to work fine!

[dsnopek]

Thanks!

The code is looking great to me from the perspective of XR and rendering, and it's working fine in my testing

It sounds like there may need to be some changes based on @lawnjelly's note regarding physics interpolation, but aside from that, IMO this is ready to go :-)

[devloglogan]

Tested this PR a fair bit and I wasn't bumping into anything unexpected. Explored @BastiaanOlij's projections test project, the mobile vr interface demo, as well as looked at some XR sample projects on Meta Quest 3 and Galaxy XR, as well as a recent XR jam project. It all appeared to be working correctly. 🙂