Incorrect projection on passthrough geometry

[celyk]

While trying out OpenXRFbPassthroughGeometry with hole punch enabled, I noticed a distortion of the image up close to the mesh. Meanwhile, creating passthrough with shadow_to_opacity looks fine. You can see it for yourself in the Meta Passthrough XR Sample

I suspect one of the transformations in OpenXRFbPassthroughExtensionWrapper::set_geometry_instance_transform must be off

I did notice a problem in this line. Should it not be using Node3D::get_global_transform() ? Even then, I'm not sure this is the culprit

godot_openxr_vendors/plugin/src/main/cpp/classes/openxr_fb_passthrough_geometry.cpp

Lines 133 to 137 in f9096c0

	void OpenXRFbPassthroughGeometry::update_passthrough_geometry_transform() {
	if (geometry_instance) {
	OpenXRFbPassthroughExtensionWrapper::get_singleton()->set_geometry_instance_transform(geometry_instance, get_transform());
	}
	}

My hardware:
Quest 2 standalone

[celyk]

I tried a couple more things, and it appears I was right about get_transform. But the issue is more complex than that. Even if the pose of the mesh is wrong, youd think the projection would still work out, right?

The distortion is symptomatic of a slight offset in the viewpoint of each eye. I tried adding a MeshInstance3D of the same mesh to see if it aligns with the OpenXR one, and it does perfectly. This baffles me.

I'm not sure what could be going wrong with the rendering here. One theory I have is that the projection is coming out from the cameras themselves, hence the slight offset. It would make sense for e.g motion reprojection. I'm really just guessing as I am not familiar with these APIs

[celyk]

I've learned a lot investigating this. Reading https://developers.meta.com/horizon/documentation/native/android/mobile-passthrough-customization/#surface-projected-passthrough

surface geometries provided by the app should match real-world surfaces as closely as possible. If they differ significantly, users will receive conflicting depth cues, and objects may appear too small or large.

Looks like it would be projected out from the cameras then, good stuff. Where does that leave this issue? Assuming this is the intended use, I recommend clarifying the difference between projected vs reconstructed passthrough in the docs, and adding an option for reconstructed passthrough in OpenXRFbPassthroughGeometry. Floating cubes that dont represent real-world geometry should be avoided for projected passthrough. But there is one surface that will always work: the floor!

Keep in mind the Quest 2 does not provide a global mesh, so it would be hard to get an accurate mesh for projected passthrough. For a static room reconstructed passthrough tends to look pretty good and reduces choppiness. However, I do appreciate having the projected passthrough as a low level option. In fact I would love to have more tools to manipulate the projection.

[devloglogan]

I recommend clarifying the difference between projected vs reconstructed passthrough in the docs, and adding an option for reconstructed passthrough in OpenXRFbPassthroughGeometry. Floating cubes that dont represent real-world geometry should be avoided for projected passthrough. But there is one surface that will always work: the floor!

Thanks for bringing some of this to attention! Some clarification in the docs seems warranted. Adding an option for reconstruction passthrough by way of OpenXRFbPassthroughGeometry doesn't make sense to me, considering it can only be used with surface-projected passthrough.

While I do enjoy having the moving cube as a part of the passthrough, you make a good point. I'd like the sample to illustrate that moving the geometry around is possible, but it probably won't be used like that for the vast majority of cases. I will probably update this to be the floor, like you suggest.

[celyk]

it can only be used with surface-projected passthrough

Are you sure? I was thinking that geometry could be used with the reconstructed layer, since I can't find anything against it. If you find that it's allowed, it will make sense to add it as an option to OpenXRFbPassthroughGeometry. If not, thats alright then.

There is still shadow_to_opacity, which works, but it requires transparency. I'd rather have the layer options in the one Node if it is possible.