Expose where buttons and other components are located on controllers

[mrxz]

There's currently no way for a WebXR experience to query/know the positions of the buttons on the controllers. This can be useful for things like rendering tooltips to indicate what each button does. Here's an example video showcasing what that might look like: https://designingvr.org/2018/10/21/tooltips-for-controller-buttons/

In practice, some experiences are now hardcoding these button positions for specific controllers. This in turn can create usability issues when the tooltips don't align when using a different controller.

One possible solution would be to expand the webxr-input-profiles to provide this information. It already provides 3d models for controllers along with their components (incl. buttons). It almost contains the needed information, but can't be used as none of the referenced nodes are reliable indicators of the component's position:

• The rootNodeName listed in components often doesn't exist in the asset (No matching nodes for rootNodeName for many profiles/assets webxr-input-profiles#249)
• The nodes referenced by a component as its visualResponses might be offset to act as a pivot

An alternative solution is to expose this information through the WebXR Device API. This would resolve the above issue as well and even work in cases where one doesn't depend on the webxr-input-profiles. For example, an AR application where the physical controllers can be seen in the pass-through view.

[cabanier]

/agenda

[toji]

This was discussed in the most recent call.

General consensus was that it makes sense to have these points identified in the webxr-input-profiles assets. They should be exposed as GLTF nodes with a specific label that corresponds with the transform nodes used for the visualResponses. (ie: a_button_label alongside a_button_pressed_value, a_button_pressed_min, a_button_pressed_max)

In some cases these might match the existing visualResponse nodes, but in others (such as the pivot point example mentioned above) they would have to be manually placed at a more practical point, ideally the center of the surface of the component in question.

Beyond that we will want to extend the motion controllers library to easily surface the label points, and add some additional validation and rendering to the viewer to ensure the label points can be visualized. We also discussed the possibly of extract the positions of the label points from the .glb files and copy them into the profile .json for use cases like AR where the controller model doesn't need to be rendered.

It's not clear that there's an easy programmatic way to place these nodes, so it's likely that we simply need someone to go in and add them manually to every existing mesh. We have 41 different profiles with anywhere from 1-3 meshes a piece in them, so I'd guess this would mean touching ~80 .glb files. We would want all of the meshes updated in a relatively short timeframe (on the order of a few months from start to finish) so that we could release a single update to the packages that added this new data for every device. That's a tractable but reasonably large chunk of work, and we don't have anybody who is currently in a position to do it, so volunteers welcome! If anyone is interested in helping make this contribution to the community please reach out and we can coordinate efforts.

[cabanier]

@toji, did you have a chance to tackle this?

[toji]

I did not. I wouldn't mind doing it, but given that this likely represents multiple days worth of work I don't know that I'll be allocated the time to work on it any time soon.

[AdaRoseCannon]

/facetoface