Joystick Interposer evdev support, fixes SDL, Wine, Retroarch, RPCS3, and more.

[danisla]

Changes

• Fixes joystick interposer compatibility with Retroarch, wine, and RPCS3.
• All 4 interposed joysticks are now initialized automatically on startup.
• Intercepts calls to open64 and makes socket non-blocking when added to an epoll.
• Implements support for interposing /dev/input/event* joystick devices.
• The SDL_JOYSTICK_DEVICE workaround for SDL apps is no longer needed.
• Fixes Joystick Interposer cannot be detected as a controller device in Steam Proton/Wine and RPCS3 #168
• Fixes issues with multiple joysticks.
• Fixed minor bug with cursor monitor thread not starting up.

Application Specific

General

All interposed joysticks will now appear connected even if no joystick is connected to the browser. This makes for a better hotplugging experience.

Retroarch

To use with retroarch, edit your retroarch.cfg (at ~/.config/retroarch/retroarch.cfg) and set:

input_joypad_driver = "linuxraw"

RPCS3

Select SDL as controller handler then select Selkies Controller 1 from the device list.

Wine

Run wine control and then open the Game Controllers app to test joystick support.

Testing

nvidia-egl-desktop:24.04

Dockerfile.egl.jstest:

FROM ghcr.io/selkies-project/nvidia-egl-desktop:24.04
ARG CACHEBUST

USER root

# Update interposer library and selkies python
RUN cd /opt && git clone https://github.com/selkies-project/selkies-gstreamer.git -b fix-js-interposer
RUN \
    cd /opt/selkies-gstreamer && \
    cp src/selkies_gstreamer/* src/selkies_gstreamer/webrtc_input.py /usr/local/lib/python3.*/dist-packages/selkies_gstreamer/
RUN \
    cd /opt/selkies-gstreamer/addons/js-interposer && \
    make deps && \
    make
# Workaround until entrypoint segfault is resolved.
RUN \
    echo 'cp /opt/selkies-gstreamer/addons/js-interposer/selkies_joystick_interposer.so /usr/lib/x86_64-linux-gnu/selkies_joystick_interposer.so' >> /etc/bash.bashrc && \
    echo 'cp /opt/selkies-gstreamer/addons/js-interposer/selkies_joystick_interposer_i386.so /usr/lib/i386-linux-gnu/selkies_joystick_interposer.so' >> /etc/bash.bashrc

# Update entrypoint scripts
RUN \
    cd /opt && git clone https://github.com/selkies-project/docker-nvidia-egl-desktop.git -b update-joystick-support && \
    cd docker-nvidia-egl-desktop && \
    cp entrypoint.sh /etc/entrypoint.sh && \
    cp selkies-gstreamer-entrypoint.sh /etc/selkies-gstreamer-entrypoint.sh && \
    chmod +x /etc/entrypoint.sh /etc/selkies-gstreamer-entrypoint.sh
    
USER ubuntu

Build and run:

docker build --build-arg CACHEBUST=$(date +%s) -f Dockerfile.egl.jstest -t ghcr.io/selkies-project/nvidia-egl-desktop:24.04-jstest .

docker run --rm --name egl -it -d --gpus 1 --tmpfs /dev/shm:rw -e TZ=UTC -e DISPLAY_SIZEW=1920 -e DISPLAY_SIZEH=1080 -e DISPLAY_REFRESH=60 -e DISPLAY_DPI=96 -e DISPLAY_CDEPTH=24 -e SELKIES_ENABLE_BASIC_AUTH=false -e SELKIES_ENCODER=nvh264enc -e SELKIES_VIDEO_BITRATE=8000 -e SELKIES_FRAMERATE=60 -e SELKIES_AUDIO_BITRATE=128000 -p 8080:8080 ghcr.io/selkies-project/nvidia-egl-desktop:24.04-jstest

gst-py-example:main-ubuntu24.04

Dockerfile.example.jstest

FROM ghcr.io/selkies-project/selkies-gstreamer/gst-py-example:main-ubuntu24.04
ARG CACHEBUST

USER root

# Update interposer library and selkies python
RUN cd /opt && git clone https://github.com/selkies-project/selkies-gstreamer.git -b fix-js-interposer
RUN \
    cd /opt/selkies-gstreamer && \
    cp src/selkies_gstreamer/* /usr/local/lib/python3.*/dist-packages/selkies_gstreamer/
RUN \
    cd /opt/selkies-gstreamer/addons/js-interposer && \
    make deps && \
    make
RUN \
    cp /opt/selkies-gstreamer/addons/js-interposer/selkies_joystick_interposer.so /usr/lib/x86_64-linux-gnu/selkies_joystick_interposer.so && \
    cp /opt/selkies-gstreamer/addons/js-interposer/selkies_joystick_interposer_i386.so /usr/lib/i386-linux-gnu/selkies_joystick_interposer.so

# Update entrypoint script
RUN \
    cp /opt/selkies-gstreamer/addons/example/entrypoint.sh /etc/entrypoint.sh && \
    cp /opt/selkies-gstreamer/addons/example/selkies-gstreamer-entrypoint.sh /etc/selkies-gstreamer-entrypoint.sh
    
USER ubuntu

Build:

docker build --build-arg CACHEBUST=$(date +%s) -f Dockerfile.example.jstest -t ghcr.io/selkies-project/selkies-gstreamer/gst-py-example:main-ubuntu24.04-jstest .

Run:

docker run --name selkies -it -d --rm -e SELKIES_ENABLE_BASIC_AUTH=false -e SELKIES_TURN_PROTOCOL=udp -e SELKIES_TURN_PORT=3478 -e TURN_MIN_PORT=65534 -e TURN_MAX_PORT=65535 -p 8080:8080 -p 3478:3478 -p 3478:3478/udp -p 65534-65535:65534-65535 -p 65534-65535:65534-65535/udp ghcr.io/selkies-project/selkies-gstreamer/gst-py-example:main-ubuntu24.04-jstest

[ehfd]

@danisla Feel free to switch from Draft PR if you are ready.

[danisla]

@danisla Feel free to switch from Draft PR if you are ready.

@ehfd I temporarily remove the Conga build steps because they were failing, can you help resolve the errors? You can run the conga docker build locally and see the same failures.

[ehfd]

That is in order for the near future. I know the source of error.

[danisla]

That is in order for the near future. I know the source of error.

Ok maybe we’ll just leave it disabled for now and you can re-enable it the build steps later.

[ehfd]

Anyways, when you are ready, press "Ready for review" and I'll directly make necessary edits.

[danisla]

I'm seeing a segfault during the entrypoint when running with the nvidia-egl-desktop:24.04 image.

/etc/entrypoint.sh: line 89:   200 Segmentation fault      (core dumped) /usr/bin/Xvfb "${DISPLAY}" -screen 0 "8192x4096x${DISPLAY_CDEPTH}" -dpi "${DISPLAY_DPI}" +extension "COMPOSITE" +extension "DAMAGE" +extension "GLX" +extension "RANDR" +extension "RENDER" +extension "MIT-SHM" +extension "XFIXES" +extension "XTEST" +iglx +render -nolisten "tcp" -ac -noreset -shmem

It looks like the LD_PRELOAD is interfering with the entrypoint.

Moving this back to draft to fix the entrypoint issue.

[ehfd]

#175 #171

@danisla I am seeing that this issue might also be relevant, although separate.

The concern I have with #171 is whether the below can be made more simple in any way; need a sufficient understanding of the websockets package. This is mostly redundant except allows 'body' type to be either bytes or string.

    def custom_response(self, status, custom_headers, body):
        """A wrapper indentical to https://github.com/python-websockets/websockets/blob/main/src/websockets/server.py#L482 
        but allows 'body' type to be either bytes or string.
        """
        status = http.HTTPStatus(status)
        headers = Headers(
            [
                ("Date", email.utils.formatdate(usegmt=True)),
                ("Connection", "close"),
                ("Content-Length", str(len(body))),
                ("Content-Type", "text/plain; charset=utf-8"),
            ]
        )

        # overriding and appending headers if provided
        for key, value in custom_headers:
            if headers.get(key) is not None:
                del headers[key]
            headers[key] = value

        # Expecting bytes, but if it's string then convert to bytes
        if isinstance(body, str):
            body = body.encode()
        return Response(status.value, status.phrase, headers, body)

[danisla]

#175 #171

@danisla I am seeing that this issue might also be relevant, although separate.

The concern I have with #171 is whether the below can be made more simple in any way; need a sufficient understanding of the websockets package. This is mostly redundant except allows 'body' type to be either bytes or string.

    def custom_response(self, status, custom_headers, body):
        """A wrapper indentical to https://github.com/python-websockets/websockets/blob/main/src/websockets/server.py#L482 
        but allows 'body' type to be either bytes or string.
        """
        status = http.HTTPStatus(status)
        headers = Headers(
            [
                ("Date", email.utils.formatdate(usegmt=True)),
                ("Connection", "close"),
                ("Content-Length", str(len(body))),
                ("Content-Type", "text/plain; charset=utf-8"),
            ]
        )

        # overriding and appending headers if provided
        for key, value in custom_headers:
            if headers.get(key) is not None:
                del headers[key]
            headers[key] = value

        # Expecting bytes, but if it's string then convert to bytes
        if isinstance(body, str):
            body = body.encode()
        return Response(status.value, status.phrase, headers, body)

This doesn't seem related to the joystick interposer function, so we'll solve it in the respective issue.

[ehfd]

Yeah, just for reference.

[ehfd]

@danisla For the purpose of documentation, what are the exhaustive interface (and/or driver) options that this interposer now supports?

(e.g. joydev, udev/evdev, sdl1/2, hid, etc...)

And can one use udev and sdl2 options here if one chooses so? https://docs.libretro.com/guides/input-controller-drivers/#linux

[danisla]

@danisla For the purpose of documentation, what are the exhaustive interface (and/or driver) options that this interposer now supports?

(e.g. joydev, udev/evdev, sdl1/2, hid, etc...)

And can one use udev and sdl2 options here if one chooses so? https://docs.libretro.com/guides/input-controller-drivers/#linux

udev is not supported. only evdev devices, so sdl and direct raw joystick drivers.

[ehfd]

Applications using libudev will fail when relying on sysfs (/sys), but will work when only probing evdev devices from /dev/input/event*. In most cases, users are advised to first use the SDL or joydev/linuxraw interfaces without such issues, if available.

[ehfd]

Discussion with @ABeltramo (A) and @danisla (D), myself (E):

A: It wouldn't work with a ton of modern games running on Steam
E: What's the technical reason?

A: There are 2 main issues on the top of my head:
There's not a lot of written info about how Proton works, you'll have to mostly dig up Github issues/discussions or just read the code yourself, but, depending on the game it'll require to access joypads directly via hidraw and not via evdev. So some games would just not even see your virtual joypads.
On top of that, for some games, Steam Input is a hard requirement to have joypads working. The way Valve implemented it is by using uinput (funny, right?) so without that exposed you can't use Steam Input and without that a lot of games would just not work.

For Number 1:
A: There's not a lot of written info about how Proton works, you'll have to mostly dig up Github issues/discussions or just read the code yourself, but, depending on the game it'll require to access joypads directly via hidraw and not via evdev. So some games would just not even see your virtual joypads.

E:
Launch the registry editor in the relevant Wine prefix
Navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\winebus
Set DisableHidraw to 0 (-> 1?)

https://nick.tay.blue/2024/01/21/wine-dualsense/

What would this do, here?

A: Nope, that's not how Proton and the hidraw stuff works. Proton uses SDL when hidraw is not available. There are games that will require access to the hidraw device because that's how they work on Windows: they access the raw USB device directly and don't go via xinput

E: The last lines of the link you sent me says differently, unless I'm reading it incorrectly:
https://github.com/GloriousEggroll/proton-ge-custom/blob/master/docs/CONTROLLERS.md

One final snag is that many distros do not allow user access to hidraw devices. Steam ships some udev rules to allow this for certain common controller types, but not most. In other words, your user may not have access to the hidraw device for your controller, especially if it is a less well-known controller. In those cases, we access it through SDL2 via its linux js backend and try to treat it as an Xbox controller, even if it is not mapped with the Steam client mapping feature.

A: I'm not sure what to say.. If you try out games in Proton you'll quickly see that is not always the case, just having /dev/input/event* present there is usually not enough.
I suspect the Steam virtual joypad is treated differently here and probably why sometimes you have to enable that to have even basic joypad working..
IIRC it's handled differently in SDL too, it's a completely different backend than say evdev or hidraw
I should refresh my memory though, it has been a long while since I've last dived into this

For Number 2:
A: On top of that, for some games, Steam Input is a hard requirement to have joypads working. The way Valve implemented it is by using uinput (funny, right?) so without that exposed you can't use Steam Input and without that a lot of games would just not work.

E: May the below improve things (i.e. reduce the percentage of games where this is an impasse)? https://steamcommunity.com/groups/SteamClientBeta/discussions/3/3123786356703008701/?ctp=4

I'm happy to report I basically got them to finally relent and give you that option 🙂

You have two options:

ValvePlug
Disables Steam Input using a registry key
Valve's new -nojoy client launch command
Awkward to setup

Either will get the job done, I prefer my own ValvePlug software because it's integrated with Special K. I can enable Steam Input (as if), then restart the Steam client from SKIF.

https://github.com/SpecialKO/ValvePlug

A: If you don't expose /dev/uinput Steam will run with Steam Input disabled, or you can easily disable it in the settings. That doesn't mean that games that require it will have a working joypad

E: So you mean that in any game that doesn't use xinput and requires raw gamepads in Windows will all use hidraw in Wine?

A: Ok found what I was looking for, there's something here: https://github.com/GloriousEggroll/proton-ge-custom/blob/master/docs/CONTROLLERS.md

E: What's the percentage of case 1 and case 2 in general? Any representative offline AAA games?

A: I don't want to give you a fake number I obviously don't have stats on this. For example any recent Sony exclusive game (Helldivers, Horizon, God of war) will probably have issues with your implementation

E: So basically God of War's Windows port?

A: Yep, correct. Ragnarok doesn't pick up some joypads even on a normal desktop if you don't enable Steam Input. Not sure if that's just specific to the DualSense controller (what I've tested) or it also applies to the Xbox joypads
Generally, you'll encounter issues with your implementation depending on what the game supports. It's pretty well explained why in those two links that I've shared above

E: Our experiences are generally not from Steam (Wine + GoG) so yeah I understand that there are substantial edge cases. f you bring us back data from an Xbox-compatible controller in your setup, we'll be much more convinced about this.

A: It's fairly well known that Steam Input is required for some games, I do have the same issues on my Steam Deck for example when using the integrated joypad which isn't a PS pad. I can quickly try out the Xbox one joypad on GOW Ragnarock in a minute..

E: It might also be possible that different Proton variants may produce different results.

A: Ragnarock with a real xbox one joypad works without steam input.

For Number 3:
D: If he has a way to dynamically mount host devices into running containers his approach would work for us. The only way I’ve done this is with my uinput-device-plugin which requires a background process with host access.

A: Yep, we've got that with hotplug in docker since 2023/11

E: Will containerd have any additional reason not to be possible compared to Docker?

A: The only thing needed is mknod it shouldn't be an issue with containerd
There's also this discussion which was very informative for me flatpak/xdg-desktop-portal#536

D: exposing /dev/uinput also requires exposing the entire /dev/input/ directory to all continers. when you create a device with uinput, it also creates a /dev/input/eventX device. The uinput-device-plugin would use nsenter to wire these mounts together when a new uinput device was created, so /dev/input/event100 on the host would get mapped to something like /dev/input/event0 in the container.
I think you understand the mounting device mounting issue right?

A: That is not technically correct. For devices that you create via uinput you can mknod just the devices that you control without exposing the full /dev/input. This is what we do in Wolf
For Steam Input there are ways to achieve the same

E: Relevant information contained within the games-on-whales/wolf#81 issue.

[srinidhikrs]

The following link points to information on implementation of virtual xbox joypad ( using libevdev and moonlight protocol) and steam recognizes it without issues.

https://abeltra.me/blog/inputtino-uhid-2/

https://github.com/games-on-whales/wolf/blob/stable/src/moonlight-server/control/input_handler.cpp

https://github.com/games-on-whales/inputtino/blob/stable/src/uinput/joypad_xbox.cpp

[ehfd]

@srinidhikrs

As you might know, @ABeltramo is the developer of the projects you identified, and we have had a very constructive discussion above. We will keep on communicating with him regarding the landscape around this in general.

[ehfd]

More relevant resources (need to identify the stack): https://github.com/ShadowBlip/InputPlumber

Feedback by @danisla

I just met with the InputPlumber devs in person. They have a cool project with similar goals and approaches. For containers, it’s heavily dependent on mounting udev, uhid, uinput and /dev/input from the host. So needs a lot of work to adapt for multi-container.

Their primary use case is handheld gaming devices. They had about 10 of them on display at the booth.

[ehfd]

Method to create a mount on sysfs for udev support.

https://github.com/colinmarc/magic-mirror/blob/7fa6f67092fcc3a149c42a3a2a2f707f35741625/mm-server/src/session/input/udevfs.rs#L184