implement `keyboard-shortcuts-inhibit` and `wlr-virtual-pointer` #630

sodiboo · 2024-08-26T20:56:40Z

My motivation for this is the use of KVM switching software, in particular i noticed that lan-mouse requires them (hence my branch name).

Two things need to be done:

Please someone check that what i am doing with pointer axis frames is sane. This was painful. It works. Smithay docs refer to a nonexistent function so i had to check 2-year-old git blames to find out what replaced it. We just ignore the axis source if it's done before an arbitrary other thing?? Help.
Need to implement a keyboard-shortcuts-inhibit release bind. The current implementation already doesn't inhibit the ChangeVt binds ever.

You can try the functionality of both of these protocols using lan-mouse --test-emulation (will cause the pointer to move in a lemniscate) or lan-mouse --test-capture (WARNING: YOUR FOCUS WILL BE UNRECOVERABLE FROM WITHIN NIRI. use Ctrl+Alt+F2 and then pkill -f test-capture to recover. the Ctrl+Alt+{F1..F12} binds are the ChangeVt binds i've mentioned already)

Another thing i've noticed is that virtual-keyboard inputs aren't ever handled by the compositor (i.e. Super+Left is passed to whatever client, instead of focusing the column to the left)? This is irrelevant to this PR, but something i noticed while using lan-mouse. This is probably a Smithay issue? Since it's uhh mostly implemented in Smithay.

YaLTeR

Smithay docs refer to a nonexistent function so i had to check 2-year-old git blames to find out what replaced it. We just ignore the axis source if it's done before an arbitrary other thing??

I haven't looked in detail but if it involves axis v120 events then yeah it's like this pretty much because of legacy, v120 events replaced the old ones: https://wayland.freedesktop.org/libinput/doc/latest/wheel-api.html

YaLTeR · 2024-08-27T13:42:57Z

src/input/mod.rs

@@ -2221,6 +2247,18 @@ fn should_intercept_key(
        }
    }

+    if is_inhibiting_shortcuts


I have a feeling that this should interact in a more complex manner with the match below because starting to inhibit while some modifier is held (and in suppressed_keys) then releasing the modifier will currently leave it in suppressed_keys, which does not seem right.

This behavior should also be tested in the unit tests below.

I've since updated the handling in a way that i think works with supressed_keys properly (and i've commented the behaviour), but to be honest, thinking about the order of events kinda makes my head hurt and i'm dreading writing the requested tests. I'm also not entirely confident that my intuition of how it should behave exactly aligns with what you think it should do? Do you wanna take a shot at writing tests for this? 🥺👉👈

Seems fine to me. For the tests, just add a few cases with the shortcuts inhibited = true. For pressing and releasing allow-inhibit, !allow-inhibit, which also check suppress keys. Basically check that a shortcut that would return Intercept() returns instead Forward if inhibited = true. And maybe also something which first does a press with inhibited = true, then release with inhibited = false in case that logic would change later.

src/input/mod.rs

sodiboo · 2024-09-07T20:56:56Z

I've implemented the allow-inhibiting property for binds, and the toggle-keyboard-shortcuts-inhibit action, which weren't present in the initial draft of this PR. This is basically the only thing that was missing from making this PR "complete". keyboard-shortcuts-inhibit is now fully implemented (should probably have some more tests).

I've also made sure you can't bind toggle-keyboard-shortcuts-inhibit but allow that bind to be inhibited. the allow-inhibiting property is ignored for this action

wlr-virtual-pointer worked flawlessly right off the bat for my use case, and you wouldn't be able to tell from using, but the underlying code is definitely janky!!! I'd like someone to verify that this is how the Axis* events are suppposed to work. Basically if we get more than one timestamp in a single frame, all but the first are ignored. And... AxisDiscrete overrides the values set by Axis? It feels very dirty, but judging by the AxisFrame struct in Smithay, it feels like the protocol should not really be allowing any of these requests to be sent twice in one frame. It feels very jank, and i know little about how axis events actually work, i just plugged together the parts of Smithay that "looked" about right to me.

The uhh... AxisSource request is optional (i think?) but PointerAxisEvent::source() does not allow a None value, so this is definitely wrong. The AxisFrame struct cannot be constructed without a timestamp, so for the AxisSource event, which doesn't include a timestamp, it's just dropped because no AxisFrame can be started; a different Axis* event must be sent first for the AxisSource to amend it.

YaLTeR

I'd like someone to verify that this is how the Axis* events are suppposed to work. Basically if we get more than one timestamp in a single frame, all but the first are ignored. And... AxisDiscrete overrides the values set by Axis? It feels very dirty, but judging by the AxisFrame struct in Smithay, it feels like the protocol should not really be allowing any of these requests to be sent twice in one frame. It feels very jank, and i know little about how axis events actually work, i just plugged together the parts of Smithay that "looked" about right to me.

The uhh... AxisSource request is optional (i think?) but PointerAxisEvent::source() does not allow a None value, so this is definitely wrong. The AxisFrame struct cannot be constructed without a timestamp, so for the AxisSource event, which doesn't include a timestamp, it's just dropped because no AxisFrame can be started; a different Axis* event must be sent first for the AxisSource to amend it.

Yeah idk tbh, I would just test with several wlr-virtual-pointer clients to see that they work. And if there's something weird, check in WAYLAND_DEBUG=1 what events they send.

(btw, what wlr-virtual-pointer clients are there apart from lan-mouse? I'll need a few to test this I guess)

YaLTeR · 2024-10-06T07:14:03Z

resources/default-config.kdl

+    //
+    // The allow-inhibiting=false property can be applied to other binds as well,
+    // which ensures niri always processes them, even when an inhibitor is active.
+    Mod+Shift+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }


I think we should bind this to Mod+Escape by default because Super+Escape is the default in GNOME: https://gitlab.gnome.org/GNOME/mutter/-/blob/d7d92c68bd5c76525752feab65c44b3157deb7f1/data/org.gnome.mutter.wayland.gschema.xml.in#L58-61

The reasoning for not doing this is so:

The primary release mechanism should be implemented in the client. It will know best what keys are in use and easily accessible. In many cases, we'd expect the client to release on Super+Escape (because inhibiting shortcuts means you're free to use Super for whatever binds, and Super+Escape is unlikely to mean anything else), but if it does mean something else then the client knows best what is an appropriate release shortcut.

The reason why i chose Mod+Shift+Escape is because that's the shortcut to open Task Manager on Windows; so it already has the connotation of "An app is misbehaving!! Help me, operating system!" in some users' minds (including my own).

The difference is, however, ultimately arbitrary. So if you disagree with the above, i'll change it to Mod+Escape.

In many cases, we'd expect the client to release on Super+Escape (because inhibiting shortcuts means you're free to use Super for whatever binds, and Super+Escape is unlikely to mean anything else), but if it does mean something else then the client knows best what is an appropriate release shortcut.

Well, my thinking is, if Super+Escape is the default on GNOME, then client devs might decide not to do anything on Super+Escape client-side and generally expect it to already work, and definitely not bind it to something other important (considering GNOME is by and large the "default desktop", and any client testing likely occurs on GNOME).

The reason why i chose Mod+Shift+Escape is because that's the shortcut to open Task Manager on Windows

Oh? I always used Ctrl+Shift+Escape (and Ctrl-Alt-Delete before that started bringing up the menu).

Oh? I always used Ctrl+Shift+Escape

actually that sounds about right. i may have misremembered and just Made Up This Shortcut (by changing the first modifier in the sequence)

then yeah, let's make it parity with GNOME. i'm still happy with that 'cause i can just use the one i prefer in my config :3

Suggested change

Mod+Shift+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }

Mod+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }

so yeah

src/handlers/mod.rs

src/input/mod.rs

YaLTeR · 2024-10-06T07:27:29Z

niri-config/src/lib.rs

                Mod+T allow-when-locked=true { spawn "alacritty"; }
                Mod+Q { close-window; }
                Mod+Shift+H { focus-monitor-left; }
                Mod+Ctrl+Shift+L { move-window-to-monitor-right; }
                Mod+Comma { consume-window-into-column; }
                Mod+1 { focus-workspace 1; }
                Mod+Shift+1 { focus-workspace "workspace-1"; }
-                Mod+Shift+E { quit skip-confirmation=true; }
+                Mod+Shift+E allow-inhibiting=false { quit skip-confirmation=true; }


I wonder if allow-inhibiting=false is a bit hard to wrap one's head around? Maybe something along prevent-inhibiting=true would be better, human config file reading and writing wise?

i don't feel like that change is an improvement, because prevent-inhibiting=true is a double negative. but also, i agree with your concern, and i think it's slightly confusing because allow-inhibiting=false is also a double negative, because the value it will be set to is false.

is there a way to name it that isn't a double negative? always-available=true, but more descriptive? borrow an idiom from CSS and call it important=true? maybe with the exclamation point to drive home the correlation? !important=true. though it reads as "not".

Hmmm, difficult question indeed. I like the word "inhibit" being there because it's the technical term for this (unless it isn't always? do some apps re-phrase it?).

If we drop inhibit, maybe something like "always-accessible" is better than "always-available"? That's kind of still not very descriptive, while it does control a very specific thing, so it would be good if it were descriptive...

src/input/mod.rs

src/protocols/virtual_pointer.rs

src/handlers/mod.rs

YaLTeR · 2024-10-06T07:51:08Z

src/input/mod.rs

@@ -2221,6 +2247,18 @@ fn should_intercept_key(
        }
    }

+    if is_inhibiting_shortcuts


Seems fine to me. For the tests, just add a few cases with the shortcuts inhibited = true. For pressing and releasing allow-inhibit, !allow-inhibit, which also check suppress keys. Basically check that a shortcut that would return Intercept() returns instead Forward if inhibited = true. And maybe also something which first does a press with inhibited = true, then release with inhibited = false in case that logic would change later.

sodiboo · 2024-10-06T12:34:26Z

(btw, what wlr-virtual-pointer clients are there apart from lan-mouse? I'll need a few to test this I guess)

just about any KVM or RDP software. Something like deskflow or input leap should do the same thing as lan-mouse. I also noticed that WayVNC works just fine with niri as of this PR, as it has several capture protocols including wlr-screencopy, so you can connect to it from wlvncc.

Most of these should probably also be using keyboard shortcuts inhibit? Unsure. I know lan-mouse does, at least.

ConsularParadi · 2024-10-19T17:38:22Z

Hi, I was trying to setup wayvnc and having some issues with wlr-virtual-pointer. Has this issue been resolved in niri-0.1.9-1. If yes, how can I set it up?

YaLTeR · 2024-10-19T18:39:45Z

wlr-virtual-pointer is part of this PR. It is not yet merged

sodiboo · 2024-10-19T20:46:42Z

I was trying to setup wayvnc [...]

Note that wayvnc still won't work smoothly once this is merged, at the very least because Smithay doesn't handdle ext-virtual-keyboard properly; all keyboard inputs are sent to the focused applications, and the compositor can't currently intercept them for purposes such as window management (e.g. you can only change focus by clicking with the mouse; over a VNC connection it will behave as if you have zero binds configured)

Currently, i'm fairly busy with a bunch of exams and other assignments at school, which is why i haven't been very actively working on this PR (and others) for some time. I'll get to it sometime, and also improve the virtual keyboard situation such that WayVNC will work properly, as i do actually also wish to use WayVNC eventually. I have no ETA on when all of this will be finished; it's a fairly broad scope of work requiring changes in Smithay's input backend and not just niri.

If you don't find the lack of keyboard shortcuts to be problematic, and still wish to use WayVNC and want to do it right now, you can just run this PR as-is, before it's merged. Its not necessarily up-to-date with the most recent version of niri, so you'd be "downgrading" slightly (not by much). I am running a branch based on this PR live on my machines and it works fine.

sodiboo · 2024-11-08T08:13:41Z

by the way:

closes Implement keyboard-shortcuts-inhibit-unstable-v1 #198

i didn't realize this was an issue that existed but it should definitely be linked to this PR.

current status: i have a lot of stuff in my backlog, including a lot of schoolwork. i'll get back to this (and my other PRs) eventually but right now it's not my top priority

the keyboard shortcuts inhibition part is actually stable and worked well last i recall, so like, if you want it right now just go merge it manually. i guess i could open another PR to separate these, because i think that one requires less work to "finish" and actually be good to merge.

sodiboo · 2024-11-23T20:30:04Z

Okay, so respecting the output parameter was actually surprisingly less work than expected. It's fairly trivial to define an extension trait for the input backend, such that we can easily expose the VirtualPointer::output() to the event handlers.

I've implemented this in the commit titled "add InputBackend extensions; use Device::output() for absolute pos events". This is the latest commit.

Virtual Pointer absolute motion when provided with an output should now work, and actually give output-relative coordinates.

I've also hooked up the touch and tablet positions to this function, falling back to the previous behaviour when it returns None, which is actually always for these event types at this time. But, this makes it easier to implement the ei protocol and also the potential to expand the config to allow per-tablet configuration of which output it maps to. Just change the impls in backend_ext.rs to make sure they return the correct Output, and everything else should just work as is. (and, maybe need to change its signature to take additional params, but besides that the callsites already work).

salman-farooq-sh · 2024-12-08T20:09:45Z

Thanks @sodiboo and @YaLTeR. What work is left in this now?

salman-farooq-sh · 2024-12-25T19:55:18Z

I am very excited for this.

YaLTeR · 2025-01-02T18:22:00Z

Remind me, what is left to do here? (Apart from the whole "virtual keyboard doesn't do compositor shortcuts" which is not really related to these code changes.) Just review and merge?

sodiboo · 2025-01-03T10:40:09Z

uihhh yeah, just review i guess. everything works i think.

YaLTeR

Some minor things left, otherwise looks good.

YaLTeR · 2025-01-05T07:10:50Z

resources/default-config.kdl

+    //
+    // The allow-inhibiting=false property can be applied to other binds as well,
+    // which ensures niri always processes them, even when an inhibitor is active.
+    Mod+Shift+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }


Suggested change

Mod+Shift+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }

Mod+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }

so yeah

YaLTeR · 2025-01-05T07:18:07Z

src/input/backend_ext.rs

+impl NiriInputDevice for WinitVirtualDevice {
+    fn output(&self, state: &State) -> Option<Output> {
+        match state.backend {
+            Backend::Winit(ref winit) => Some(winit.single_output().clone()),


Does this actually do anything? Maybe just replace the whole thing with None and remove the single_output() function?

Basically, it asserts that the mouse position on the winit backend will break if/when it ever adds another output. Like, it'd try to scale it relative to the global extents of all monitors instead of just the one. That's obviously like, super wrong, for everything on the winit backend.

I guess it might also correctly transform the tablet position? If we return None, we hit that code path:

niri/src/input/mod.rs

Line 290 in f337a27

(geo, false, 1. / scale, Transform::Normal)

(the Transform::Normal here looks suspicious to me)

in general, it just makes sure the more relevant known-output code paths are hit. It feels more correct, if nothing else.

src/input/mod.rs

YaLTeR · 2025-01-05T07:23:04Z

src/input/mod.rs

                });
            }
        }
    }

    match (final_bind, pressed) {
        (Some(bind), true) => {
-            suppressed_keys.insert(key_code);
-            FilterResult::Intercept(Some(bind))
+            if is_inhibiting_shortcuts && bind.allow_inhibiting {


Bumping my comment to add a few test cases for this that got lost due to code changes.

src/protocols/virtual_pointer.rs

…ents

This was referenced Aug 26, 2024

idle inhibit on unfocused computers feschber/lan-mouse#181

Open

binds are not processed for virtual-keyboard events #631

Closed

YaLTeR reviewed Aug 27, 2024

View reviewed changes

sodiboo changed the title ~~implement wlr-keyboard-shortcuts-inhibit and wlr-virtual-pointer~~ implement keyboard-shortcuts-inhibit and wlr-virtual-pointer Aug 27, 2024

YaLTeR force-pushed the main branch from 62f2af1 to a2c8553 Compare September 4, 2024 09:39

sodiboo force-pushed the support-lan-mouse branch 3 times, most recently from bfd9d78 to 62982c4 Compare September 7, 2024 19:53

sodiboo marked this pull request as ready for review September 7, 2024 21:08

YaLTeR reviewed Oct 6, 2024

View reviewed changes

sodiboo force-pushed the support-lan-mouse branch from 62982c4 to 234b6b5 Compare October 6, 2024 13:25

sodiboo force-pushed the support-lan-mouse branch 2 times, most recently from 33cdeac to e5298b7 Compare October 20, 2024 21:16

sodiboo force-pushed the support-lan-mouse branch 8 times, most recently from d19f77f to 2febcdc Compare November 23, 2024 20:22

YaLTeR linked an issue Nov 23, 2024 that may be closed by this pull request

Implement keyboard-shortcuts-inhibit-unstable-v1 #198

Open

sodiboo force-pushed the support-lan-mouse branch from 4b29f6d to 9e8a5de Compare November 23, 2024 21:36

sodiboo force-pushed the support-lan-mouse branch from 9e8a5de to f337a27 Compare December 25, 2024 18:35

YaLTeR requested changes Jan 5, 2025

View reviewed changes

sodiboo force-pushed the support-lan-mouse branch from f337a27 to 4189442 Compare January 12, 2025 18:58

sodiboo added 7 commits January 12, 2025 20:02

stub keyboard-shortcuts-inhibit and virtual-pointer impls

0dade28

implement keyboard-shortcuts-inhibit

132dc72

implement virtual-pointer

38615a6

deal with supressed key release edge-case; add allow-inhibiting property

8e32053

add toggle-keyboard-shortcuts-inhibit bind

3e819c8

add InputBackend extensions; use Device::output() for absolute pos ev…

71f3b2c

…ents

implement WinitVirtualDevice::output() to the one single virtual output

5440002

sodiboo force-pushed the support-lan-mouse branch from 4189442 to 5440002 Compare January 12, 2025 19:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement `keyboard-shortcuts-inhibit` and `wlr-virtual-pointer` #630

implement `keyboard-shortcuts-inhibit` and `wlr-virtual-pointer` #630

sodiboo commented Aug 26, 2024

YaLTeR left a comment

YaLTeR Aug 27, 2024

sodiboo Sep 7, 2024

YaLTeR Oct 6, 2024

sodiboo commented Sep 7, 2024

YaLTeR left a comment

YaLTeR Oct 6, 2024

sodiboo Nov 23, 2024

YaLTeR Nov 28, 2024

sodiboo Nov 28, 2024

YaLTeR Jan 5, 2025

YaLTeR Oct 6, 2024

sodiboo Nov 23, 2024

YaLTeR Nov 28, 2024

YaLTeR Oct 6, 2024

sodiboo commented Oct 6, 2024

ConsularParadi commented Oct 19, 2024

YaLTeR commented Oct 19, 2024

sodiboo commented Oct 19, 2024

sodiboo commented Nov 8, 2024

sodiboo commented Nov 23, 2024

salman-farooq-sh commented Dec 8, 2024

salman-farooq-sh commented Dec 25, 2024

YaLTeR commented Jan 2, 2025

sodiboo commented Jan 3, 2025

YaLTeR left a comment

YaLTeR Jan 5, 2025

YaLTeR Jan 5, 2025

sodiboo Jan 12, 2025

YaLTeR Jan 5, 2025

	Mod+Shift+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }
	Mod+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }

implement keyboard-shortcuts-inhibit and wlr-virtual-pointer #630

Are you sure you want to change the base?

implement keyboard-shortcuts-inhibit and wlr-virtual-pointer #630

Conversation

sodiboo commented Aug 26, 2024

YaLTeR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sodiboo commented Sep 7, 2024

YaLTeR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sodiboo commented Oct 6, 2024

ConsularParadi commented Oct 19, 2024

YaLTeR commented Oct 19, 2024

sodiboo commented Oct 19, 2024

sodiboo commented Nov 8, 2024

sodiboo commented Nov 23, 2024

salman-farooq-sh commented Dec 8, 2024

salman-farooq-sh commented Dec 25, 2024

YaLTeR commented Jan 2, 2025

sodiboo commented Jan 3, 2025

YaLTeR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

implement `keyboard-shortcuts-inhibit` and `wlr-virtual-pointer` #630

implement `keyboard-shortcuts-inhibit` and `wlr-virtual-pointer` #630