Does linux have a 'focus' concept? #503
-
|
In another discussion, it was mentioned that:
I have been trying to get a better picture of 'focus' generally. I am familiar with VoiceOver on apple platforms. Over there, there is a concept of a voiceover 'cursor' that can be moved 'forward' and 'back' (and 'up') to 'focus' on a particular widget, and the current focused widget can be queried via an API like I have not been able to find an analogous concept on Linux through orca or ATSPI. For example over here, the property
I'm developing the impression that Linux has no "accessibility focus" concept and relies on keyboard focus exclusively. Is that correct? If so, that presents real challenges taking code I wrote and just expecting it to work on Linux:
Does accesskit (or any other middleware) do things to 'smooth over' these platform differences? Where can I go to find out about how other people are porting accessible software? |
Beta Was this translation helpful? Give feedback.
Replies: 6 comments 11 replies
-
|
On the Linux desktop (AT-SPI) focus is a state. It would technically be possible for multiple widgets to be focused, although AccessKit will make sure this does not happen. Focus is not necessarily tied to keyboard events. AccessKit basically considers that any node which supports the action Bounding box is not linked to focus, although it is frequently used in conjunction with it: think of screen magnifiers which move the magnified area to track the focus, or various other assistive technologies that might display a rectangle around the focused widget. There are a few platform-specific behaviors around focus though. One that come to my mind right now is whether the focus follows the selection. This is something AccessKit doesn't address. My only knowledge on this is some personal experience and reading through Chromium's source code. |
Beta Was this translation helpful? Give feedback.
-
|
I think I must be missing something very obvious. Here is some toy code: fn request_initial_tree(&mut self) -> Option<TreeUpdate> {
let mut window = accesskit::Node::new(Role::Window);
window.set_bounds(Rect::new(0.0, 0.0, 800.0, 600.0));
let mut push_button = accesskit::Node::new(Role::Button);
push_button.set_label("My first button");
push_button.set_bounds(Rect::new(10.0, 10.0, 100.0, 50.0));
push_button.add_action(Action::Focus);
push_button.add_action(Action::Click);
let mut push_button_2 = accesskit::Node::new(Role::Button);
push_button_2.set_label("My second button");
push_button_2.set_bounds(Rect::new(10.0, 10.0, 100.0, 50.0));
push_button_2.add_action(Action::Focus);
push_button_2.add_action(Action::Click);
window.set_children(vec![NodeId(2), NodeId(3)]);
let tree = accesskit::Tree {
root: NodeId(1),
app_name: Some("app_window".to_string()),
toolkit_name: Some("app_window".to_string()),
toolkit_version: Some("0.1.0".to_string()),
};
let mut update = accesskit::TreeUpdate {
nodes: vec![(NodeId(1),window), (NodeId(2), push_button), (NodeId(3), push_button_2)],
tree: Some(tree),
focus: NodeId(1),
};
Some(update)
}Testing this code:
|
Beta Was this translation helpful? Give feedback.
-
|
The AT-SPI protocol defines a method We have not yet properly implemented some states related to the visibility of nodes. This is not an issue for now so I wouldn't worry too much about what you're seeing in KDE's Inspector. Feel free to open an issue if you encounter an issue though. Given your UI, you can explore the content of your window with Orca using "Flat review". I don't think it moves the focus though. I don't remember the key bindings but they involve Numpad keys on a desktop keyboard layout. The Tab key is not used by Orca to move the focus, this is the UI toolkit's responsibility. And, of course, it would be bad in practice to have two focusable buttons but still put the focus on the window. |
Beta Was this translation helpful? Give feedback.
-
orca has a command called 'Speak entire window using flat review'. (I have remapped this key in orca settings to verify orca can at least recognize the keys I am pressing, since I am running in VMs and maybe input routing is strange). As far as I can determine, this command has no effect, not for this example and not for any GNOME app I have tried. But perhaps I am 'holding it wrong' somehow.
So on macOS, it is instead the screen reader's responsibility to set the focus. With the screenreader enabled, pressing tab focuses many more elements than when it's not running, because tab is driven by the screenreader. The screenreader also draws a visual indicator (black box) around the focused element. And it offers a lot of commands to move focus such as control-alt-rightarrow – I'm unfamiliar with how they all differ exactly. So this brings me back to "maybe on Linux, a lot of this is my problem". If so, it complicates the idea of cross-compiling the same code quite a bit. |
Beta Was this translation helpful? Give feedback.
-
|
Even on macOS, Tab is implemented by the UI toolkit, not by VoiceOver. The reason why the behavior of Tab changes when VoiceOver is enabled is that VoiceOver enables a setting called Full Keyboard Access. So, to match native behavior, you'd need to check that setting using the There are also UI elements, like labels, that a screen reader user can move their screen reader cursor to, but which don't receive keyboard focus. Android has the concept of "accessibility focus" as distinct from keyboard focus, so Android applications can know and control the position of the screen reader cursor. And I believe iOS does too. I'm not sure if macOS does; my work on AccessKit so far assumed that it doesn't. Windows definitely doesn't, and I'm not sure about Linux either. But in all cases, you definitely shouldn't put labels (a.k.a. static text) in the tab order so they can receive keyboard focus. Screen reader users will access these elements exclusively with screen reader keyboard commands. So, if you want to test with Orca, you'll need to figure out why those commands aren't working for you. Are you using Wayland or X11? |
Beta Was this translation helpful? Give feedback.
-
This is because it apparently uses the more generic |
Beta Was this translation helpful? Give feedback.
-
|
Well on macOS the borders between the toolkit and the screen reader are perhaps a bit permeable ;-) But it may bring clarity to express the situation in code: let strong = NonKeyboardFocusableElement()
let strong2 = NonKeyboardFocusableElement()
final class NonKeyboardFocusableElement: NSAccessibilityElement {
override func accessibilityPerformPress() -> Bool {
fatalError()
}
}
class ViewController: NSViewController {
override func viewDidLoad() {
super.viewDidLoad()
strong.setAccessibilityRole(.button)
strong.setAccessibilityParent(self.view)
strong.setAccessibilityLabel("Hi from a non-keyboard focusable element!")
strong2.setAccessibilityFrameInParentSpace(.init(x: 0, y: 0, width: 50, height: 50))
strong2.setAccessibilityRole(.button)
strong2.setAccessibilityParent(self.view)
strong2.setAccessibilityLabel("Hi from a different non-keyboard focusable element!")
strong2.setAccessibilityFrameInParentSpace(.init(x: 0, y: 50, width: 50, height: 50))
view.setAccessibilityChildren([strong, strong2])
}
}Side note, I don't know any way to press these buttons from FKA alone, just voiceover. But regardless of which OS component is which, from my perspective as an application developer I assume my main problem is to produce this kind of tree for my software, and express the tree to the platform, and tools like accesskit basically wrap the kinds of APIs I use here. Then somebody else from the platform will come along with an implementation for navigating the tree, remembering which element is focused, drawing a visual indicator over the focused control, providing some way for the user to perform actions reported available by the tree, and so on. And there's probably more than one implementation, since screen readers and inspectors have different workflows, users have different needs, etc. However given that accerciser didn't think to implement focus at all, that KDE's version doesn't play well with accesskit, and I still haven't figured out what key combination will focus the next element in orca, my assumptions seem more dubious. Perhaps the reason I can't find these features in the tooling is because each piece of software is supposed to do that instead.
On the other hand this suggests to me there is such a command. I'm not sure why I can't seem to either find it or get it to work.
Wayland, Fedora 41, VMWare Fusion. Also tried CentOS 10. My experience is that orca works well reading off controls with keyboard focus, but things like window decorations for example are not usually in the tab order. |
Beta Was this translation helpful? Give feedback.
-
I figured out where it is. Will follow up later with an explanation... |
Beta Was this translation helpful? Give feedback.
-
|
I have learned a new reason that orca can't read my accesskit examples: Evidently, "There's currently no accepted way to pass keyboard input to screen readers on Wayland.". Without seeing the keyboard input orca can't respond to keyboard commands so it just seems not to work. (I believe the toolkits have some kind of workaround for this but I have not identified what they are doing specifically.) One reason this took so long for me to work out is it was frankly unbelievable to me that major distros like Fedora and RHEL would ship Wayland-only without an accessibility solution in place and say it's "in good shape", so I just assumed the problem must be in my code. Evidently it is true. It does seem to be worked on and I am hopeful things will improve with more attention on Wayland. A related GNOME bug is here, KDE bug is here COSMIC bug is here flatpak bug is here, orca bug is here, among others. |
Beta Was this translation helpful? Give feedback.
-
|
I wouldn't be opposed to exposing a focus action through the AT-SPI action interface. Maybe we've over-optimized for what Orca needs. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, you're correct about the problem of Orca not being able to receive keyboard input on Wayland.
Yes, there's an old way for toolkits to pass keyboard input to ATs. GTK supported this through version 3.x, and Qt still does. We chose to follow GTK 4's lead and not support this old solution in AccessKit, because it would have required all toolkit developers to make major changes to their event loop that weren't necessary on any other platform, and frankly, when we made that decision at the end of 2022, we thought the problem would be fixed quickly, because of GTK 4. I suppose we could at least provide the option to do what GTK 3 and current Qt are doing. Are you using winit? |
Beta Was this translation helpful? Give feedback.
-
Now that you mention it, this illuminates for me why I've been very slowly going insane. I had formed the impression the GTK4 apps were generally pretty usable through using them with orca myself. But taking a second look, that is all using toolkit commands, not orca commands which indeed do not work there at all. So the reason I got turned around on "is screenreader focus even a real thing on Linux or is everything just the toolkit?" – is that in GTK4 it's not a real thing and it is all the toolkit. I even spent an hour or two yesterday digging around in the GTK4 source to see if I could find and borrow their workaround for myself. Because in my mind their apps are usable in orca so they must have a workaround somewhere in the labyrinth of projects. Also in my mind, it is impossible to ship the leading linux toolkit for 4 years with no workaround, so it's gotta be there somewhere I'm just not looking hard enough. GTK3 never occurred to me. It really is just my failure of imagination that things in the ecosystem could really be this bad. I can't imagine how people who depend on this stuff working get by.
If there were an API I could send keystrokes to myself, and it broadcasted them over atspi or something, I would use it.
No, I'm basically rewriting winit myself. I already needed major changes from winit's event loop for other reasons on other platforms so that is the easy part. The hard part is, if I am writing new event loops anyway what are the other cars to attach to that train. Keyboard handling for orca does not sound very difficult to bolt on, then again I have failed to imagine how hard things can be several times so far. |
Beta Was this translation helpful? Give feedback.
-
Blind Linux users, at least those in the know, still use X11. |
Beta Was this translation helpful? Give feedback.
-
|
I suppose they just don't use Fedora anymore, and won't use RHEL 10 either. Truly baffling. |
Beta Was this translation helpful? Give feedback.
-
Having Orca tell you what your mouse is pointing at doesn't work either under Wayland, just to save you some time. I tried fully using Linux for a year or so, went back to Windows and I don't plan to switch again until major changes happen. |
Beta Was this translation helpful? Give feedback.
-
|
I took a look at trying to implement this myself – I think the key events are sent via via DeviceEventController's notify_listener_sync. After trying that, orca seems to partly understand my keystrokes although not entirely well. I think there are issues with how I represent the events. Is there a way to snoop traffic from an app that works so i can see the keycodes it is sending etc? |
Beta Was this translation helpful? Give feedback.
-
|
After tracing through many layers of GNOME code I think I can cobble together an implementation for my event loop. To save anyone else some time,
#!/usr/bin/env python3
import gi
gi.require_version('Atspi', '2.0')
gi.require_version('Gdk', '3.0')
from gi.repository import Atspi
from gi.repository import GLib
from gi.repository import Gdk
class KeyWatcher:
def __init__(self):
self.device = None
self.mainloop = GLib.MainLoop()
def start(self):
"""Initialize and start the key watcher."""
print("Starting key watcher... Press keys to see events (Ctrl+C to exit)")
self.device = Atspi.Device.new()
# Add our callback function as the key event listener
self.device.add_key_watcher(self.on_key_event)
try:
# Start the main event loop
self.mainloop.run()
except KeyboardInterrupt:
print("\nStopping key watcher...")
finally:
self.stop()
def stop(self):
"""Clean up and stop the key watcher."""
self.device = None
self.mainloop.quit()
def on_key_event(self, device, pressed, keycode, keysym, modifiers, text):
"""Callback function that handles key events."""
# Log all raw parameters
print("=== Raw Event Data ===")
print(f"device: {device!r}")
print(f"pressed: {pressed!r}")
print(f"keycode: {keycode!r}")
print(f"keysym: {keysym!r}")
print(f"modifiers: {modifiers!r} (0b{bin(modifiers)[2:]})")
print(f"text: {text!r}")
# Log modifier flags
print("\n=== Modifier Flags ===")
for mod_type in dir(Atspi.ModifierType):
if not mod_type.startswith('_'):
try:
mod_value = getattr(Atspi.ModifierType, mod_type)
if isinstance(mod_value, int):
is_set = bool(modifiers & (1 << mod_value))
print(f"{mod_type}: {is_set} (bit {mod_value})")
except Exception as e:
print(f"{mod_type}: <error: {e}>")
# Log key name from Gdk
print("\n=== Gdk Key Info ===")
key_name = Gdk.keyval_name(keysym)
print(f"Gdk.keyval_name(keysym): {key_name!r}")
# Get all possible key names for the keycode
keymap = Gdk.Keymap.get_default()
result = keymap.get_entries_for_keycode(keycode)
if result[-1]: # The keyvals are in the last element of the tuple
print(f"All possible keyvals for keycode {keycode}:")
for keyval in result[-1]:
if keyval: # Filter out None values
name = Gdk.keyval_name(keyval)
print(f" {keyval}: {name!r}")
print("-" * 50)
return False # Allow event propagation
def main():
watcher = KeyWatcher()
watcher.start()
if __name__ == "__main__":
main() |
Beta Was this translation helpful? Give feedback.
Blind Linux users, at least those in the know, still use X11.