Window actions?

[drewcrawford]

(apologies for the flood of discussions)

My first non-toy project is to make some window decorations accessible (e.g. minimize, maximize, close, etc.) After looking at how some GNOME apps do this, maybe the idiomatic way is to attach actions to the window. Here was my very first guess:

let mut window = accesskit::Node::new(Role::Window);
window.add_action(Action::Expand); //map to maxmimize?
window.add_action(Action::Collapse); //map to minimize?
window.add_action(Action::CustomAction); //map to close?

...and it turns out this has no effect. However I can debug the noops a lot better now:

fn n_actions(&self) -> i32 {
        if self.0.is_clickable() {
            1
        } else {
            0
        }
    }

    fn get_action_name(&self, index: i32) -> String {
        if index != 0 {
            return String::new();
        }
        String::from(if self.0.is_clickable() { "click" } else { "" })
    }

So I guess it has no effect because click is the only supported action. Some questions are:

1. Does it work this way because click is the only idiomatic action, and what I am doing amounts to 'holding it wrong'? Or is the unix adapter missing implementation?
2. Could accesskit adopt a forward-looking policy of "log when something is unimplemented / has no effect". This would help me nail down what I am doing that won't work for my users, and gives me a list of PRs I could send when I encounter them.

[DataTriny]

Window manipulation with AT-SPI occurs through a dedicated interface. We don't implement the methods to minimize, maximize, resize or move the window, and I don't know if we'll ever do. These actions on windows are, to the best of my knowledge, not used by any assistive technology. Also keep in mind that some (if not most) of them would only make sense on X11.

Depending on how you manage your windows, you could use AccessKit to describe the window decorations, but this would be GNOME specific.

I'm sorry if most of my answers aren't useful to you. I'm still figuring out some of this myself and we often need to ask for clarifications or changes in the AT-SPI protocol to make progress.

[drewcrawford]

Let me explain a bit more about my tarpit and how I got into it.

Mainly, I need to ship software that's reasonably portable across modern Linux distros. Accordingly I can't use a toolkit like GTK or QT which is not reliably present on a Linux desktop, I need to code against something more common than that. In this case the common denominator is Wayland.

Then due to some... interesting decisions by the GNOME team, I also need to create my own window decorations. So now I am responsible to implement some way to move windows, close them, resize them, and so on.

I'd like to imagine that part of working on any feature is considering how to make it accessible. So while I am already going to the trouble to implement closing the window with a mouse, it occurs to me to figure out some way to get this into the access tree as well. How hard could it be? I should mention winit has the same problems with their decorations, to the best of my knowledge their decorations are not accessible. So maybe, it is hard.

These actions on windows are, to the best of my knowledge, not used by any assistive technology.

This and the implementation make me suspect that actions generally (not just window) aren't used/useful on linux. This would be unfortunate as I lean on them a lot in apple-land. This seems like yet another way I wind up maintaining separate accessibility code for every platform.

In general, having API that can express this idea, paired with a backend that cannot, feels like a footgun to me. Ideally the API design would steer me towards ideas that can work in unfamiliar territory.

We don't implement the methods to minimize, maximize, resize or move the window, and I don't know if we'll ever do.

Another problem for me to consider is that from time to time I want to do things that aren't yet implemented. It seems kind of tricky to me to integrate accesskit with a custom zbus object here and there.

That makes me wonder if it's more expedient to just crank out my own solution with smaller API surface area and more aggressive translation of platform differences. It doesn't feel ideal but perhaps it is practical.

[mwcampbell]

AccessKit does implement the Click action on all platforms. So to make your window decorations accessible, you can add a node with the TitleBar role to the tree, then add Label and Button node as children of that. Then handle the Click action on the buttons. It's true that these buttons aren't focusable, but a screen reader user should be able to use screen reader commands to click the buttons if they really want to. I suspect that this accessibility would more likely be used by users of other assistive technologies besides screen readers.

I haven't yet tried making window decorations accessible like this myself. To make this work in winit would require a deeper integration with winit than what we currently have, and I think it would also require a way of embedding a subtree (the application-provided tree) inside an outer tree (the one with the window decorations). This is something I've known for a while that we'll need sooner or later, but haven't yet designed.

I hope this helps. I think AccessKit can still do what you need. Thanks for your questions and feedback.