Experiment with WPT tests using python handlers and testdriver.js in linux

[spectranaut]

Check out this little experiment: Igalia/wpt#1

We can run python with WPT server, so we could write tests that test the accessibility tree this way.

Here is the documentation for WPT python handlers: https://web-platform-tests.org/writing-tests/python-handlers/index.html
Here are the request/response objects: https://web-platform-tests.org/tools/wptserve/docs/

Some thoughts:

1. Each test would have two test files, which is a bit of a bummer.
2. Unlike the rest of the WPT tests, we would need to run different tests for different OSs, but we could figure out someway to handle that I'm sure.
3. Unlike the rest of WPT, we need to know the name of the browser we are testing, in order to find the accessibility tree. We could maybe do this from the UA string, and try a bunch of different names (like both "chrome" and "chromium"). These tests couldn't run in parallel and it feels a little fragile.

[spectranaut]

More notes about the branch (after Alice's commit to show the example of "serializing" the node, to keep all the test logic in the javascript):

Apparently, the logic you have to make sure we found the correct "web document" by checking the document attribute DocURL does not work in chrome -- chrome returns in the URL in the document attribute "URI" -- so i switch to looking for the web document that has the same name as the html title.

If using the name turns out to not be consistent across browsers/platforms, another possibility it to go the active tab route that chromium ax_dump_tree tools use:
https://source.chromium.org/search?q=AXTreeSelector::ActiveTab%20f:accessi&ss=chromium%2Fchromium%2Fsrc
https://source.chromium.org/chromium/chromium/src/+/main:ui/accessibility/platform/inspect/ax_inspect_utils_auralinux.cc;l=456;drc=f0f5f3ceebb00da9363ccc7a1e2c0f17b6b383ba?q=AXTreeSelector::ActiveTab%20f:accessi&ss=chromium%2Fchromium%2Fsrc

When running the tests from the CLI I ran into another problem:

spectranaut@pop-os:~/repos/wpt$ ./wpt run chrome /core-aam/acacia/test.html
Running 1 tests in web-platform-tests

[0/1] /core-aam/acacia/test.html
▶ Unexpected subtest result in /core-aam/acacia/test.html:
│ FAIL [expected PASS] An acacia test!
│ → assert_equals: expected "Found" but got "could not find HeadlessChrome"
│
└ at Test. (http://web-platform.test:8000/core-aam/acacia/test.html:22:7)

Ran 1 tests finished in 1.6 seconds.
• 0 ran as expected. 0 tests skipped.
• 1 tests had unexpected subtest results

Now that I've updated it to look for "chrome" if it finds the family "HeadlessChrome", it crashes on a "getRoleName()" when trying to find the tab within the tree.... I think because accessibility was not enabled? So I added an option to turn on accessibility for chrome:
./wpt run --force-renderer-accessibility chrome /core-aam/acacia/test.html

Note: you can also run for debugging:
./wpt run --no-headless --force-renderer-accessibility chrome /core-aam/acacia/test.html

Questions

1. can we extend test driver somehow, to get either the browser name of process id, then pass it to the python handler......? Presumably to run the webdriver tests they need to know what browser the request came from, so maybe we can also know that. See the tutorial: https://web-platform-tests.org/writing-tests/testdriver-extension-tutorial
2. I think I can into some test flakiness that I don't quite understand, but for a while I was getting getting Atspi errors when trying to getRoleName. This was happening when running wpt from the commandline.

(process:2452580): dbind-CRITICAL **: 11:27:52.941: atspi_accessible_get_role_name: assertion 'obj != NULL' failed
terminate called after throwing an instance of 'std::logic_error'
  ▶ Unexpected subtest result in /core-aam/acacia/test.html:
  │ FAIL [expected PASS] An acacia test!
  │   → promise_test: Unhandled rejection with value: object "TypeError: Failed to fetch"
  │ 
  │ Error
  │     at get_stack (http://web-platform.test:8000/resources/testharness.js:4544:21)
  │     at new AssertionError (http://web-platform.test:8000/resources/testharness.js:4537:22)
  │     at assert (http://web-platform.test:8000/resources/testharness.js:4521:19)
  │     at Test.<anonymous> (http://web-platform.test:8000/resources/testharness.js:764:29)
  │     at Test.step (http://web-platform.test:8000/resources/testharness.js:2622:25)
  └     at http://web-platform.test:8000/resources/testharness.js:2669:35

  CRITICAL Main thread got signal; waiting for TestRunnerManager threads to exit.
  [1/1] No tests running.

[spectranaut]

On to experimenting with testdriver.js, for example, used by these aria tests:

./wpt run --no-headless --log-mach-level debug --log-mach - chrome wai-aria/role/button-roles.html

Here are some webdriver design documents:
https://web-platform-tests.org/writing-tests/testdriver.html
https://web-platform-tests.org/writing-tests/testdriver-extension-tutorial.html
https://github.com/web-platform-tests/wpt/blob/master/tools/wptrunner/docs/design.rst

[spectranaut]

Notes about getting the PID:

• The class Firefox object gets and surfaces the PID: https://github.com/web-platform-tests/wpt/blob/500f11b59f5417c9b29080d27ef1fd47f4b1495b/tools/wptrunner/wptrunner/browsers/firefox.py#L502
  • In executormarionetter, however, we don't have this browser object, we have an "ExecutorBrowser" which does not have a PID attribute. I wonder if we can pass the PID to it.
• A few other browsers do, but not chrome, chromium or webkit.

Chrome family browsers:
Chrome (and chrome family browsers) use a chromedriver to start the browser, you can see the command they run with verbose output:
* chromium: DEBUG Starting WebDriver: /home/spectranaut/repos/wpt/_venv3/bin/chromium/chromedriver --port=51753 --url-base=/ --enable-chrome-logs --disable-build-check
* chrome: DEBUG Starting WebDriver: /home/spectranaut/repos/wpt/_venv3/bin/chrome/chromedriver --port=48437 --url-base=/ --enable-chrome-logs --disable-build-check

So we can't get the browser URLs from them. I think we are going to have to guess at the name if we want to find the tree to dump. Which might be fine because I'm not sure the PID would have worked on windows anyway but it's a bit more fragile.

[spectranaut]

Here I am trying to get the product name which seems like it accurately reflects ./wpt run <product_name>: Igalia/wpt@9c37e34

Also, it doesn't seem like these tests belong in the WebDriverProtocol -- but it seems like every product maps to exactly one executor in the directory wpt/tools/wptrunner/wptrunner/executors. And we don't want to replace the WebDriverProtocol, we need it, because eventually we will want to do webdriver things (to click buttons, listen to events).

[alice]

Thanks for the investigation and notes; frustrating that we won't be able to get the PID. I guess WebDriver communicates via the port, so they don't need to worry about this.

[alice]

TODO: Figure out how to make this cross-platform?

[alice]

Possibly relevant: https://docs.google.com/document/d/1OBoZTcC9vDoLTgv_5WUznRFrmwXP0Gprj7V9oOzH9cU/edit

This runs a specific back-end for WebTransport tests.