[Do not merge] Test262 run #56705
Draft
[Do not merge] Test262 run #56705
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jcscottiii
force-pushed
the
jcscottiii/test262-upstream-everything
branch
from
3507fe0 to
c29bc27
Compare
This commit introduces the foundational infrastructure for the WPT manifest
to recognize and process Test262 ECMA-262 conformance tests. This is
the first step towards the goal of running the entire Test262 suite
within the WPT infrastructure, as outlined in the approved RFC.
The core of this change is a new `TestRecord` parser in `test262.py`
which is responsible for extracting the YAML frontmatter from Test262
`.js` files. This metadata is then used by the manifest generation
process to correctly identify and classify the tests.
Key changes included in this commit:
- A new `tools/manifest/test262.py` module containing the YAML
frontmatter parser.
- A corresponding `Test262Test` manifest item type defined in
`tools/manifest/item.py` to represent these tests.
- Updates to `tools/manifest/sourcefile.py` to identify Test262 files
by their directory path (`/test262/`) and invoke the new parser.
- The addition of `pyyaml` as a project dependency to handle the YAML
parsing to tools/wpt/requirements.txt
- **Note to reviewers:** PyYaml is lazily loaded. This is to prevent
the modification of too many commands.json files. This may not be
preferred since WPT does not do that often but open to other
options. Please check failing checks for commits on the original PR
for how I kept trying to track down the next commands.json file to
fix.
- New unit tests for the parser and the manifest generation logic for
Test262 files.
This work is part of the effort to add Test262 support as laid out in
the RFC: web-platform-tests/rfcs#229
This commit is the first in a series of smaller PRs split from the larger,
original implementation in web-platform-tests#55997.
Subsequent PRs will add the server-side logic to serve the tests and the
wptrunner integration to execute them.
This commit introduces the necessary server-side handlers within `wptserve`
to dynamically generate HTML wrappers for Test262 JavaScript tests.
This is needed to enable Test262 execution within WPT.
Key changes and their purpose:
- Introduction of several new `HtmlWrapperHandler` and `WrapperHandler`
subclasses (e.g., `Test262WindowHandler`, `Test262WindowTestHandler`,
`Test262StrictHandler`). These handlers are responsible for:
- Identifying Test262 test requests based on URL patterns.
- Dynamically constructing an HTML page that loads the Test262
`.js` test file within an isolated `iframe`.
- Injecting the required Test262 harness files (`assert.js`, `sta.js`)
and the WPT-specific `testharness-client.js` and `harness-adapter.js`
into the generated HTML.
- Processing Test262-specific metadata (like `includes` and `negative`
expectations) extracted by the manifest tooling from PR 1.
- Updates to `RoutesBuilder` in `serve.py` to map incoming requests
for Test262 test URLs to the appropriate new handler.
- Unit tests in `test_serve.py` to validate the correct
behavior of these new handlers, including URL rewriting, metadata
processing, and the structure of the generated HTML wrappers.
This work directly supports the integration of Test262 into WPT as detailed
in the RFC: web-platform-tests/rfcs#229
This commit is the second in a series of smaller PRs split from the larger,
original implementation in web-platform-tests#55997.
…e tests This commit completes the core integration of Test262 into the WPT runner (`wptrunner`), enabling the execution of Test262 JavaScript conformance tests within the browser environment. This work builds upon the manifest tooling (PR 1 web-platform-tests#56840) and server-side handling (PR 2 web-platform-tests#56841) to provide a full end-to-end testing solution for Test262. Key components introduced and integrated: - **Client-Side Harness:** New JavaScript files (`harness-adapter.js`, `testharness-client.js`, `testharness.js`) under `resources/test262/` provide the necessary environment for Test262 tests to execute in an `iframe` and report results back to the main WPT testharness. - **Test262 Assertion Libraries (Vendored):** For the purpose of enabling these initial smoke tests, the standard Test262 assertion libraries (`assert.js` and `sta.js`) are directly placed into `third_party/test262/harness/`. This provides the immediate dependencies required by the smoke tests. An autoamted vendoring solution, including version tracking via `vendored.toml`, will be implemented in a subsequent PR focused on automated updates, as described in the RFC. - **Wptrunner Integration:** Modifications to `tools/wptrunner/wpttest.py` and `tools/wptrunner/browsers/*.py` enable `wptrunner` to recognize the `test262` test type and execute it using existing testharness executors. - **Smoke Tests and Metadata:** A suite of basic Test262 smoke tests (`infrastructure/test262/`) and their corresponding expected results (`infrastructure/metadata/infrastructure/test262/`) are added to verify the correct functioning of the entire integration. - **Linting Exemption:** `lint.ignore` is updated to exclude the vendored Test262 harness files from linting, respecting their upstream style. This integration allows browser vendors to run Test262 directly with `wptrunner`, streamlining conformance testing efforts as specified in the RFC: web-platform-tests/rfcs#229 This commit is the third in a series of smaller PRs split from the larger, original implementation in web-platform-tests#55997.
This commit integrates Test262 test execution into the WPT Taskcluster
Continuous Integration system.
Key changes and their purpose:
- **Task Definition (`tools/ci/tc/tasks/test.yml`):**
- A new `wpt-test262` component is defined, specifying the common
properties and chunking strategy for Test262 tasks within Taskcluster.
- The main Taskcluster task generation matrix is updated to include
`test262` as a recognized test suite. This ensures that Taskcluster
will automatically create jobs for Test262 across configured
browsers and channels (e.g., Firefox Nightly, Chrome Canary).
- **CI Validation (`tools/ci/tc/tests/test_valid.py`):**
- The integration tests for Taskcluster's decision logic (`test_valid.py`)
are updated to expect the new `wpt-*-test262-*` tasks in the generated
task graph. This provides automated validation that the CI configuration
is correctly interpreted and will schedule the Test262 tasks as intended.
With these changes, Taskcluster will automatically detect and execute Test262
tests that are present in the `third_party/test262` directory (as populated by
the automated vendoring workflow).
This work is a fundamental part of the broader Test262 integration effort,
as detailed in the RFC: web-platform-tests/rfcs#229
This commit is the final in a series of smaller PRs split from the larger,
original implementation in web-platform-tests#55997.
jcscottiii
force-pushed
the
jcscottiii/test262-upstream-everything
branch
from
ebf5eb2 to
b5620f9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#55997 + the files to run