Integrate Test262 into WPT #229

jcscottiii · 2025-11-04T01:20:21Z

jcscottiii · 2025-11-04T16:29:32Z

rfcs/test262_integration.md

+
+## Alternatives Considered
+
+### Using `--test-types=test262`


We should move to this instead.

If we do use a separate test type internally (not testharness.js) then I agree, this would be the natural way.

I don't think the internal implementation details are the important thing. From the point of view of a consumer test262 tests are just another kind of test we can run, just like crashtests are basically a special kind of reftest, but we expose them as a separate test type because that makes more sense to users.

jcscottiii · 2025-11-04T16:38:08Z

Should protect the WPT source from landing manual changes to that repo since they won't be synced back upstream.

gsnedders · 2025-11-04T16:45:48Z

Can we open web-platform-tests/wpt@master...o-:wpt:test262-upstream as a draft PR?

jgraham · 2025-11-07T09:23:39Z

rfcs/test262_integration.md

+
+## Summary
+
+This RFC proposes integrating the Test262 (ECMAScript) test suite into WPT. Due to its large size (>50,000 files), tests will be fetched on-demand rather than vendored into the repository. Execution will be opt-in via a new `--test262` flag. This provides a unified way to run this conformance suite while protecting the WPT repository from bloat and performance degradation.


I strongly prefer just vendoring the test262 files directly.

Although that number is large it's about a 50% increase on the current number of files in the repo, so it's not unmanagable.

Doing the vendoring directly means that there's no novel requirements for wpt consumers who want to run these tests (e.g. in vendor repos), apart from selecting the new test type.

I'd also propose a path like third_party/test262 so that it's at least clear that this is not code that lives in wpt. I do think we're likely to run into cases where people are surprised when they make changes to this directory and they aren't upstreamed; at least the third_party convention provides a clue that this might not work.

It looks like a shallow clone of Test262 is only 16M in the .git/objects/ directory, so I think that's about how much WPT would grow, which seems OK.

rfcs/test262_integration.md

jgraham · 2025-11-07T09:26:43Z

rfcs/test262_integration.md

+
+### 1. Source Management
+
+The WPT repository will contain a reference to the official `tc39/test262` repository (e.g., via a `.gitmodules` entry). This reference specifies the expected local path for the Test262 source and a pinned commit SHA.


I think we should try and be specific and not use "e.g." here.

I don't think we should use git submodules, or .gitmodules, but just use our own format e.g. a file like third_party/vendor.toml with a format like:

[test262] source = <git url> commit = <sha1>

I went ahead and changed the language to specify that we would be copying the files into the directory instead. Similar to the wasm/core directory.

jgraham · 2025-11-07T09:29:45Z

rfcs/test262_integration.md

+WPT's manifest generation and serving logic will be extended to support the on-demand nature of Test262 tests.
+
+*   **Manifest Generation:** The `wpt manifest` command will recognize the Test262 source directory and traverse its `test` directory to discover tests.
+*   **Metadata Parsing:** A new, Test262-specific parser will read the YAML frontmatter from each `.js` file to extract metadata (e.g., ES features, negative test expectations, execution flags).


To be clear: where will this end up? In MANIFEST.json? Or is the proposal to have a new special manifest format just for test262? Ideally I'd like to avoid that, but the RFC isn't providing enough information to understand the tradeoffs.

Same MANIFEST.json.

I did try it out with this PR: web-platform-tests/wpt#55997

The uncompressed manifest.json file is 36M in size now.

jgraham · 2025-11-07T09:32:35Z

rfcs/test262_integration.md

+
+### 3. Acquiring Test262 Source Code
+
+The Test262 source code is acquired on-demand. `wpt` can handle this automatically for users in a Git environment, while providing a manual path for CI systems.


As already mentioned I don't think this is the correct approach.

I think we should have a CI job that fetches the latest commit of test262, copies that over to the destination path and then opens a PR with the update. The CI should run some kind of smoketests to ensure that the update hasn't broken test262 support. Possibly a bot should be trusted to auto-merge these PRs (although that means we have to trust the test262 repository).

I changed it to this suggested approach.

gsnedders · 2025-11-07T21:52:33Z

See also: https://github.com/web-platform-tests/wpt/issues/8308

jcscottiii · 2025-11-13T01:32:06Z

@jgraham @gsnedders @foolip I updated the RFC. Also. I have updated the WIP PR too web-platform-tests/wpt#55997 so you can see what happens

foolip

Looks good, just a few comments!

foolip · 2025-11-24T16:04:40Z

rfcs/test262_integration.md

+## Implementation Considerations & Risks
+
+*   **Manifest Generation Performance:** Including the 50,000+ Test262 tests in the manifest by default will increase the time required to run `wpt manifest`. This performance cost is accepted to ensure a simpler user experience, where a single, default manifest contains all tests.
+*   **Repository Size:** Vendoring Test262 will significantly increase WPT repository size (~50,000 files), a manageable increase.


Maybe note that vendors that don't want it can exclude this directory using their existing mechanisms to exclude parts of WPT?

This would add the requirement that WPT has to work even with the directory deleted, should we test that in CI somehow?

Is there anyone who is known to not want these tests and can't simply ignore them? I feel like we're worrying a lot about people potentially being annoyed by the size and/or runtime increases, but without any specific examples of people who are actually worried by those things.

Not sure. But I just updated the RFC to have it run by default.

foolip · 2025-11-24T16:13:20Z

rfcs/test262_integration.md

+
+The current dynamic approach requires two disk I/O operations per test: one for the `wptserve` handler to parse YAML frontmatter, and a second for the browser to fetch the `.js` content. Storing metadata in `MANIFEST.json` would eliminate the first disk I/O and YAML parsing, effectively halving disk I/O operations per test during runtime.
+
+However, this optimization is deferred for initial implementation simplicity. Metadata will be parsed dynamically by `wptserve` handlers at runtime.


FWIW, I think it's a good thing to defer work until the tests are running when possible. Less time spent building the manifest makes ./wpt run feel faster as it starts running tests faster. (But total execution time should be the goal I think, and I'm sure it's sometimes faster to do the work once up front.)

rfcs/test262_integration.md

jgraham

Basically this seems fine to me, but ideally we'd change a couple of things:

Move the tests into a third_party directory with a manifest to indicate the state of the vendoring.
Make test262 tests behave like other tests as much as possible rather than making them a special case. In particular make them run by default as with every other test type.

rfcs/test262_integration.md

jgraham · 2025-11-25T10:32:04Z

rfcs/test262_integration.md

+
+*   **Manifest Generation:** `wpt manifest` will, by default, recognize `tc39/test262`, traverse its `test` directory, and add discovered tests to `MANIFEST.json`.
+*   **Metadata Parsing:** A new Test262-specific parser will read YAML frontmatter from `.js` files to extract metadata (e.g., ES features, negative expectations, execution flags). Unlike `wasm/core` tests, which are pre-processed, Test262 metadata will be used by `wptserve` handlers at runtime.
+*   **Harness and Server:** Specialized Test262 harness files and `wptserve` handlers will use parsed metadata to construct the execution environment. Test modes will use distinct URL conventions based on file extensions (e.g., `path/to/test.js` served as `path/to/test.test262.html` or `path/to/test.test262-module.html`).


I don't understand why we need test262 as flags in the test URL. I think we can wire up the handlers so they only apply in the relevant test262 directories, and use a simpler mapping from source file to test URL.

That's a very good point about simplifying the URL mapping.

My main reason for using the explicit .test262.html extension in the current proof-of-concept was to handle the smoke tests. Since they live in infrastructure/test262/, a global rule based on the file extension was the most straightforward way to ensure they were picked up by the correct wptserve handler.

If we switched to a handler that only activates for paths inside third_party/test262/, it wouldn't find the smoke tests. We could explicitly add a second check for infrastructure/test262/, but hardcoding special-case paths like that into the server logic feels a bit brittle.

Another alternative could be to make the handler apply to any path that contains a test262 subdirectory. That would cover both cases neatly.

Let me know your thoughts.

We have quite a lot of precedent for the directory name affecting test generation, so I'd go with that.

rfcs/test262_integration.md

jgraham · 2025-11-25T10:39:19Z

rfcs/test262_integration.md

+## Implementation Considerations & Risks
+
+*   **Manifest Generation Performance:** Including the 50,000+ Test262 tests in the manifest by default will increase the time required to run `wpt manifest`. This performance cost is accepted to ensure a simpler user experience, where a single, default manifest contains all tests.
+*   **Repository Size:** Vendoring Test262 will significantly increase WPT repository size (~50,000 files), a manageable increase.


Is there anyone who is known to not want these tests and can't simply ignore them? I feel like we're worrying a lot about people potentially being annoyed by the size and/or runtime increases, but without any specific examples of people who are actually worried by those things.

jgraham · 2025-11-26T09:14:42Z

rfcs/test262_integration.md

+
+*   **Manifest Generation:** `wpt manifest` will, by default, recognize `tc39/test262`, traverse its `test` directory, and add discovered tests to `MANIFEST.json`.
+*   **Metadata Parsing:** A new Test262-specific parser will read YAML frontmatter from `.js` files to extract metadata (e.g., ES features, negative expectations, execution flags). Unlike `wasm/core` tests, which are pre-processed, Test262 metadata will be used by `wptserve` handlers at runtime.
+*   **Harness and Server:** Specialized Test262 harness files and `wptserve` handlers will use parsed metadata to construct the execution environment. Test modes will use distinct URL conventions based on file extensions (e.g., `path/to/test.js` served as `path/to/test.test262.html` or `path/to/test.test262-module.html`).


We have quite a lot of precedent for the directory name affecting test generation, so I'd go with that.

rfcs/test262_integration.md

Co-authored-by: jgraham <[email protected]>

jcscottiii · 2025-12-04T17:35:16Z

@gsnedders Are you able to take a look at this please?

jcscottiii · 2025-12-15T21:58:10Z

Any final thoughts before I land that RFC? (Looking at the timeout rule on the RFC README)

I know Europe is already offline. I will wait until tomorrow before merging to give time.

This commit introduces the foundational infrastructure for the WPT manifest to recognize and process Test262 ECMA-262 conformance tests. This is the first step towards the goal of running the entire Test262 suite within the WPT infrastructure, as outlined in the approved RFC. The core of this change is a new `TestRecord` parser in `test262.py` which is responsible for extracting the YAML frontmatter from Test262 `.js` files. This metadata is then used by the manifest generation process to correctly identify and classify the tests. Key changes included in this commit: - A new `tools/manifest/test262.py` module containing the YAML frontmatter parser. - A corresponding `Test262Test` manifest item type defined in `tools/manifest/item.py` to represent these tests. - Updates to `tools/manifest/sourcefile.py` to identify Test262 files by their directory path (`/test262/`) and invoke the new parser. - The addition of `pyyaml` as a project dependency to handle the YAML parsing to tools/wpt/requirements.txt - **Note to reviewers:** PyYaml is lazily loaded. This is to prevent the modification of too many commands.json files. This may not be preferred since WPT does not do that often but open to other options. Please check failing checks for commits on the original PR for how I kept trying to track down the next commands.json file to fix. - New unit tests for the parser and the manifest generation logic for Test262 files. This work is part of the effort to add Test262 support as laid out in the RFC: web-platform-tests/rfcs#229 This commit is the first in a series of smaller PRs split from the larger, original implementation in #55997. Subsequent PRs will add the server-side logic to serve the tests and the wptrunner integration to execute them.

This commit introduces the necessary server-side handlers within `wptserve` to dynamically generate HTML wrappers for Test262 JavaScript tests. This is needed to enable Test262 execution within WPT. Key changes and their purpose: - Introduction of several new `HtmlWrapperHandler` and `WrapperHandler` subclasses (e.g., `Test262WindowHandler`, `Test262WindowTestHandler`, `Test262StrictHandler`). These handlers are responsible for: - Identifying Test262 test requests based on URL patterns. - Dynamically constructing an HTML page that loads the Test262 `.js` test file within an isolated `iframe`. - Injecting the required Test262 harness files (`assert.js`, `sta.js`) and the WPT-specific `testharness-client.js` and `harness-adapter.js` into the generated HTML. - Processing Test262-specific metadata (like `includes` and `negative` expectations) extracted by the manifest tooling from PR 1. - Updates to `RoutesBuilder` in `serve.py` to map incoming requests for Test262 test URLs to the appropriate new handler. - Unit tests in `test_serve.py` to validate the correct behavior of these new handlers, including URL rewriting, metadata processing, and the structure of the generated HTML wrappers. This work directly supports the integration of Test262 into WPT as detailed in the RFC: web-platform-tests/rfcs#229 This commit is the second in a series of smaller PRs split from the larger, original implementation in #55997.

…e tests This commit completes the core integration of Test262 into the WPT runner (`wptrunner`), enabling the execution of Test262 JavaScript conformance tests within the browser environment. This work builds upon the manifest tooling (PR 1 #56840) and server-side handling (PR 2 #56841) to provide a full end-to-end testing solution for Test262. Key components introduced and integrated: - **Client-Side Harness:** New JavaScript files (`harness-adapter.js`, `testharness-client.js`, `testharness.js`) under `resources/test262/` provide the necessary environment for Test262 tests to execute in an `iframe` and report results back to the main WPT testharness. - **Test262 Assertion Libraries (Vendored):** For the purpose of enabling these initial smoke tests, the standard Test262 assertion libraries (`assert.js` and `sta.js`) are directly placed into `third_party/test262/harness/`. This provides the immediate dependencies required by the smoke tests. An autoamted vendoring solution, including version tracking via `vendored.toml`, will be implemented in a subsequent PR focused on automated updates, as described in the RFC. - **Wptrunner Integration:** Modifications to `tools/wptrunner/wpttest.py` and `tools/wptrunner/browsers/*.py` enable `wptrunner` to recognize the `test262` test type and execute it using existing testharness executors. - **Smoke Tests and Metadata:** A suite of basic Test262 smoke tests (`infrastructure/test262/`) and their corresponding expected results (`infrastructure/metadata/infrastructure/test262/`) are added to verify the correct functioning of the entire integration. - **Linting Exemption:** `lint.ignore` is updated to exclude the vendored Test262 harness files from linting, respecting their upstream style. This integration allows browser vendors to run Test262 directly with `wptrunner`, streamlining conformance testing efforts as specified in the RFC: web-platform-tests/rfcs#229 This commit is the third in a series of smaller PRs split from the larger, original implementation in #55997.

This commit introduces a GitHub Actions workflow to automate the vendoring of the Test262 test suite into `third_party/test262`. This automation is the final piece of the integration, ensuring that the tests can be kept up-to-date with the upstream `tc39/test262` repository. The workflow is configured to run weekly or can be triggered manually. It performs the following steps: 1. Checks out both the WPT and `tc39/test262` repositories. 2. Selectively copies a subset of the Test262 tests into WPT. 3. Copies the essential Test262 harness files. 4. Creates a `vendored.toml` file at the root of the vendored directory, recording the specific commit SHA from `tc39/test262` that the files were sourced from. This ensures traceability, as described in the RFC. 5. Commits the updated files and opens a pull request with the changes. Initially, only a subset of Test262 tests related to Interop 2026 features (`Temporal` and `top-level-await`) are being imported. This is a temporary measure to manage the volume of new test results and to avoid overloading wpt.fyi, which currently has limitations in processing the full Test262 result set (see wpt-fyi/issues/4681). The workflow can be expanded to include more tests in the future. This work directly supports the integration of Test262 into WPT as detailed in the RFC: web-platform-tests/rfcs#229 This commit is part of a series of smaller PRs split from the larger, original implementation in #55997.

This commit integrates Test262 test execution into the WPT Taskcluster Continuous Integration system. Key changes and their purpose: - **Task Definition (`tools/ci/tc/tasks/test.yml`):** - A new `wpt-test262` component is defined, specifying the common properties and chunking strategy for Test262 tasks within Taskcluster. - The main Taskcluster task generation matrix is updated to include `test262` as a recognized test suite. This ensures that Taskcluster will automatically create jobs for Test262 across configured browsers and channels (e.g., Firefox Nightly, Chrome Canary). - **CI Validation (`tools/ci/tc/tests/test_valid.py`):** - The integration tests for Taskcluster's decision logic (`test_valid.py`) are updated to expect the new `wpt-*-test262-*` tasks in the generated task graph. This provides automated validation that the CI configuration is correctly interpreted and will schedule the Test262 tasks as intended. With these changes, Taskcluster will automatically detect and execute Test262 tests that are present in the `third_party/test262` directory (as populated by the automated vendoring workflow). This work is a fundamental part of the broader Test262 integration effort, as detailed in the RFC: web-platform-tests/rfcs#229 This commit is the final in a series of smaller PRs split from the larger, original implementation in #55997.