Skip to content

Commit

Permalink
feat(roll): roll to ToT Playwright (22-11-24) (#1603)
Browse files Browse the repository at this point in the history
  • Loading branch information
playwrightmachine authored Nov 22, 2024
1 parent 4dde2b8 commit 373b52a
Show file tree
Hide file tree
Showing 16 changed files with 543 additions and 88 deletions.
4 changes: 2 additions & 2 deletions dotnet/docs/api/class-tracing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,10 @@ Creates a new group within the trace, assigning any subsequent API calls to this
```csharp
// All actions between GroupAsync and GroupEndAsync
// will be shown in the trace viewer as a group.
await Page.Context().Tracing.GroupAsync("Open Playwright.dev > API");
await Page.Context.Tracing.GroupAsync("Open Playwright.dev > API");
await Page.GotoAsync("https://playwright.dev/");
await Page.GetByRole(AriaRole.Link, new() { Name = "API" }).ClickAsync();
await Page.Context().Tracing.GroupEndAsync();
await Page.Context.Tracing.GroupEndAsync();
```

**Arguments**
Expand Down
72 changes: 59 additions & 13 deletions dotnet/docs/aria-snapshots.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
id: aria-snapshots
title: "Aria snapshots"
title: "Snapshot testing"
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
Expand All @@ -10,23 +10,69 @@ import LiteYouTube from '@site/src/components/LiteYouTube';

## Overview

In Playwright, aria snapshots provide a YAML representation of the accessibility tree of a page. These snapshots can be stored and compared later to verify if the page structure remains consistent or meets defined expectations.
With the Playwright Snapshot testing you can assert the accessibility tree of a page against a predefined snapshot template.

```csharp
await page.GotoAsync("https://playwright.dev/");
await Expect(page.Locator("banner")).ToMatchAriaSnapshotAsync(@"
- banner:
- heading ""Playwright enables reliable end-to-end testing for modern web apps."" [level=1]
- link ""Get started""
- link ""Star microsoft/playwright on GitHub""
- link /[\\d]+k\\+ stargazers on GitHub/
");
```

<LiteYouTube id="P4R6hnsE0UY" title="Getting started with ARIA Snapshots" />

The YAML format describes the hierarchical structure of accessible elements on the page, detailing **roles**, **attributes**, **values**, and **text content**. The structure follows a tree-like syntax, where each node represents an accessible element, and indentation indicates nested elements.
## Assertion testing vs Snapshot testing

Following is a simple example of an aria snapshot for the playwright.dev homepage:
Snapshot testing and assertion testing serve different purposes in test automation:

```yaml
- banner:
- heading /Playwright enables reliable/ [level=1]
- link "Get started"
- link "Star microsoft/playwright on GitHub"
- main:
- img "Browsers (Chromium, Firefox, WebKit)"
- heading "Any browser • Any platform • One API"
```
### Assertion testing

Assertion testing is a targeted approach where you assert specific values or conditions about elements or components. For instance, with Playwright, [Expect(Locator).ToHaveTextAsync()](/api/class-locatorassertions.mdx#locator-assertions-to-have-text) verifies that an element contains the expected text, and [Expect(Locator).ToHaveValueAsync()](/api/class-locatorassertions.mdx#locator-assertions-to-have-value) confirms that an input field has the expected value. Assertion tests are specific and generally check the current state of an element or property against an expected, predefined state. They work well for predictable, single-value checks but are limited in scope when testing the broader structure or variations.

**Advantages**
- **Clarity**: The intent of the test is explicit and easy to understand.
- **Specificity**: Tests focus on particular aspects of functionality, making them more robust against unrelated changes.
- **Debugging**: Failures provide targeted feedback, pointing directly to the problematic aspect.

**Disadvantages**
- **Verbose for complex outputs**: Writing assertions for complex data structures or large outputs can be cumbersome and error-prone.
- **Maintenance overhead**: As code evolves, manually updating assertions can be time-consuming.

### Snapshot testing

Snapshot testing captures a “snapshot” or representation of the entire state of an element, component, or data at a given moment, which is then saved for future comparisons. When re-running tests, the current state is compared to the snapshot, and if there are differences, the test fails. This approach is especially useful for complex or dynamic structures, where manually asserting each detail would be too time-consuming. Snapshot testing is broader and more holistic than assertion testing, allowing you to track more complex changes over time.

**Advantages**
- **Simplifies complex outputs**: For example, testing a UI component's rendered output can be tedious with traditional assertions. Snapshots capture the entire output for easy comparison.
- **Quick Feedback loop**: Developers can easily spot unintended changes in the output.
- **Encourages consistency**: Helps maintain consistent output as code evolves.

**Disadvantages**
- **Over-Reliance**: It can be tempting to accept changes to snapshots without fully understanding them, potentially hiding bugs.
- **Granularity**: Large snapshots may be hard to interpret when differences arise, especially if minor changes affect large portions of the output.
- **Suitability**: Not ideal for highly dynamic content where outputs change frequently or unpredictably.

### When to use
- **Snapshot testing** is ideal for:
- UI testing of whole pages and components.
- Broad structural checks for complex UI components.
- Regression testing for outputs that rarely change structure.
- **Assertion testing** is ideal for:
- Core logic validation.
- Computed value testing.
- Fine-grained tests requiring precise conditions.

By combining snapshot testing for broad, structural checks and assertion testing for specific functionality, you can achieve a well-rounded testing strategy.

## Aria snapshots

In Playwright, aria snapshots provide a YAML representation of the accessibility tree of a page. These snapshots can be stored and compared later to verify if the page structure remains consistent or meets defined expectations.

The YAML format describes the hierarchical structure of accessible elements on the page, detailing **roles**, **attributes**, **values**, and **text content**. The structure follows a tree-like syntax, where each node represents an accessible element, and indentation indicates nested elements.

Each accessible element in the tree is represented as a YAML node:

Expand Down
10 changes: 5 additions & 5 deletions dotnet/docs/ci.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ jobs:
name: 'Playwright Tests'
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright/dotnet:v1.48.0-noble
image: mcr.microsoft.com/playwright/dotnet:v1.49.0-noble
options: --user 1001
steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -164,7 +164,7 @@ trigger:
pool:
vmImage: ubuntu-latest
container: mcr.microsoft.com/playwright/dotnet:v1.48.0-noble
container: mcr.microsoft.com/playwright/dotnet:v1.49.0-noble
steps:
- task: UseDotNet@2
Expand All @@ -187,7 +187,7 @@ Running Playwright on CircleCI is very similar to running on GitHub Actions. In
executors:
pw-noble-development:
docker:
- image: mcr.microsoft.com/playwright/dotnet:v1.48.0-noble
- image: mcr.microsoft.com/playwright/dotnet:v1.49.0-noble
```

Note: When using the docker agent definition, you are specifying the resource class of where playwright runs to the 'medium' tier [here](https://circleci.com/docs/configuration-reference?#docker-execution-environment). The default behavior of Playwright is to set the number of workers to the detected core count (2 in the case of the medium tier). Overriding the number of workers to greater than this number will cause unnecessary timeouts and failures.
Expand All @@ -198,7 +198,7 @@ Jenkins supports Docker agents for pipelines. Use the [Playwright Docker image](

```groovy
pipeline {
agent { docker { image 'mcr.microsoft.com/playwright/dotnet:v1.48.0-noble' } }
agent { docker { image 'mcr.microsoft.com/playwright/dotnet:v1.49.0-noble' } }
stages {
stage('e2e-tests') {
steps {
Expand All @@ -215,7 +215,7 @@ pipeline {
Bitbucket Pipelines can use public [Docker images as build environments](https://confluence.atlassian.com/bitbucket/use-docker-images-as-build-environments-792298897.html). To run Playwright tests on Bitbucket, use our public Docker image ([see Dockerfile](./docker.mdx)).
```yml
image: mcr.microsoft.com/playwright/dotnet:v1.48.0-noble
image: mcr.microsoft.com/playwright/dotnet:v1.49.0-noble
```
### GitLab CI
Expand Down
14 changes: 7 additions & 7 deletions dotnet/docs/docker.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ This Docker image is intended to be used for testing and development purposes on
### Pull the image

```bash
docker pull mcr.microsoft.com/playwright/dotnet:v1.48.0-noble
docker pull mcr.microsoft.com/playwright/dotnet:v1.49.0-noble
```

### Run the image
Expand All @@ -34,15 +34,15 @@ By default, the Docker image will use the `root` user to run the browsers. This
On trusted websites, you can avoid creating a separate user and use root for it since you trust the code which will run on the browsers.

```bash
docker run -it --rm --ipc=host mcr.microsoft.com/playwright/dotnet:v1.48.0-noble /bin/bash
docker run -it --rm --ipc=host mcr.microsoft.com/playwright/dotnet:v1.49.0-noble /bin/bash
```

#### Crawling and scraping

On untrusted websites, it's recommended to use a separate user for launching the browsers in combination with the seccomp profile. Inside the container or if you are using the Docker image as a base image you have to use `adduser` for it.

```bash
docker run -it --rm --ipc=host --user pwuser --security-opt seccomp=seccomp_profile.json mcr.microsoft.com/playwright/dotnet:v1.48.0-noble /bin/bash
docker run -it --rm --ipc=host --user pwuser --security-opt seccomp=seccomp_profile.json mcr.microsoft.com/playwright/dotnet:v1.49.0-noble /bin/bash
```

[`seccomp_profile.json`](https://github.com/microsoft/playwright/blob/main/utils/docker/seccomp_profile.json) is needed to run Chromium with sandbox. This is a [default Docker seccomp profile](https://github.com/docker/engine/blob/d0d99b04cf6e00ed3fc27e81fc3d94e7eda70af3/profiles/seccomp/default.json) with extra user namespace cloning permissions:
Expand Down Expand Up @@ -75,9 +75,9 @@ See our [Continuous Integration guides](./ci.mdx) for sample configs.
See [all available image tags].

We currently publish images with the following tags:
- `:v1.48.0` - Playwright v1.48.0 release docker image based on Ubuntu 24.04 LTS (Noble Numbat).
- `:v1.48.0-noble` - Playwright v1.48.0 release docker image based on Ubuntu 24.04 LTS (Noble Numbat).
- `:v1.48.0-jammy` - Playwright v1.48.0 release docker image based on Ubuntu 22.04 LTS (Jammy Jellyfish).
- `:v1.49.0` - Playwright v1.49.0 release docker image based on Ubuntu 24.04 LTS (Noble Numbat).
- `:v1.49.0-noble` - Playwright v1.49.0 release docker image based on Ubuntu 24.04 LTS (Noble Numbat).
- `:v1.49.0-jammy` - Playwright v1.49.0 release docker image based on Ubuntu 22.04 LTS (Jammy Jellyfish).

:::note

Expand All @@ -99,7 +99,7 @@ Browser builds for Firefox and WebKit are built for the [glibc](https://en.wikip
You can use the [.NET install script](https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-install-script) in order to install different SDK versions:

```bash
curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin --install-dir /usr/share/dotnet --channel 6.0
curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin --install-dir /usr/share/dotnet --channel 9.0
```


Expand Down
81 changes: 81 additions & 0 deletions dotnet/docs/release-notes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,87 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import HTMLCard from '@site/src/components/HTMLCard';

## Version 1.49

### Aria snapshots

New assertion [Expect(Locator).ToMatchAriaSnapshotAsync()](/api/class-locatorassertions.mdx#locator-assertions-to-match-aria-snapshot) verifies page structure by comparing to an expected accessibility tree, represented as YAML.

```csharp
await page.GotoAsync("https://playwright.dev");
await Expect(page.Locator("body")).ToMatchAriaSnapshotAsync(@"
- banner:
- heading /Playwright enables reliable/ [level=1]
- link ""Get started""
- link ""Star microsoft/playwright on GitHub""
- main:
- img ""Browsers (Chromium, Firefox, WebKit)""
- heading ""Any browser • Any platform • One API""
");
```

You can generate this assertion with [Test Generator](./codegen) or by calling [Locator.AriaSnapshotAsync()](/api/class-locator.mdx#locator-aria-snapshot).

Learn more in the [aria snapshots guide](./aria-snapshots).

### Tracing groups

New method [Tracing.GroupAsync()](/api/class-tracing.mdx#tracing-group) allows you to visually group actions in the trace viewer.

```csharp
// All actions between GroupAsync and GroupEndAsync
// will be shown in the trace viewer as a group.
await Page.Context.Tracing.GroupAsync("Open Playwright.dev > API");
await Page.GotoAsync("https://playwright.dev/");
await Page.GetByRole(AriaRole.Link, new() { Name = "API" }).ClickAsync();
await Page.Context.Tracing.GroupEndAsync();
```

### Breaking: `chrome` and `msedge` channels switch to new headless mode

This change affects you if you're using one of the following channels in your `playwright.config.ts`:
- `chrome`, `chrome-dev`, `chrome-beta`, or `chrome-canary`
- `msedge`, `msedge-dev`, `msedge-beta`, or `msedge-canary`

After updating to Playwright v1.49, run your test suite. If it still passes, you're good to go. If not, you will probably need to update your snapshots, and adapt some of your test code around PDF viewers and extensions. See [issue #33566](https://github.com/microsoft/playwright/issues/33566) for more details.

### Try new Chromium headless

You can opt into the new headless mode by using `'chromium'` channel. As [official Chrome documentation puts it](https://developer.chrome.com/blog/chrome-headless-shell):

> New Headless on the other hand is the real Chrome browser, and is thus more authentic, reliable, and offers more features. This makes it more suitable for high-accuracy end-to-end web app testing or browser extension testing.
See [issue #33566](https://github.com/microsoft/playwright/issues/33566) for the list of possible breakages you could encounter and more details on Chromium headless. Please file an issue if you see any problems after opting in.

```xml title="runsettings.xml"
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<Playwright>
<BrowserName>chromium</BrowserName>
<LaunchOptions>
<Channel>chromium</Channel>
</LaunchOptions>
</Playwright>
</RunSettings>
```

```bash
dotnet test -- Playwright.BrowserName=chromium Playwright.LaunchOptions.Channel=chromium
```

### Miscellaneous
- There will be no more updates for WebKit on Ubuntu 20.04 and Debian 11. We recommend updating your OS to a later version.
- `<canvas>` elements inside a snapshot now draw a preview.

### Browser Versions
- Chromium 131.0.6778.33
- Mozilla Firefox 132.0
- WebKit 18.2

This version was also tested against the following stable channels:
- Google Chrome 130
- Microsoft Edge 130

## Version 1.48

### WebSocket routing
Expand Down
4 changes: 2 additions & 2 deletions java/docs/api/class-tracing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,10 @@ Creates a new group within the trace, assigning any subsequent API calls to this
```java
// All actions between group and groupEnd
// will be shown in the trace viewer as a group.
page.context().tracing.group("Open Playwright.dev > API");
page.context().tracing().group("Open Playwright.dev > API");
page.navigate("https://playwright.dev/");
page.getByRole(AriaRole.LINK, new Page.GetByRoleOptions().setName("API")).click();
page.context().tracing.groupEnd();
page.context().tracing().groupEnd();
```

**Arguments**
Expand Down
Loading

0 comments on commit 373b52a

Please sign in to comment.