Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions agents/agents/issue-poster.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ Apply these and no others:
- One `area/*` where the area is clear from the issue, and none where it is not. `infrastructure`
for the build and the tracker.
- When the issue is about the guarantee: `security`, `needs-security-review`, and a `severity/*`.
- `flaky-test`, in addition to the labels above, when the issue is about a test whose outcome
changes between runs of the same build.
- The axes `importance/*`, `urgency/*` and `size/*`, proposed from the scales in
`labelling-issues.md`, when the task says a person asked for this issue. Never `urgency/p1`.
Apply no axis to a security issue, or when the task says the issue is a finding nobody has read.
Expand Down
13 changes: 7 additions & 6 deletions contrib/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,12 +231,13 @@ contrib/measure-flakes.py --runs 30 --test-threads 4 # what make check-linux c
Nothing fails over a rate. A job that goes red for a known two percent flake is a job people learn
to ignore, which is the problem this exists for, so a rate is reported and never gated on.

`--file-issues OWNER/REPO` opens one issue per flaky test, titled `Flaky test: <name>`, saying the
rate, the parallelism it was measured at and what the failure said. A test that already has an issue
under that title gets nothing, whoever opened it, so a weekly run adds only what is new instead of
filing the same test every Monday. `--assign LOGIN` puts that login on every issue the run opens:
nothing here is triaged, so an issue naming nobody waits until a person notices it and assigns it by
hand. The workflow passes `netzenbot`; a run by hand assigns nobody unless it says otherwise.
`--file-issues OWNER/REPO` opens one issue per flaky test, titled `Flaky test: <name>` and labelled
`flaky-test`, saying the rate, the parallelism it was measured at and what the failure said. A test
that already has an issue under that title gets nothing, whoever opened it, so a weekly run adds
only what is new instead of filing the same test every Monday. `--assign LOGIN` puts that login on
every issue the run opens: nothing here is triaged, so an issue naming nobody waits until a person
notices it and assigns it by hand. The workflow passes `netzenbot`; a run by hand assigns nobody
unless it says otherwise.
`--dry-run` prints what it would send and sends nothing; `--issue-limit` bounds how many one run may
open, since the first measurement of a suite nobody has measured could otherwise arrive as twenty
issues at once. The exit code is still not a verdict on
Expand Down
28 changes: 27 additions & 1 deletion contrib/measure-flakes.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,11 @@
# issue for is recognised as covered rather than filed a second time.
TITLE = "Flaky test"

# What every issue this opens carries, so `label:flaky-test` finds them whoever filed them. It is
# the only label: nothing this files has been read by a person, so no kind, area or axis is claimed
# for it. `gh issue create` refuses a label the repository lacks, and a person creates labels.
LABEL = "flaky-test"

# How many issues one run may open. A suite nobody has measured could have twenty flaky tests in it,
# and twenty issues arriving at once is a thing people close in a batch without reading. The rest
# are named in the report and the next run files them.
Expand Down Expand Up @@ -467,7 +472,7 @@ def file_issues(flaky, samples, threads, repo, assign, limit, gh):
if filed >= limit:
held.append(test)
continue
create = ["issue", "create", "--repo", repo, "--title", title]
create = ["issue", "create", "--repo", repo, "--title", title, "--label", LABEL]
# No login is no flag at all rather than an empty one, the way an empty --file-issues files
# nothing: a run by hand assigns nobody, since the person running it is already looking.
if assign:
Expand Down Expand Up @@ -841,6 +846,19 @@ def assignees(calls):
]


def labels(calls):
"""The labels each issue a recording was asked to open would carry."""
return [
[
arguments[at + 1]
for at, argument in enumerate(arguments)
if argument == "--label"
]
for arguments, _ in calls
if arguments[:2] == ["issue", "create"]
]


def round_trip(outcomes, second, samples):
"""Whether --save then --load restores the threads and the machine and files without a report.

Expand Down Expand Up @@ -1050,6 +1068,14 @@ def selftest():
)
)

# The tracker is searched by this label, and an issue without it is one the search never lists.
checks.append(
(
"every issue carries the flaky-test label and no other",
labels(calls) == [["flaky-test"], ["flaky-test"]],
)
)

unowned, calls = recording_gh()
file_issues(flaky, samples, 16, "owner/repo", "", ISSUE_LIMIT, unowned)
checks.append(
Expand Down
11 changes: 6 additions & 5 deletions docs/development/checks.md
Original file line number Diff line number Diff line change
Expand Up @@ -429,11 +429,12 @@ sandbox is the cause. A test that fails under `run` and has no such skip is a bu
[Test determinism](../../.github/workflows/test-determinism.yml) workflow runs the suite a hundred
times against one build, on Linux at four threads and at sixteen and on macOS and Windows at cargo's
default, and names every test whose outcome changed, with the rate, the platform and the panic it
produced. Each one gets an issue titled `Flaky test: <name>`, filed by a separate job that holds the issue
grant and compiles nothing, except from the Windows leg, which
files nothing because the suite is not expected to pass there yet (#88). A test that already has an
issue gets nothing further, so the list of open flakes is the search rather than
a job summary somebody has to remember to read. Search the issues before treating a failure as a
produced. Each one gets an issue titled `Flaky test: <name>` and labelled `flaky-test`
([labelling-issues.md](labelling-issues.md#flaky-tests)), filed by a separate job that holds the
issue grant and compiles nothing, except from the Windows leg, which files nothing because the
suite is not expected to pass there yet (#88). A test that already has an issue gets nothing
further, so the list of open flakes is the search rather than a job summary somebody has to
remember to read. Search the issues before treating a failure as a
known flake, and run `contrib/measure-flakes.py --runs 30` to ask this machine over a shorter sweep.
Neither fails a build over a rate.

Expand Down
18 changes: 18 additions & 0 deletions docs/development/labelling-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,24 @@ previously named `blocked`:
gh label edit blocked --repo brave/bravebot --name blocked/needs-planning
```

## Flaky tests

`flaky-test` says the issue is about a test whose outcome changes between runs of the same build.
It marks what the issue is about and is not a kind, so `is:open label:flaky-test` lists every flake
however it was filed.

The Test determinism workflow files one issue per flaky test, titled `Flaky test: <name>`, through
`contrib/measure-flakes.py`. It applies `flaky-test` and no other label, because nobody has read
what it files, so no kind, `area` or axis is claimed for it. The poster applies `flaky-test` to an
issue about a flaky test filed from a session, in addition to the kind and the `area` that issue
would otherwise get. The poster refuses to post while `flaky-test` does not exist, because a person
creates labels:

```bash
gh label create flaky-test --repo brave/bravebot --color 5ff714 \
--description "A test whose outcome changes between runs of the same build"
```

## Security

`security` says the issue is about the guarantee this repository exists for, and it goes on with a
Expand Down
Loading