Create template for failing tests. #21728

ryanthompson591 · 2022-06-06T21:32:34Z

This is a template for when tests are red and makes creating failing test issues easier.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Add a link to the appropriate issue in your description, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

asf-ci · 2022-06-06T21:32:36Z

Can one of the admins verify this patch?

asf-ci · 2022-06-06T21:32:36Z

Can one of the admins verify this patch?

damccorm

Mostly LGTM, just had a few notes

damccorm · 2022-06-06T21:42:51Z

.github/ISSUE_TEMPLATE/failing_test.yml

+name: Bug Report
+description: File a bug report


These probably need to be changed to something like "Failing Test"

.github/ISSUE_TEMPLATE/failing_test.yml

damccorm · 2022-06-06T21:49:53Z

.github/ISSUE_TEMPLATE/failing_test.yml

+
+          If this isn't a bug and you have a question or support request, please email [email protected] with a description of the problem instead of opening this issue.
+    validations:
+      required: true


Might be good to add a check box as well that asks something along the lines of "Is this a known flaky test?"

I expect someone reporting a flaky test would be reasonably likely to choose this template, so it would be good to cover that one. You could even add an entry to this file -

beam/.github/issue-rules.yml

Line 17 in 3c623ab

- valueFor: 'Priority'

- so that it automatically gets labeled "flaky" (docs on how that file works are here - https://github.com/damccorm/tag-ur-it)

Looks like no checkboxes yet.
damccorm/tag-ur-it#21

How about a dropdown with an option to mark flakey or red.

Co-authored-by: Danny McCormick <[email protected]>

codecov · 2022-06-07T01:16:58Z

Codecov Report

Merging #21728 (5b84992) into master (c4cad83) will not change coverage.
The diff coverage is n/a.

❗ Current head 5b84992 differs from pull request most recent head f45389b. Consider uploading reports for the commit f45389b to get more accurate results

@@           Coverage Diff           @@
##           master   #21728   +/-   ##
=======================================
  Coverage   73.46%   73.46%           
=======================================
  Files         714      714           
  Lines       96497    96497           
=======================================
  Hits        70896    70896           
  Misses      24279    24279           
  Partials     1322     1322

Flag	Coverage Δ
go	`51.45% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

damccorm · 2022-06-07T13:48:42Z

.github/ISSUE_TEMPLATE/failing_test.yml

+  - type: dropdown
+    id: flakey
+    attributes:
+      label: Issue Component


This should probably be something like "Is this test flaky or consistently failing?"

I think this comment is resolved.

damccorm · 2022-06-07T13:53:15Z

.github/ISSUE_TEMPLATE/failing_test.yml

+      - "flakey"
+      - "red"


Suggested change

- "flakey"

- "red"

- "Test is flaky"

- "Test is consistently failing"

Then in the tagging config we can search for the string "Test is flaky" and add the flaky label.

Also prefer spelling flaky instead of flakey for consistency with existing labels

Could you also update the wording to something like "Test is consistently failing"/"Test is flaky"? A newer contributor might not grok the difference between red and flaky (because a flaky test is sometimes red), and that gives us a string to search for in our labeler config.

Second, would you mind adding:

- contains: 'Test is flaky' # (or whatever string you decide on) addLabels: ['flaky']

to the rules in .github/issue-rules.yml

.github/issue-rules.yml

aaltay · 2022-06-18T03:35:07Z

What is the next step on this PR?

damccorm

This is waiting on the author. I also just added one more comment, once that and the other lingering comment are addressed we can merge

.github/ISSUE_TEMPLATE/failing_test.yml

Co-authored-by: Danny McCormick <[email protected]>

ryanthompson591

I think I fixed the merge problem. Have a look. One of the problems I have, is I don't have a good way to test this change to see if it works. Do you know how to do that?

ryanthompson591 · 2022-11-21T17:01:28Z

.github/ISSUE_TEMPLATE/failing_test.yml

+  - type: dropdown
+    id: flakey
+    attributes:
+      label: Issue Component


I think this comment is resolved.

.github/ISSUE_TEMPLATE/failing_test.yml

damccorm · 2022-11-21T17:15:04Z

.github/ISSUE_TEMPLATE/failing_test.yml

+      options:
+      - "Test is flaky"
+      - "Test is continually failing"
+    required: true


Suggested change

required: true

validations:

required: true

.github/ISSUE_TEMPLATE/failing_test.yml

damccorm · 2022-11-21T17:21:30Z

I think I fixed the merge problem. Have a look. One of the problems I have, is I don't have a good way to test this change to see if it works. Do you know how to do that?

If you merge it into master of your repo, you can try creating issues and validate that it does the right thing (that should give you the template).

If you don't want to do that in your main repo, you can copy the templates and labeler actions config to a test repo, that's a little more work but keeps your fork clean

Co-authored-by: Danny McCormick <[email protected]>

ryanthompson591 · 2022-11-21T19:32:09Z

I tried this out in my repo and it worked fine. It's a good thing I tested this first because I needed a small change.

damccorm

LGTM

initial template

78af9ad

github-actions bot added the build label Jun 6, 2022

damccorm reviewed Jun 6, 2022

View reviewed changes

ryanthompson591 and others added 2 commits June 6, 2022 20:17

Update .github/ISSUE_TEMPLATE/failing_test.yml

a9a9752

Co-authored-by: Danny McCormick <[email protected]>

Update .github/ISSUE_TEMPLATE/failing_test.yml

fa6e0c6

Co-authored-by: Danny McCormick <[email protected]>

ryanthompson591 added 3 commits June 6, 2022 21:21

added checkbox for flakey

354f933

merge

f0d65f0

fix issue type

209dd72

damccorm reviewed Jun 7, 2022

View reviewed changes

ryanthompson591 added 2 commits June 7, 2022 10:04

flaky

0f3e532

added issue rules

ff4ab52

damccorm reviewed Jun 8, 2022

View reviewed changes

.github/issue-rules.yml Outdated Show resolved Hide resolved

damccorm reviewed Jun 21, 2022

View reviewed changes

.github/ISSUE_TEMPLATE/failing_test.yml Outdated Show resolved Hide resolved

ryanthompson591 and others added 10 commits November 21, 2022 09:39

Update .github/ISSUE_TEMPLATE/failing_test.yml

a6ebfe9

Co-authored-by: Danny McCormick <[email protected]>

initial template

4c75c80

added checkbox for flakey

54b6629

Update .github/ISSUE_TEMPLATE/failing_test.yml

74a11b8

Co-authored-by: Danny McCormick <[email protected]>

Update .github/ISSUE_TEMPLATE/failing_test.yml

1f1b7c3

Co-authored-by: Danny McCormick <[email protected]>

fix issue type

7657fc5

flaky

ddef58a

added issue rules

e55941f

fixed merge

18de0b9

changed template

5b84992

github-actions bot added core dataflow labels Nov 21, 2022

github-actions bot removed go runners jet samza learning java extensions core dataflow fn-execution spark direct docker kotlin twister2 io portability model flink labels Nov 21, 2022

ryanthompson591 commented Nov 21, 2022

View reviewed changes

damccorm reviewed Nov 21, 2022

View reviewed changes

ryanthompson591 and others added 5 commits November 21, 2022 12:46

Update .github/ISSUE_TEMPLATE/failing_test.yml

13141e4

Co-authored-by: Danny McCormick <[email protected]>

Update .github/ISSUE_TEMPLATE/failing_test.yml

c37467c

Co-authored-by: Danny McCormick <[email protected]>

copied bug info so it is the same

94d93a7

fixed bad template

e9a25bf

returned issue-rules to state

f45389b

damccorm approved these changes Nov 21, 2022

View reviewed changes

damccorm merged commit 4269e8a into apache:master Nov 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create template for failing tests. #21728

Create template for failing tests. #21728

ryanthompson591 commented Jun 6, 2022 •

edited

Loading

asf-ci commented Jun 6, 2022

asf-ci commented Jun 6, 2022

damccorm left a comment

damccorm Jun 6, 2022

ryanthompson591 Jun 7, 2022

damccorm Jun 6, 2022

ryanthompson591 Jun 7, 2022

codecov bot commented Jun 7, 2022 •

edited

Loading

damccorm Jun 7, 2022

ryanthompson591 Nov 21, 2022

damccorm Jun 7, 2022

ryanthompson591 Jun 7, 2022

damccorm Jun 7, 2022

aaltay commented Jun 18, 2022

damccorm left a comment

ryanthompson591 left a comment

ryanthompson591 Nov 21, 2022

damccorm Nov 21, 2022

damccorm commented Nov 21, 2022

ryanthompson591 commented Nov 21, 2022

damccorm left a comment

Create template for failing tests. #21728

Create template for failing tests. #21728

Conversation

ryanthompson591 commented Jun 6, 2022 • edited Loading

GitHub Actions Tests Status (on master branch)

asf-ci commented Jun 6, 2022

asf-ci commented Jun 6, 2022

damccorm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 7, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaltay commented Jun 18, 2022

damccorm left a comment

Choose a reason for hiding this comment

ryanthompson591 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

damccorm commented Nov 21, 2022

ryanthompson591 commented Nov 21, 2022

damccorm left a comment

Choose a reason for hiding this comment

ryanthompson591 commented Jun 6, 2022 •

edited

Loading

codecov bot commented Jun 7, 2022 •

edited

Loading