-
Notifications
You must be signed in to change notification settings - Fork 61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using QUnit.todo() causes build to fail instead of succeed #247
Comments
Krinkle
added a commit
to qunitjs/browserstack-runner
that referenced
this issue
Jun 6, 2021
Important changes: * Mocha 8 and later are now supported. Previously, it's pending tests were incorrectly reported as undefined or failing. * `QUnit.todo()` is now supported. Previously, these intentional failures were reported by browserstack-runner as failing tests. With js-reported 2.1 the runEnd report counts them as `todo` instead of as `failed`. The API that browserstack-runner depends on has not changed since 1.1.0. <https://github.com/js-reporters/js-reporters/blob/v2.1.0/CHANGELOG.md>. Specifically, browserstack-runner captures the following data in lib/_patch/reporter.js, and proccesses it in lib/server.js under the "_progress" and "_report" handlers: * The `testEnd` event, reading: - `name` string, - `suiteName` string, - `status` string, - `errors` array. * The `runEnd` event, reading: - `status` string, - `testCounts` object with ints `total`, `passed`, `failed`, `skipped`, - `runtime` int. These still exist the same way in the latest release: <https://github.com/js-reporters/js-reporters/blob/v2.1.0/spec/cri-draft.adoc#runend> Fixes browserstack#248. Ref browserstack#247.
Krinkle
added a commit
to qunitjs/browserstack-runner
that referenced
this issue
Jun 6, 2021
The update to js-reporters 2.1.0 in the previous commit has fixed the most important problem, which is the build status reported by the `runEnd` event, handled via "_report" submission. However, the todo tests were still printed as errors in the output which is confusing. Before: ``` [Windows 8, Chrome 91.0] Error: "test.todo.each [0]" failed Expected: true Actual: false [Windows 8, Chrome 91.0] Passed: 332 tests, 321 passed, 0 failed, 7 skipped; ran for 1255ms ``` After: ``` [Windows 10, Firefox 88.0] Passed: 332 tests, 321 passed, 0 failed, 7 skipped; ran for 1910ms ``` Fixes browserstack#247.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected Behavior
Passing test.
Actual Behavior
Details
This is happening because when browserstack-runner is consuming the
testEnd
event from js-reporters, it is ignoring thestatus
field. Instead of only priting formatted errors forquery.test.status === "failed"
, it is instead iterating overquery.test.errors
and always print each one as a formatted error, even if it is an expected error (such as the case with a "todo" test).Here is the test suite for the linked example in the browser visually (passing):
And here is what browserstack-runner report for that same build (failing):
https://github.com/qunitjs/qunit/runs/2753071016
The text was updated successfully, but these errors were encountered: