You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This doesn't really provide much useful information for anything that may want to consume this information (CI being the prime example). The xunit xml spec allows any string to be inside this tag.
I am not intimately familiar with XCTest, so as the fixme mentioned below indicates it may be that there just needs to be better reporting given by XCTIssue to include more relevant details.
Possible Fix
As the FIXME comment here suggests, we could simply stick the result.output onto line 1284
""" if !result.success { content += "<failure message=\"failed\">\(result.output)</failure>\n" }"""
which generates XML that looks like this (just has stdout from the test invocation mushed into it):
<testcaseclassname="ExampleTests.Tests"name="testFail"time="0.029471092">
<failuremessage="failed">Test Suite 'Selected tests' started at 2024-06-01 21:02:59.244
Test Suite 'ExampleTests' started at 2024-06-01 21:02:59.246
Test Case 'Tests.testFail' started at 2024-06-01 21:02:59.246
/home/test-ex/Tests/test.swift:8: error: Tests.testFail : XCTAssertEqual failed: ("1") is not equal to ("2") -
/home/test-ex/Tests/test.swift:9: error: Tests.testFail : XCTAssertEqual failed: ("foo") is not equal to ("bar") -
/home/test-ex/Tests/test.swift:10: error: Tests.testFail : XCTAssertTrue failed -
Test Case 'Tests.testFail' failed (0.001 seconds)
Test Suite 'ExampleTests' failed at 2024-06-01 21:02:59.247
Executed 1 test, with 3 failures (0 unexpected) in 0.001 (0.001) seconds
Test Suite 'Selected tests' failed at 2024-06-01 21:02:59.247
Executed 1 test, with 3 failures (0 unexpected) in 0.001 (0.001) seconds
<failure>
That isn't ideal, but it's at least more detail than is given currently
If the full output is too verbose, there's plenty of room to process it down to something cleaner, but that would obviously be beholden to the output format remaining the same.
This does seem generally like it may need a wider refactor to pull additional information from the test reporter though, adjacent issues:
Description
Problem
Currently, when creating an xunit xml output file with the
--xunit-output
option inswift test
, test failures generate an empty<failure>
tag:This doesn't really provide much useful information for anything that may want to consume this information (CI being the prime example). The xunit xml spec allows any string to be inside this tag.
I am not intimately familiar with
XCTest
, so as the fixme mentioned below indicates it may be that there just needs to be better reporting given byXCTIssue
to include more relevant details.Possible Fix
As the FIXME comment here suggests, we could simply stick the
result.output
onto line 1284which generates XML that looks like this (just has stdout from the test invocation mushed into it):
That isn't ideal, but it's at least more detail than is given currently
If the full output is too verbose, there's plenty of room to process it down to something cleaner, but that would obviously be beholden to the output format remaining the same.
This does seem generally like it may need a wider refactor to pull additional information from the test reporter though, adjacent issues:
Expected behavior
xunit.xml
failures include relevant failure informationActual behavior
xunit.xml
failures are emptySteps to reproduce
Run
swift test --parallel --xunit-output test.xml
in any project with a test failureSwift Package Manager version/commit hash
5.10
Swift & OS version (output of
swift --version && uname -a
)Swift version 5.10 (swift-5.10-RELEASE)
Ubuntu 20.04 & macOS
The text was updated successfully, but these errors were encountered: