Skip to content

Empty failure tags on iOS JUnit XML report #999

@msd117c

Description

@msd117c

Describe the bug
When running iOS tests we have detected failed tests with no content in the failure tag:

    ...
    <testcase name="testAccessibility" time="13.387" classname="TestTarget.Tests">
      <failure></failure>
    </testcase>
    ...

To Reproduce
Steps to reproduce the behaviour:

  1. Full configuration
# Check Marathon documentation in:
# https://docs.marathonlabs.io/intro/configure
# https://github.com/MarathonLabs
name: "E2E tests"
outputDir: "./reports"
screenRecordingPolicy: "ON_FAILURE"
deviceInitializationTimeoutMillis: 480000
debug: true
uncompletedTestRetryQuota: 0
batchingStrategy:
  type: "fixed-size"
  size: 10
shardingStrategy:
  type: "count"
  count: 1
executionStrategy:
  mode: ALL_SUCCESS
  fast: true
vendorConfiguration:
  type: "iOS"
  bundle:
    application: "./App.app"
    testApplication: "./Tests.xctest"
    testType: "xcuitest"
  xcresult:
    pull: true
    remoteClean: true
  xcodebuildTestArgs:
    "-test-timeouts-enabled": "YES"
    "-maximum-test-execution-time-allowance": "180"
  testParserConfiguration:
    type: "xctest"
  screenRecordConfiguration:
    preferableRecorderType: "screenshot"
  compactOutput: true
  permissions:
    grant:
      - all
  devices: "./Marathondevices"
outputConfiguration:
  maxPath: 1024
filteringConfiguration:
  allowlist:
    - type: "method"
      values:
        - "testAccessibility"
  1. Any additional information about the device provider (devices that were stuck, etc.)
    • The error is because the log from the xcrun does not indicate the class throwing / causing the test to fail, and it just prints <unknown>:0 instead of class/path/:line format.
    • It seems to happen consistently when an a11y test fails.

Expected behavior
The error message displayed on the logs and the xcresult instead of an empty tag.

Logs and reports

...
<DebugLogPrinter> <unknown>:0: error: -[TestTarget.Tests testAccessibility] : Element has no description
<DebugLogPrinter>     t =     4.79s Find the Any (Descendant with identity binding to AX element pid: 62723, elementOrHash.elementID: 105553170498336.190)
<DebugLogPrinter>     t =     4.80s Checking existence of `"empty state view" StaticText`
<DebugLogPrinter>     t =     4.81s Find the "empty state view" StaticText
...

Devices (please complete the following information):

  • Device: iPhone 16
  • OS: iOS 18

Additional context
PR with an initial proposal to fix the issue: #1000

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions