Upgrading Xcode version from 15.2 to {15.3-15.4} results in CI/CD simulator architecture error

[dvanluijpen]

Description

We've been running our unit and snapshot tests through our Azure DevOps CI/CD pipeline for a while now, all going well. Now we decided to upgrade our current settings to the new runner, xcode version and simulator.

Working settings before upgrade:

- name: runnerImage
  value: 'macOS-13'
- name: XcodeVersion
  value: 'Xcode_15.2.app'
- name: iOSSimulator
  value: 'iPhone 15 Pro,OS=17.2'

Failing settings after upgrade:

- name: runnerImage
  value: 'macOS-14'
- name: XcodeVersion
  value: 'Xcode_15.4.app'
- name: iOSSimulator
  value: 'iPhone 15 Pro,OS=17.5'

Error message
xcodebuild: error: Failed to build project with scheme .: Cannot test target Tests on iPhone 15 Pro: Tests does not support any of iPhone 15 Pro’s architectures: x86_64.

With some trial and error on the steps between this upgrade, we managed to have successful tests using macOS-14 and iOSSimulator iPhone 15 Pro,OS=17.5. However, it seems the Xcode version is causing the error. Using macOS-14 and iPhone 15 Pro,OS=17.5, combined with Xcode_15.2.app, we get successful tests. Using Xcode_15.3.app or Xcode15.4.app, we get above error message.

To be complete: Our builds on release are succeeding. Our tests on debug are failing. We use xcodegen to generate our project. Based on intensive google search I found some hints towards some build settings that could be changed, like setting Architectures to x86_64 for Debug (though that cannot be the option since we also need to debug on devices instead of just simulator) or setting Build Active Architectures Only to No, so x86_64 would also be embedded in the build.

Looking for a fix when this ends up being a bug, or some hints to resolve this failed tests on Azure DevOps.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

Runner Image
Image: macos-14
Version: 20240616.1

Is it regression?

Yes, same runner image however. Failure started upgrading from Xcode app 15.2 to 15.3. Still exists on 15.4.

Expected behavior

Tests succeeding like they did on Xcode_15.2.app, where the x86_64 architecture for iOS Simulator is not an issue.

Actual behavior

Tests are failing on Xcode_15.3.app and Xcode_15.4.app, where the x86_64 architecture for iOS Simulator is an issue.

Repro steps

1. Setup an Azure DevOps Pipeline
2. Configure a job to use the vmImage: macOS-14 from the pool
3. Setup xcode-select to use a specific xcode version (to succeed: Xcode_15.2.app -- to fail either: Xcode_15.3.app or Xcode_15.4.app
4. Configure an Xcode@5 task with actions: test, scheme: Debug and destinationPlatform: iPhone 15 Pro,OS=17.5
5. Run the pipeline

[erik-bershel]

Hey @dvanluijpen!
Please clarify a few points. You say this is a regression. Does this mean that you have not observed this behaviour before with a different version of the image (2024xxxx.y) or when running your tests in the local environment?
Could you also please provide more detailed steps to reproduce the problem? I'm afraid that we won't be able to run tests without a minimum project.

[erik-bershel]

I’ll make a special note. The behaviour of some simulators can indeed vary greatly between versions of Xcode. I cannot say that this is your case. Example: #8621. There is also some perf issues: #9591