Issue: Lack of iOS Support for Swift Benchmark

[sinanege-tomtom]

Description

I attempted to use Swift Benchmark for performance testing within an iOS project. While the package looks working for macOS and iOS, it currently does not support iOS, resulting in an error during the benchmarking process.

Steps to Reproduce

Create a Swift Package with the following Package.swift configuration:

// swift-tools-version:5.9
import PackageDescription

let package = Package(
    name: "PerformanceTest",
    platforms: [.iOS(.v16)],
    products: [
        .library(
            name: "PerformanceTest",
            targets: ["PerformanceTest"]),
    ],
    dependencies: [
        .package(url: "https://github.com/ordo-one/package-benchmark", branch: "main"),
    ],
    targets: [
        .target(
            name: "PerformanceTest"
        ),
        .executableTarget(
            name: "MyNewBenchmarkTarget",
            dependencies: [
                "PerformanceTest",
                .product(name: "Benchmark", package: "package-benchmark"),
            ],
            path: "Benchmarks/MyNewBenchmarkTarget"
        ),
    ]
)

When I run swift package benchmark run it gives me following error:

error: the executable 'MyNewBenchmarkTarget' requires macos 10.13, but depends on the product 'Benchmark' which requires macos 13.0; consider changing the executable 'MyNewBenchmarkTarget' to require macos 13.0 or later, or the product 'Benchmark' to require macos 10.13 or earlier.

Although iOS support appears to be included in the repository, I have not been able to get it to work. Could you please assist with this issue?

[hassila]

What happens if you add the macOS platform requirement too as suggested? Just copy the platform requirements you linked to.

[sinanege-tomtom]

It builds successfully. problem begins when we add iOS specific dependecies to the package then it started to give same error. The dependencies to be added to the new package must be built for both macOS and iOS. It does not work only with iOS ones as far as i understood

[hassila]

Alright, we don't use anything except Linux and macOS and that is what we test - would be happy to review any PR that enables iOS support if you can get it work though - not sure what the constraints are there though - the benchmark plugin starts child processes to run the benchmarks - perhaps that's not ok?

[hassila]

@axelandersson maybe you have some idea?

[axelandersson]

The original error is due to it using a default platform of macOS 10.13 if you don't pass any other macOS version in Package(platforms:). Adding .macOS(.v13) makes it build.

The bigger issue is iOS not being supported as a benchmark platform though, due to processes not being allowed to fork or launch other processes on iOS. I don't remember the exact details, but I think there are also privacy limitations on iOS system calls to query device information, in order to prevent fingerprinting—uniquely identifying a device.

@sinanege-tomtom Can you elaborate a bit on what you're trying to benchmark? Would it be possible to extract that code into some shared library that can be built for both iOS and macOS, and then benchmark that instead?

[stephencelis]

I came across this issue when trying to add benchmarks to a package with macOS 10.15/iOS 13 support. Is this just not supported via this package or SPM? Is the workaround a completely separate package for benchmarks?

[hassila]

Hi @stephencelis, the best practice would be to embed a separate package for SPM, see e.g. how Foundation does it - then you don't "pollute" the top level Package.swift with the Benchmark dependency, but more importantly can use a more recent tools version for the benchmarks (e.g. move completely to Swift 6 for example).