1.0.1 Scheme count, app verification errors

[johnrogers]

Hi all,

After reading about the release of 1.0.1 in this thread, I thought the issue may be solved but it doesn't appear to fix it for me.

I am running Xcode 10.1, and version 1.0.1 of the plugin.

I get the error The app bundle may contain only one scheme, 2 found; when I remove the __xctestrun_metadata__ dict and associated keys from the build_products.zip file and re-package, I get the following error:

app verification failed: There must be only one .xctestrun files in the ZIP file.

There's definitely only one .xctestrun file in my ZIP - that and contents of the Debug-iphoneos folder are the only two items in there.

I tried uploading the ZIP manually via Firebase Console however it gives me the same error (there's a Stack Overflow issue here about it).

Any suggestions?

[MartyCatawiki]

Hi John, it works perfectly for me , can you attach your .xctestrun file to see if this is different?

[powerivq]

@johnrogers Hi, that does not appear to be a plugin issue. The plugin finds all files with .xctestrun extension in the archive and check if there are more than one. Can you verify by inspecting the zip file? Since both the plugin and FTL are reporting the same issue, it is likely that the archive is messed up.

I would suggest you to clean the xcode workspace first before building. Keeping multiple build variants is a sure way of getting into this problem.

[johnrogers]

Hey guys.

Definitely only one .xctestrun file in there, along with one Debug-iphoneos folder. The ZIP file is exactly as output from scan - here are the relevant lines of my fast file:

    scan(slack_url: ENV["SLACK_URL"],
      slack_message: "@here :rotating_light: test failures :rotating_light:",
      slack_only_on_failure: true,
      open_report: false,
      scheme: "<redacted>",
      clean: true,
      skip_detect_devices: true,
      build_for_testing: true,
      sdk: 'iphoneos',
      fail_build: false,
      should_zip_build_products: true
    )
    
    firebase_test_lab_ios_xctest(
      gcp_project: '<redacted>', 
      devices: [                          
        {
            ios_model_id: 'iphonex',
            ios_version_id: '11.2',
            locale: 'en_US',
            orientation: 'portrait'
        }
      ])

fastlane terminal output:

[✔] 🚀 
+-----------------------------------+---------+------------------------------+
|                                Used plugins                                |
+-----------------------------------+---------+------------------------------+
| Plugin                            | Version | Action                       |
+-----------------------------------+---------+------------------------------+
| fastlane-plugin-trainer           | 0.4.0   | trainer                      |
| fastlane-plugin-firebase_test_lab | 1.0.1   | firebase_test_lab_ios_xctest |
+-----------------------------------+---------+------------------------------+

... <scan build process> ...

[09:41:24]: Zipping build products
[09:41:24]: $ cd '/Users/john/Library/Developer/Xcode/DerivedData/<redacted>-ffpetwiwlsslogdbnggaguwiwsmh/Build/Products' && rm -f '/Users/john/Developer/<redacted>/fastlane/test_output/build_products.zip' && zip -r '/Users/john/Developer/<redacted>/fastlane/test_output/build_products.zip' *
[09:41:37]: Succesfully zipped build products: /Users/john/Developer/<redacted>/fastlane/test_output/build_products.zip
[09:41:37]: ------------------------------------------
[09:41:37]: --- Step: firebase_test_lab_ios_xctest ---
[09:41:37]: ------------------------------------------
+------------------------------+------------------------------------------------------------------------------------+
|                                                   Lane Context                                                    |
+------------------------------+------------------------------------------------------------------------------------+
| DEFAULT_PLATFORM             | ios                                                                                |
| PLATFORM_NAME                | ios                                                                                |
| LANE_NAME                    | ios test                                                                           |
| SCAN_ZIP_BUILD_PRODUCTS_PATH | /Users/john/Developer/<redacted>/fastlane/test_output/build_products.zip   |
| SCAN_DERIVED_DATA_PATH       | /Users/john/Library/Developer/Xcode/DerivedData/<redacted>-ffpetwiwlsslogdbnggaguwiwsmh |
| SCAN_GENERATED_PLIST_FILES   | []                                                                                 |
| SCAN_GENERATED_PLIST_FILE    |                                                                                    |
+------------------------------+------------------------------------------------------------------------------------+
[09:41:37]: The app bundle may contain only one scheme, 2 found

+------+------------------------------+-------------+
|                 fastlane summary                  |
+------+------------------------------+-------------+
| Step | Action                       | Time (in s) |
+------+------------------------------+-------------+
| 1    | default_platform             | 0           |
| 2    | cocoapods                    | 57          |
| 3    | scan                         | 287         |
| 💥   | firebase_test_lab_ios_xctest | 0           |
+------+------------------------------+-------------+

[09:41:37]: fastlane finished with errors

[!] The app bundle may contain only one scheme, 2 found

After unzipping the ZIP file that scan produced:

⋊> ~/D/d/f/t/build_products on feature/ui_tests ⨯ pwd                                                                                                                                                                   09:49:14
/Users/john/Developer/<redacted>/fastlane/test_output/build_products            
⋊> ~/D/d/f/t/build_products on feature/ui_tests ⨯ ls                                                                                                                                                                    09:49:30
Debug-iphoneos                      <redacted>_iphoneos12.1-arm64e.xctestrun

The clean: true parameter passed to scan should take care of the cleaning process, right? I'm trying to set this up to be run in a CI environment.

Again, this is using v1.0.1 of the plugin. Any ideas?

[MartyCatawiki]

Hi @johnrogers, thanks for the description, but it's important what's in the xctestrun file, can you please attach it? The plugin counts the number of schemes present in that file, it should only consist of one key with your scheme and one key with xctestrun_metadata

[powerivq]

@johnrogers Yeah that log shows there's only one .xctestrun, that's why it's not showing the "multiple .xctestrun files" error, but instead the "multiple schemes" error. Can you open the .xctestrun file and see what's in it?

[johnrogers]

Ok, I'm getting somewhere. I wasn't specifying the correct scheme in scan's scheme parameter.

I've hit another snag though - I am getting the following error:

The app bundle is not a UI test bundle. Did you build with build-for-testing argument?

Here is the complete output (minus redacted) from my .xctestrun file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>__xctestrun_metadata__</key>
	<dict>
		<key>FormatVersion</key>
		<integer>1</integer>
	</dict>
	<key>redactedUITests</key>
	<dict>
		<key>BundleIdentifiersForCrashReportEmphasis</key>
		<array>
			<string>redacted</string>
			<string>redacted</string>
			<string>redacted</string>
		</array>
		<key>CommandLineArguments</key>
		<array/>
		<key>DependentProductPaths</key>
		<array>
			<string>__TESTROOT__/Debug-iphoneos/redacted.framework</string>
			<string>__TESTROOT__/Debug-iphoneos/redacted-Runner.app/PlugIns/redacted.xctest</string>
			<string>__TESTROOT__/Debug-iphoneos/redacted.app</string>
		</array>
		<key>EnvironmentVariables</key>
		<dict>
			<key>OS_ACTIVITY_DT_MODE</key>
			<string>YES</string>
			<key>SQLITE_ENABLE_THREAD_ASSERTIONS</key>
			<string>1</string>
			<key>TESTS_USE_DEMO</key>
			<string>1</string>
		</dict>
		<key>IsUITestBundle</key>
		<true/>
		<key>IsXCTRunnerHostedTestBundle</key>
		<true/>
		<key>ProductModuleName</key>
		<string>redactedUITests</string>
		<key>RunOrder</key>
		<integer>0</integer>
		<key>SystemAttachmentLifetime</key>
		<string>deleteOnSuccess</string>
		<key>TestBundlePath</key>
		<string>__TESTHOST__/PlugIns/redactedUITests.xctest</string>
		<key>TestHostBundleIdentifier</key>
		<string>com.apple.test.redactedUITests-Runner</string>
		<key>TestHostPath</key>
		<string>__TESTROOT__/Debug-iphoneos/redactedUITests-Runner.app</string>
		<key>TestingEnvironmentVariables</key>
		<dict>
			<key>DYLD_FRAMEWORK_PATH</key>
			<string></string>
		</dict>
		<key>ToolchainsSettingValue</key>
		<array/>
		<key>UITargetAppCommandLineArguments</key>
		<array/>
		<key>UITargetAppPath</key>
		<string>__TESTROOT__/Debug-iphoneos/redcated.app</string>
		<key>UserAttachmentLifetime</key>
		<string>deleteOnSuccess</string>
	</dict>
</dict>
</plist>

I did some digging into the plugin's source, and found this section of code in ios_validation.rb:

conf = Plist.parse_xml(xctestrun_files.first.get_input_stream)

...

_, scheme_conf = conf.first
unless scheme_conf["IsUITestBundle"]
    UI.user_error!("The app bundle is not a UI test bundle. Did you build with build-for-testing argument?")
end

Now, I'm not that great with Ruby, but it looks like conf is an array of keys and their values - it's checking to see if __xctestrun_metadata__ is present and reducing the size in order to detect the number of schemas, however, this line:

scheme_conf = conf.first

Looks like it's assigning the first key-value pair to scheme_conf. In the case of my .xctestrun file, this would be __xctestrun_metadata__. Seems to me that a better way to do this would be to remove the first key-value pair altogether, then use conf.first?

Or am I missing something here? I've tried the build process several time and every time it comes out true for IsUITestBundle.

[johnrogers]

I've created a PR to demonstrate a fix (works for me). It seems to me that we're better off selecting the schema that's not __xctestrun_metadata__, right?

[johnrogers]

Not specifically plugin-related, but...

Now we've worked out a fix for the plugin, I have to work out why Test Lab is telling me that I have an invalid ZIP file. Whether I use the plugin or the manual upload via Firebase Console, I get an error telling me that I should only have one xctestrun file in my ZIP file along with the contents of the Products folder.

Given that the validation is passing on the plugin, I really can't fathom this error. I've double checked the ZIP millions of times and tried a few different ways of packaging it. Below is a snippet of how it was packaged by scan. I've sent a message to Firebase help but not hopeful I'll get a quick reply. Any ideas?

⋊> ~/D/d/f/test_output on feature/ui_tests ⨯ ls                                                                                                                                                    08:56:30
build_products     build_products.zip report.html        report.junit
⋊> ~/D/d/f/test_output on feature/ui_tests ⨯ cd build_products/                                                                                                                                    09:38:49
⋊> ~/D/d/f/t/build_products on feature/ui_tests ⨯ ls                                                                                                                                               09:38:52
Debug-iphoneos                               <redacted> UI Tests_iphoneos12.1-arm64e.xctestrun

[✔] Uploading the app to GCS... Done
[09:38:09]: Submitting job(s) to Firebase Test Lab
[09:38:11]: Matrix ID for this submission: matrix-ra6mhdv4r70na
[✖] Starting tests... Failed
[09:38:32]: The XCTest zip file was malformed. The zip did not contain a single .xctestrun file and the contents of the DerivedData/Build/Products directory.

[MartyCatawiki]

@johnrogers Maybe you can try-out without the fail_build-option in scan, i don't use that one.
One tip: The Firebase Community on Slack is very active, its firebase-community.slack.com, there is a special channel for test-lab.

[johnrogers]

@johnrogers Maybe you can try-out without the fail_build-option in scan, i don't use that one.
One tip: The Firebase Community on Slack is very active, its firebase-community.slack.com, there is a special channel for test-lab.

Thanks Marty. They're onto it!

Issue track here: https://issuetracker.google.com/issues/121101228