Briefcase CI doesn't fully validate that Briefcase can be executed

[freakboy3742]

Describe the bug

#1377 added support for Android APK builds. It fully passed CI, and was merged.

The problem is that it added a build_command() method to the Gradle mixing which overrode the build_command() that... does actual builds.

This wasn't picked up in testing because manual testing only checked run without a cascading build requirement; and CI doesn't invoke the __call__ method so as to trigger the full cascade of build dependencies. It wasn't picked up during the CI builds of apps because Briefcase's CI doesn't use the version of briefcase from its own branch to build apps.

Steps to reproduce

The problem here is that there isn't an error.

If you actually build a project, you see the error

1. Generate a hello world project
2. run briefcase run android
3. See error

Traceback (most recent call last):
  File "/Users/runner/hostedtoolcache/Python/3.10.12/x64/bin/briefcase", line 8, in <module>
    sys.exit(main())
  File "/Users/runner/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/briefcase/__main__.py", line 25, in main
    command(**options)
  File "/Users/runner/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/briefcase/commands/run.py", line 287, in __call__
    state = self.build_command(
TypeError: GradleMixin.build_command() got an unexpected keyword argument 'update'

However, The CI for #1377 didn't generate such an error.

Expected behavior

A gradle build should complete without error.

Screenshots

No response

Environment

• Operating System: all
• Python version: all
• Software versions:
  • Briefcase: Main branch, as of 9c0c752

Logs

Additional context

This was picked up because Toga builds started failing CI e.g., this CI build in Toga.

[rmartin16]

It wasn't picked up during the CI builds of apps because Briefcase's CI doesn't use the version of briefcase from its own branch to build apps.

CI creates a wheel from the version of Briefcase that triggered the workflow run. The "verify app build" workflow installs this wheel for running the testing.

So, maybe if CI runs briefcase run android and transitively calls build_command(), this would have been caught??

[freakboy3742]

You're right - my analysis wasn't correct. Briefcase doesn't run briefcase run at all. Toga's tests do run briefcase run, and do so in a transitive way - hence why the problem was triggered.

This particular problem also doesn't manifest if you explicitly run briefcase create android, briefcase build android, briefcase run android - it's only a run with an implied build that triggers the problem.

However, outside of doing a verification run that does all possible combinations of transitive builds, I'm not sure how we avoid this class of error. The only alternative I can see is unit tests that invoke all the possible transitive combinations, and having those on every possible platform.... maybe if we mock tools entirely and don't pay any attention to output results, just "does the command complete"?

[freakboy3742]

Briefcase's CI now runs Briefcase as part of a CI run, so I think we can close this.