CLI Driven Execution

[alexplischke]

Context

saucectl relies on the majority of test settings to be set inside its config.yml, while providing flags to override some of those settings via CLI flags.

Without said config.yml, the user cannot run any tests at all.

While this approach works well for CI pipelines, it impedes the feedback cycle time when throubleshooting tests or adding new ones, since the developer cannot selectively target a subset of tests without changing the config.

Proposal

Instead of executing tests via a generic call that is saucectl run, we'll navigate to a more framework specific run command: saucectl {framework} run.

Because run is now specific to a framework, this allows us to introduce framework specific flags on the CLI itself.

Currently, we have the notion of suites that represent a set of tests and their settings. Effectively, the user should be able to create such a suite on the fly via CLI args/flags alone.

Examples

Cypress

suites:
  - name: "Chrome"
    browser: "chrome"
    platformName: "Windows 10"
    config:
      testFiles: [ "A.js", "B.js" ]

The equivalent command for the above could look like this:

saucectl cypress run A.js B.js --name "Chrome" --browser "chrome" --platformName "Windows 10"

Espresso

espresso:
  app: calc.apk
  testApp: calc-success.apk
suites:
  - name: "saucy barista"
    testOptions:
      class:
        - com.example.android.testing.androidjunitrunnersample.CalculatorAddParameterizedTest
        - com.example.android.testing.androidjunitrunnersample.CalculatorInstrumentationTest
    devices:
      - name: "Android GoogleApi Emulator"
        orientation: portrait
        platformVersions:
          - "11.0"
          - "10.0"

The equivalent command for the above could look like this:

saucectl espresso run --app calc.apk --testApp calc-success.apk --name "saucy barista" --testOptions.class --device 'name=Android GoogleApi Emulator,platformVersions=11.0,10.0' --testOptions.class com.example.android.testing.androidjunitrunnersample.CalculatorAddParameterizedTest,com.example.android.testing.androidjunitrunnersample.CalculatorAddParameterizedTest

FAQ

Can I define suites in the config and supply settings via the CLI?

You can define suites in the config, but the moment you choose to provide suite-level settings via the CLI, only the CLI provisioned settings will be taken into account while the suites in the config are ignored.

Is the config still mandatory if I only supply my settings via the CLI?

Technically not, but still beneficial to have, even if you don't define suites in it. Many defaults can be set there, especially for settings that don't change very often or those that are applicable to many of the commands you intend to run (e.g. region, concurrency, npm settings etc.).

[vrunoa]

Looking at the espresso example I wonder if would people become confused about when to use a hierarchal separator vs key/vale formatting

In the example testOptions uses hierarchy, but device doesn't,

saucectl espresso run --app calc.apk --testApp calc-success.apk
--device 'name=Android GoogleApi Emulator,platformVersions=11.0,10.0' 
--testOptions.class com.example.android.testing.androidjunitrunnersample.CalculatorAddParameterizedTest

what if we keep key/value formatting for testOptions too?

saucectl espresso run --app calc.apk --testApp calc-success.apk
--device 'name=Android GoogleApi Emulator,platformVersions=11.0,10.0' 
--testOptions 'class=com.example.android.testing.androidjunitrunnersample.CalculatorAddParameterizedTest'

[alexplischke]

Aye, those are very good points and it's something we have to carefully evaluate framework by framework.

As for devices, I'd actually suggest coming up with device IDs (see our very own RDC, firebase, browserstack) so that you can do --device AndroidXZY_11 --device AndroidXYZ_10 etc. which is more CLI friendly.

[renaud-vmg]

The one most "glaring" omission is the ability to customise a given run's name - and I don't see any mention of the "job name" (sauce > name in the yml equivalent).

Other than that, I mostly agree with this proposal.

[alexplischke]

Oops! Thanks for catching that one 😄 Added --name to the espresso example 👍