Skip to content

Add scheduling.provisioningModel per-process hint for Google Batch#7343

Open
reece-maticebio wants to merge 1 commit into
nextflow-io:masterfrom
MaticeBio:feat/google-batch-spot-hint
Open

Add scheduling.provisioningModel per-process hint for Google Batch#7343
reece-maticebio wants to merge 1 commit into
nextflow-io:masterfrom
MaticeBio:feat/google-batch-spot-hint

Conversation

@reece-maticebio

@reece-maticebio reece-maticebio commented Jul 15, 2026

Copy link
Copy Markdown

Summary

closes #3530

Implements the scheduling.provisioningModel hint from the accepted hints-process-directive ADR
(20260323-hints-process-directive.md) for the Google Batch executor, resolving #3530. It lets a
pipeline choose spot / on-demand / preemptible provisioning per process (rather than only globally
via google.batch.spot), through the hints directive.

What it adds

  • A scheduling.provisioningModel hint (a String) on the Google Batch executor, mapping to the
    Google Batch ProvisioningModel enum:
    • spotSPOT
    • standard (alias: ondemand) → STANDARD
    • preemptiblePREEMPTIBLE
    • values are case-insensitive and trimmed.
  • Strict validation. Any other value throws IllegalArgumentException — no silent fallback to a
    default, so a typo fails fast instead of quietly running on the wrong provisioning model.
  • Unknown-key validation. An unrecognized google-batch/-prefixed hint key throws, per the ADR
    (unrecognized prefixed hints are errors), matching the Seqera executor. Bare and foreign-executor
    keys are left untouched.
  • Resolution. The hint resolves both the bare key (scheduling.provisioningModel) and the
    executor-prefixed key (google-batch/scheduling.provisioningModel); the prefixed form is prioritized. This
    mirrors the AWS Batch executor's existing hint resolution (get(prefix+key) ?: get(key)).
  • Fallback. When the hint is unset, provisioning is unchanged from today's global behaviour
    (google.batch.spot → spot, google.batch.preemptible → preemptible, else on-demand). STANDARD
    is applied by omitting the field so Batch's on-demand default takes effect. Pricing selection
    (PriceModel.spot) is derived from the resolved model so cost estimation stays consistent.

Tests

Unit tests added in nf-google:

  1. should resolve provisioning model from scheduling.provisioningModel hint covers
    • Bare SPOT/STANDARD/PREEMPTIBLE`
    • google-batch/-prefixed form, prefixed-wins precedence,
    • case-insensitivity and trimming, the ondemand alias, and the unset global-fallback rows
  2. should fail on invalid scheduling.provisioningModel hint value
    • a sweep proving strict value validation: non-enum strings, empty string, and non-String values all throw.
  3. should throw on unknown google-batch prefixed hint
    • an unrecognized google-batch/-prefixed key (a realistic case-typo and a wholly unknown key) throws, per the ADR's prefixed-error rule.
  4. should report all unknown google-batch hints together
    • two unknown prefixed keys produce a single exception naming both, proving errors are collected rather than thrown on the first.
  5. should ignore unknown bare and foreign-executor hints
    • a bare key and a foreign-executor (awsbatch/…) key are left untouched (no throw), since an executor only validates its own prefixed keys.
  6. should use instance template and ignore the provisioningModel hint
    • a template:// machine type with the hint set still submits successfully with the template honored (the hint is a no-op under an instance template, which replaces the instance spec wholesale).

Validation on real Google Batch

Exercised against a live Google Batch project:

  • spot → VM provisioned SPOT
  • standard and ondemandSTANDARD (on-demand)
  • preemptiblePREEMPTIBLE
  • no hint → falls back to the global google.batch.* setting
  • invalid value → run fails fast with a clear error (no silent default)

Design Questions

A few points came up while I was working on this:

  • Key name. The ADR catalogs this as scheduling.provisioningModel (what this PR implements), but
    the Seqera executor shipped machineRequirement.provisioning (via its
    SchemaMapperUtil.toProvisioningModel). I followed the ADR.
  • Value vocabulary. Google's enum and the ADR use standard; AWS and Seqera use ondemand
    (Seqera also has spotFirst). This PR accepts both standard and ondemand.
  • Unknown-hint handling differs from AWS. Unknown google-batch/* keys throw here (ADR line 114;
    matches Seqera). The AWS Batch executor currently only warns for unknown awsbatch/* keys, so the
    two Batch executors differ on this.
  • Shared resolver (candidate follow-up). hints now has three specific consumers: AWS Batch
    (awsbatch/consumableResources), this Google impl (same get(prefix+key) ?: get(key) +
    warnUnknownHints pattern), and Seqera's HintHelper, which already generalizes prefix/namespace
    resolution behind extractSeqeraHints(). AWS and Google duplicate what HintHelper does. I'd love to take
    this on as a separate PR is deemed worthwhile. I kept it out here to stay scoped to Google Batch spot enabled configuration for processes #3530.

Implements the Google Batch entry from the hints-process-directive ADR
(adr/20260323-hints-process-directive.md) / issue nextflow-io#3530 using the
generic `scheduling.` hint namespace agreed in the ADR, rather than a
narrow executor-specific key.

The `scheduling.provisioningModel` hint selects the Google Batch
ProvisioningModel for a single process, overriding the global
`google.batch.spot` / `google.batch.preemptible` config. The value is a
case-insensitive, trimmed string: `spot` -> SPOT, `standard` (or its
alias `ondemand`) -> STANDARD (on-demand), `preemptible` -> PREEMPTIBLE.
Any other value throws IllegalArgumentException (strict, no fail-open).
The executor-prefixed form `google-batch/scheduling.provisioningModel`
takes precedence over the bare form, mirroring how the AWS Batch
executor resolves its `consumableResources` hint; when unset, the global
config is used.

Hint-key validation and provisioning-model resolution are separated so
each has one clear responsibility:

- validateHints() runs exactly ONCE per task submission, at the top of
  newSubmitRequest() (before the instance-policy and machine-type work).
  It strictly validates only `google-batch/`-prefixed keys: any
  unrecognized prefixed hints throw IllegalArgumentException, per the ADR
  (unknown *prefixed* hints are errors) and matching the Seqera
  executor's HintHelper. All unknown prefixed keys are collected and
  reported together in a single exception (better debugging UX -- see
  them all at once) rather than throwing on the first one. Bare and
  foreign-executor keys are left untouched. (AWS Batch currently only
  warns -- a known divergence.) It returns early on null/empty hints.
- resolveProvisioningModel() is now a PURE resolver: it only reads the
  hint value and maps it to the effective ProvisioningModel (else global
  config, where spot wins over preemptible, default STANDARD). It no
  longer key-validates, so it never throws on unknown keys -- only on a
  bad *value*. It is called from both the instance-policy call site and
  machine-type selection without redundant re-validation. `standard` is
  Google's canonical enum term; `ondemand` (the AWS/Seqera term) is
  accepted as an alias resolving to the same STANDARD model.
- Instance-policy call site sets the resolved model; STANDARD is left
  unset so the Batch default (on-demand) applies.
- Machine-type/price selection prices as spot when the resolved model is
  SPOT or PREEMPTIBLE, standard otherwise.
- On the instance-template (`template://`) path, the whole instance
  policy is replaced by the template, so the `scheduling.provisioningModel`
  hint has no effect there -- warn when it is set, mirroring the existing
  per-setting `google.batch.spot` / `preemptible` template warnings. The
  warning is hardcoded for this hint (not auto-scanned over KNOWN_HINTS):
  whether a template overrides a hint is a per-hint decision, so a future
  job-level hint would be evaluated separately.
- Unit tests + docs (executor.md Google Batch hints). Value tests target
  resolveProvisioningModel(); key tests target validateHints() directly.

Signed-off-by: Maurice Buckner-Wolfson <mbucknerwolfson@animate.bio>
@reece-maticebio
reece-maticebio requested a review from a team as a code owner July 15, 2026 22:31
@netlify

netlify Bot commented Jul 15, 2026

Copy link
Copy Markdown

Deploy Preview for nextflow-docs ready!

Name Link
🔨 Latest commit 26d1aad
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs/deploys/6a580a3aac19d7000840d87a
😎 Deploy Preview https://deploy-preview-7343--nextflow-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@reece-maticebio
reece-maticebio marked this pull request as draft July 16, 2026 18:21
@reece-maticebio reece-maticebio changed the title Add scheduling.provisioningModel per-process hint for Google Batch #3530 Add scheduling.provisioningModel per-process hint for Google Batch Jul 16, 2026
@reece-maticebio
reece-maticebio marked this pull request as ready for review July 16, 2026 18:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Google Batch spot enabled configuration for processes

1 participant