Merge branch 'master' into docs-troubleshooting

Signed-off-by: Chris Hakkaart <[email protected]>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,24 +1,23 @@
 ---
 name: Bug report
-about: Create a report to help us improve
+about: Report a bug to help us improve
 ---
 
 ## Bug report 
 
-(Please follow this template replacing the text between parentheses with the requested information)
+(Please follow this template by replacing the text between parentheses with the requested information)
 
 ### Expected behavior and actual behavior
 
-(Give an brief description of the expected behavior 
-and actual behavior)
+(Give a brief description of the expected behavior and actual behavior)
 
 ### Steps to reproduce the problem
 
-(Provide a test case that reproduce the problem either with a self-contained script or GitHub repository)
+(Provide a test case that reproduces the problem either with a self-contained script or GitHub repository)
 
 ### Program output 
 
-(Copy and paste here output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the `.nextflow.log` file.)
+(Copy and paste the output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the `.nextflow.log` file.)
 
 ### Environment 
 

.github/ISSUE_TEMPLATE/general_question.md

@@ -1,19 +1,8 @@
 ---
 name: General question 
-about: Need for help on Nextflow language and usage
+about: Ask for help with Nextflow language and usage
 ---
 
 Hi! Thanks for using Nextflow. 
 
-If you need help about Nextflow scripting language, 
-configuration options and general Nextflow usage the better 
-channels to post this kind of questions are: 
-
-* GitHub discussions: https://github.com/nextflow-io/nextflow/discussions
-* Slack community chat: https://www.nextflow.io/slack-invite.html
-
-
-Also you may also want to have a look at the patterns page 
-for common solutions to recurrent implementation problems: 
-http://nextflow-io.github.io/patterns/index.html
-
+If you need help using or developing Nextflow pipelines, the best place to ask questions is the [community forum](https://community.seqera.io/).

.github/ISSUE_TEMPLATE/new_feature.md

@@ -1,22 +1,16 @@
 ---
 name: New feature
-about: Submit a new feature proposal
+about: Propose a new feature or enhancement
 ---
 
 ## New feature
 
-Hi! Thanks for using Nextflow and submitting the proposal 
-for a new feature or the enhancement of an existing functionality. 
+(Hi! Thanks for using Nextflow and for proposing a new feature or enhancement. Please replace this text with a brief description of your proposal.)
 
-Please replace this text providing a short description of your 
-proposal.
+## Use case 
 
-## Usage scenario 
+(What's the main use case and deployment scenario addressed by this proposal)
 
-(What's the main usage case and the deployment scenario addressed by this proposal)
-
-## Suggest implementation 
+## Suggested implementation 
 
 (Highlight the main building blocks of a possible implementation and/or related components)
-
-

.github/PULL_REQUEST_TEMPLATE/pull_request_template.md

@@ -1,16 +1,7 @@
-Hi! Thanks for contributing to Nextflow project.
+Hi! Thanks for contributing to Nextflow.
 
-When submitting a Pull Request please make sure to not include
-in the changeset any modification in these files:
-
-* `nextflow`
-* `modules/nf-commons/src/main/nextflow/Const.groovy`
-
-Also, please sign-off the DCO [1] to certify you are the author of the contribution
-and you adhere to Nextflow open source license [2] adding a `Signed-off-by` line to
-the contribution commit message. For more details check [3].
+When submitting a Pull Request, please sign-off the DCO [1] to certify that you are the author of the contribution and you adhere to Nextflow's open source license [2] by adding a `Signed-off-by` line to the contribution commit message. See [3] for more details.
 
 1. https://developercertificate.org/
 2. https://github.com/nextflow-io/nextflow/blob/master/COPYING
 3. https://github.com/apps/dco
-

.github/workflows/build.yml

@@ -39,17 +39,18 @@ jobs:
         run: |
             if [ "${{ github.event_name }}" = "pull_request" ]; then
                 echo "GitHub event=pull_request"
-                COMMIT_SHA=${{ github.event.pull_request.head.sha }}
-                COMMIT_MESSAGE=$(curl -s \
+                COMMIT_SHA="${{ github.event.pull_request.head.sha }}"
+                COMMIT_MESSAGE="$(curl -s \
                   -H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
-                  https://api.github.com/repos/${{ github.repository }}/commits/$COMMIT_SHA | jq -r '.commit.message')
+                  https://api.github.com/repos/${{ github.repository }}/commits/$COMMIT_SHA | jq -r '.commit.message')"
                 echo "Commit message=$COMMIT_MESSAGE" | head -n 1
                 echo "commit_message=$COMMIT_MESSAGE" | head -n 1 >> $GITHUB_OUTPUT
             else
-               echo "GitHub event=${{ github.event_name }}"
-               echo "Commit message=${{ github.event.head_commit.message }}" | head -n 1
-               echo "commit_message=${{ github.event.head_commit.message }}" | head -n 1 >> $GITHUB_OUTPUT
-            fi 
+                echo "GitHub event=${{ github.event_name }}"
+                COMMIT_MESSAGE="${{ github.event.head_commit.message }}"
+                echo "Commit message=$COMMIT_MESSAGE" | head -n 1
+                echo "commit_message=$COMMIT_MESSAGE" | head -n 1 >> $GITHUB_OUTPUT
+            fi
 
       - name: Get changed files
         id: changed-files
@@ -132,7 +133,7 @@ jobs:
       fail-fast: false
       matrix:
         java_version: [17, 23]
-        test_mode: ["test_integration", "test_docs", "test_aws", "test_azure", "test_google", "test_wave"]
+        test_mode: ["test_integration", "test_parser_v2", "test_docs", "test_aws", "test_azure", "test_google", "test_wave"]
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -187,18 +188,18 @@ jobs:
 
       - name: Tar integration tests
         if: always()
-        run: tar -cvf integration-tests.tar tests/checks
+        run: | 
+            tar -cvf integration-tests.tar.gz tests/checks
+            tar -cvf validation-tests.tar.gz validation
 
       - name: Publish tests report
         uses: actions/upload-artifact@v4
         if: always()
         with:
           name: report-${{ matrix.test_mode }}-jdk-${{ matrix.java_version }}
           path: |
-            validation/**/*        # All regular files
-            validation/.[!.]*      # Top-level hidden files
-            validation/**/.[!.]*   # Hidden files in subdirectories
-            integration-tests.tar
+            validation-tests.tar.gz
+            integration-tests.tar.gz
 
   test-e2e:
       if: ${{ contains(needs.build.outputs.commit_message,'[e2e stage]') || contains(needs.build.outputs.commit_message,'[e2e prod]') }}

Makefile

@@ -27,6 +27,16 @@ else
 mm = 
 endif 
 
+compile:
+	./gradlew compile exportClasspath
+	@echo "DONE `date`"
+
+assemble:
+	./gradlew buildInfo compile assemble
+
+check:
+	./gradlew check
+
 clean:
 	rm -rf .nextflow*
 	rm -rf work 
@@ -38,16 +48,6 @@ clean:
 	rm -rf plugins/*/build
 	./gradlew clean
 
-compile:
-	./gradlew compile exportClasspath
-	@echo "DONE `date`"
-
-assemble:
-	./gradlew buildInfo compile assemble
-
-check:
-	./gradlew check
-
 #
 # install compiled artifacts in Maven local dir
 # 

VERSION

@@ -1 +1 @@
-24.12.0-edge
+25.01.0-edge

build.gradle

@@ -85,6 +85,10 @@ allprojects {
         maven { url 'https://oss.sonatype.org/content/repositories/snapshots' }
         maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/releases" }
         maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/snapshots" }
+        maven {
+            url 'https://jitpack.io'
+            content { includeGroup 'com.github.nextflow-io.language-server' }
+        }
     }
 
     configurations {
@@ -103,8 +107,8 @@ allprojects {
 
         // Documentation required libraries
         groovyDoc 'org.fusesource.jansi:jansi:2.4.0'
-        groovyDoc "org.apache.groovy:groovy-groovydoc:4.0.25"
-        groovyDoc "org.apache.groovy:groovy-ant:4.0.25"
+        groovyDoc "org.apache.groovy:groovy-groovydoc:4.0.26"
+        groovyDoc "org.apache.groovy:groovy-ant:4.0.26"
     }
 
     test {

changelog.txt

@@ -1,5 +1,40 @@
 NEXTFLOW CHANGE-LOG
 ===================
+25.01.0-edge - 12 Feb 2025
+- Add DeadlineExceededException to Google Batch retryable exceptions [944f48f9]
+- Add Fusion token validation (#5614) [1dcb18d6]
+- Add stub flag to task hash only if task has a stub (#5750) [deec443d]
+- Add traceparent header to wave & tower clients (#5725) [7eea9f2d]
+- Allow Azure Batch tasks to be submitted to different pools  (#5766) [76790d2a]
+- Configure target java version alongside gradle toolchain (#5676) [2153abb1]
+- Disable cleanup on remote work dir (#5742) [ef67b81e]
+- Do not skip integration tests on PRs [4caf6ef8]
+- Fail the run if publish thread pool times out (#5578) [5325e5a6]
+- Fix Check for number of low priority nodes in Azure Batch before raising a pool resize error (#5576) [9b528c11]
+- Fix Conda directive is not honoured by task array (#5704) [f8c3b63c]
+- Fix Create parent directory for DAG report (#5752) [62263a35]
+- Fix Google Batch task array causes process to fail (#5780) [7ad7a237]
+- Fix Mount input file buckets in task arrays for Google Batch (#5739) [ba171fd1]
+- Fix bugs with workflow outputs (#5502) [ab59d30f]
+- Fix first collection method [a0ca6e4f]
+- Fix stub templates (#5751) [13072d64]
+- Ignore tests when smoke mode is enabled [3eb6efad]
+- Improve Google Batch executor stability and error handling (#5690) [b64087fc]
+- Preserve attributes of published files (take 2) (#5660) [422f2898]
+- Remove compileJava options [28e03a21]
+- Remove redundant exception handling [00376aa3]
+- Revert "Fix the cleanup of local secret file" (#5697) [55f25b34]
+- Strengthen shell directive validation (#5689) [039c8f30]
+- Update HyperQueue CLI option (#5730) [BREAKING] [b582cb57]
+- Update azure deps [b163da95]
+- Use unbound queue for transfer thread pool (#5700) [90319681]
+- Bump Fusion to version 2.5 (2nd attempt) [a7f09ae4]
+- Bump gradle 8.12.1 [1fffb85c]
+- Bump groovy 4.0.25 [19c40a4a]
+- Bump io.netty:netty-handler:4.1.118.Final [db4a9037]
+- Bump net.minidev:json-smart:2.5.2 [b5c4faf4]
+- Bump netty-common:4.1.118.Final [8574e243]
+
 24.12.0-edge - 20 Jan 2025
 - Add Warning when Cloudinfo fails in Google Batch machine type selection (#5649) [5aef532f]
 - Add rate limiter to wave requests (#5608) [ecf68294]

docs/azure.md

@@ -184,6 +184,9 @@ process EXAMPLE_PROCESS {
 
 Note when creating tasks that use fewer than 4 CPUs, Nextflow will create a pool with machines that have 4 times the number of CPUs required in order to pack more tasks onto each machine. This means the pipeline spends less time waiting for machines to be created, startup and join the Azure Batch pool. Similarly, if a process requires fewer than 8 CPUs Nextflow will use a machine with double the number of CPUs required. If you wish to override this behaviour you can use a specific `machineType` directive, e.g. using a `machineType` directive of `Standard_E2d_v5` will use always use a Standard_E2d_v5 machine.
 
+:::{note}
+You can use the regular expressions to avoid certain machine types. For example, setting `machineType` to `standard_*[^p]_v*` will avoid machine types with names that contain the letter `p`, such as ARM-based machines.
+
 The pool is not removed when the pipeline terminates, unless the configuration setting `deletePoolsOnCompletion = true` is added in your Nextflow configuration file.
 
 Pool specific settings should be provided in the `auto` pool configuration scope. If you wish to specify a single machine size for all processes, you can specify a fixed `vmSize` for the `auto` pool.
@@ -266,6 +269,18 @@ azure {
 ```
 :::
 
+### Task packing on nodes
+
+Each node is given a number of task slots, which are the number of tasks that can be run concurrently on the node. The number of task slots is determined by the number of CPUs for the selected virtual machine. Nextflow will assign each process a number of task slots equal to a percentage of the total resources available on the node, based on the `cpus`, `memory`, and `disk` directives.
+
+For example, if using a `Standard_D4d_v5` machine with 4 vCPUs, 16 GB of memory, and a 150 GB local disk. If a process has the directives `cpus 2`, `memory 8.GB`, or `disk 75.GB`, it will be assigned one task slot and two tasks will concurrently run on the node. If the process has `cpus 4`, `memory 2.GB` or `disk 150.GB`, it will be assigned two task slots, and only one task will concurrently run on the node.
+
+A node may become overprovisioned if the tasks are using more than their fraction of total resources. For example, in the above example, if a process has the `cpus` directive set to `2`, it will be assigned one task slot and two tasks will concurrently run on the node. If the process uses more than 8 GB of memory or 75 GB of disk space, the node might become overprovisioned, and performance might degrade or the task will fail.
+
+:::{warning}
+The `cpus` directive is used to determine the number of task slots, not the number of cores.
+:::
+
 ### Requirements on pre-existing named pools
 
 When Nextflow is configured to use a pool already available in the Batch account, the target pool must satisfy the following requirements:
@@ -327,14 +342,14 @@ When Nextflow creates a pool of compute nodes, it selects:
 
 Together, these settings determine the Operating System and version installed on each node.
 
-By default, Nextflow creates pool nodes based on CentOS 8, but this behavior can be customised in the pool configuration. Below are configurations for image reference/SKU combinations to select two popular systems.
+By default, Nextflow creates pool nodes based on Ubuntu 22.04, but this behavior can be customised in the pool configuration. Below are configurations for image reference/SKU combinations to select two popular systems.
 
-- Ubuntu 20.04 (default):
+- Ubuntu 22.04 (default):
 
   ```groovy
-  azure.batch.pools.<name>.sku = "batch.node.ubuntu 20.04"
-  azure.batch.pools.<name>.offer = "ubuntu-server-container"
-  azure.batch.pools.<name>.publisher = "microsoft-azure-batch"
+  azure.batch.pools.<name>.sku = "batch.node.ubuntu 22.04"
+  azure.batch.pools.<name>.offer = "ubuntu-hpc"
+  azure.batch.pools.<name>.publisher = "microsoft-dsvm"
   ```
 
 - CentOS 8:

docs/cache-and-resume.md

@@ -22,10 +22,10 @@ The task hash is computed from the following metadata:
 - Task {ref}`environment modules <process-module>` (if applicable)
 - Task {ref}`Conda environment <process-conda>` (if applicable)
 - Task {ref}`Spack environment <process-spack>` and {ref}`CPU architecture <process-arch>` (if applicable)
-- Task {ref}`process-ext` directive (if applicable)
 - Task {ref}`inputs <process-input>`
 - Task {ref}`script <process-script>`
 - Any global variables referenced in the task script
+- Any task {ref}`process-ext` properties referenced in the task script
 - Any {ref}`bundled scripts <bundling-executables>` used in the task script
 - Whether the task is a {ref}`stub run <process-stub>`
 - Task attempt

docs/cli.md

@@ -223,16 +223,38 @@ These commands will execute two different project revisions based on the given G
 
 ### Pipeline parameters
 
-Pipeline scripts can use an arbitrary number of parameters that can be overridden using the command line or Nextflow configuration files. Any script parameter can be specified on the command line by prefixing the parameter name with double-dash characters. For example:
+Pipeline scripts can define *parameters* that can be overridden on the command line.
+
+Parameters can be declared in the main script:
+
+```nextflow
+params.alpha = 'default script value'
+
+workflow {
+  println "alpha = ${params.alpha}"
+}
+```
+
+Or in a config file:
+
+```groovy
+params {
+  alpha = 'default config value'
+}
+```
+
+The above parameter can be specified on the command line as `--alpha`:
 
 ```console
-$ nextflow run <pipeline> --foo Hello
+$ nextflow run main.nf --alpha Hello
 ```
 
-Then, the parameter can be accessed in the pipeline script using the `params.foo` identifier.
+:::{note}
+Parameters that are specified on the command line without a value are set to `true`.
+:::
 
 :::{note}
-When the parameter name is formatted using `camelCase`, a second parameter is created with the same value using `kebab-case`, and vice versa.
+Parameters that are specified on the command line in kebab case (e.g., `--foo-bar`) are automatically converted to camel case (e.g., `--fooBar`). Because of this, a parameter defined as `fooBar` in the pipeline script can be specified on the command line as `--fooBar` or `--foo-bar`.
 :::
 
 :::{warning}
@@ -245,14 +267,14 @@ $ nextflow run <pipeline> --files "*.fasta"
 
 Parameters specified on the command line can be also specified in a params file using the `-params-file` option.
 
-```bash
-nextflow run main.nf -params-file pipeline_params.yml
+```console
+$ nextflow run main.nf -params-file pipeline_params.yml
 ```
 
 The `-params-file` option loads parameters for your Nextflow pipeline from a JSON or YAML file. Parameters defined in the file are equivalent to specifying them directly on the command line. For example, instead of specifying parameters on the command line:
 
-```bash
-nextflow run main.nf --alpha 1 --beta foo
+```console
+$ nextflow run main.nf --alpha 1 --beta foo
 ```
 
 Parameters can be represented in YAML format:
@@ -271,7 +293,12 @@ Or in JSON format:
 }
 ```
 
-The parameters specified in a params file are merged with the resolved configuration. The values provided via a params file overwrite those of the same name in the Nextflow configuration file, but not those specified on the command line.
+Parameters are applied in the following order (from lowest to highest priority):
+
+1. Parameters defined in pipeline scripts (e.g. `main.nf`)
+2. Parameters defined in {ref}`config files <config-params>`
+6. Parameters specified in a params file (`-params-file`)
+7. Parameters specified on the command line (`--something value`)
 
 ## Managing projects