Dynamic declaration of array size and accessing array job variables within script

[banilmohammed]

Hi all,
Im trying to write a process which runs a script that I want to submit as an array job to my cluster running SGE. Prior to using nextflow I have created a submission script that defines resources and includes the commands to run. Something akin to this:

#!/bin/bash
#$ -cwd
#$ -j y
#$ -pe shared 1
#$ -l h_rt=24:00:00,h_data=20G
# Notify when
#$ -m a
#  Job array indexes
#$ -t 1-1000:1


# Load the job environment:
. /u/local/Modules/default/init/modules.sh

module load anaconda3
conda activate env

Rscript foo.R -n ${SGE_TASK_ID}

Running this script with qsub submission.sh gets me what I want, however I have been porting this pipeline over to nextflow and struggling to get similar functionality. Currently I have my process set up as the following:

process RUN_TOOL {
        debug true

        publishDir 'results', mode: 'symlink'

        executor "sge"
        cpus 1
        memory "20GB"
        time "5h"
        array 10

        input:
                path(genotype_dir)

        output:
             path "$genotype_dir/output"

        script:
        """
        Rscript foo.R -n ${SGE_TASK_ID}
        """
}

This currently doesn't work as nextflow tells me there is no defined variable called SGE_TASK_ID.

I have 2 questions:

1. Is there a way to access the scheduler specific variable $SGE_TASK_ID?
2. Is there a way to dynamically declare array size?

In regards to 1. I looked through the following thread #1477 (comment), where there seems to be references to SGE_TASK_ID however I can't seem to access that variable. In regards to 2., I have tried to dynamically allocate array size with a closure like: array { array_size }, however get an error message saying that the array directive cannot be declared in dynamic manner with a closure. If this is currently supported, I welcome any ideas as a workaround.

Thank you all

[bentsherman]

You just need to escape the dollar sign in the script block: \${SGE_TASK_ID}. Otherwise Nextflow will treat it as a Nextflow variable (that's how string interpolation works in Nextflow).

As for dynamic array size, it is not allowed on a per-task basis because that wouldn't make any sense. But you could still set it using a param:

array params.array_size

The closure is needed only when you want to use task inputs.

[banilmohammed]

Ah i see now that i'm running univa grid engine, which is listed as supported in the docs?

[bentsherman]

Nextflow uses the same executor under the hood for UGE and SGE, so they should work the same way

Can you share the submit options that are generated in the .command.run?

[banilmohammed]

heres the .command.run:

#!/bin/bash
#$ -N nf-RUN_PIPELINE_RUN_TOOL_(1)
#$ -o /u/nmohamm/nextflow/work/d7/edc6df23c4717b9db26d1276511d0d/.command.log
#$ -j y
#$ -terse
#$ -notify

for this process:

process RUN_TOOL{
        debug true

        publishDir 'results', mode: 'symlink'

        executor "sge"
        array 5

        input:
                path(genotype_dir)

        script:
        """
        echo "testing"
        """
}

[bentsherman]

Which version of Nextflow are you using?

[banilmohammed]

24.10.4