@aybeshlikyan, @yashpatel6 : I'm having trouble figuring out how to switch to PipeVal/validate. Any suggestions on how to modify my code?

Here is the error I'm currently getting:

WARN: Input tuple does not match input set cardinality declared by process align_DNA_BWA_MEM2_workflow:validate_output_file -- offending value: /scratch
WARN: Input tuple does not match input set cardinality declared by process align_DNA_BWA_MEM2_workflow:run_validate_PipeVal -- offending value: /hot/ref/tool-specific-input/BWA-MEM2-2.2.1/GRCh38-BI-20160721/index/genome.fa.bwt.2bit.64
Error executing process > 'align_DNA_BWA_MEM2_workflow:run_validate_PipeVal (6)'

Caused by:
Not a valid path value type: org.codehaus.groovy.runtime.NullObject (null)

Yash mentioned I might need to first process the input files using a line similar to here. I'm not sure what this line is doing, and I noticed align-RNA does not include a similar step, so I'm not sure if this line is needed?

@jarbet For the first warning/error, it seems like it's caused by lines 134-137 in "module/align_DNA_BWA_MEM2.nf" ; the function validate_output_file (AKA run_validate_PipeVal from the submodule) is expecting a channel with 2 items: the mode of the file to validate (in this case it would be 'file-bam') and the file to be validated (i.e. och_bam), but it's getting a channel with the items och_bam, och_bam_index, params.work_dir (i.e. "/scratch"), and params.output_dir. That's why it's complaining about the input set cardinality-- it's expecting 2 items but getting 4. The old run_validate_PipeVal in "validate.nf" took in the output log dir as an input, but the submodule handles that with the options passed into "addParams" with the include statement at the top of the file.

For the second warning/error, it's caused by lines 91-94 in the same file. It seems like it's complaining about the cardinality again, but also the index file (I think it's talking about ich_reference_index_files?); PipeVal can't handle that type of file.

The line that Yash linked in call-gSNP basically sets up a channel with the correct cardinality, mapping the unique BAM files from the input csv into items that are tuples of the form [<PipeVal file type>, <path to BAM file>] (i.e. ['file-input', it]).

I think setting up both validate_output_file and run_validate_PipeVal to take in channels with the correct number/type of items should fix it or at least get it closer to running. However I might not be understanding what's going on fully because I'm confused about how/why the index files are passed into run_validate_PipeVal, because I thought PipeVal couldn't run on those; maybe @yashpatel6 can make sure everything I said is accurate and give more insight if I misunderstood anything.

I noticed align-RNA does not include a similar step, so I'm not sure if this line is needed?

@jarbet Align-RNA doesn't use that exact way to set up the channel but it does something similar here, here, and here.

@jarbet Also a couple other things I noticed:

• the pipeval_version variable in "default.config" is set to 3.0.0 but the submodule is still the version of the repo that's using 2.1.6; you can run git submodule update --remote external/nextflow-modules/ from the root of the align-DNA repo to pull the latest version of the submodule
• You can change docker_image_validate_params = "blcdsdockerregistry/validate:2.1.6" to docker_image_validate_params = "blcdsdockerregistry/pipeval:3.0.0" in the default.config.

@jarbet For the first warning/error, it seems like it's caused by lines 134-137 in "module/align_DNA_BWA_MEM2.nf" ; the function validate_output_file (AKA run_validate_PipeVal from the submodule) is expecting a channel with 2 items: the mode of the file to validate (in this case it would be 'file-bam') and the file to be validated (i.e. och_bam), but it's getting a channel with the items och_bam, och_bam_index, params.work_dir (i.e. "/scratch"), and params.output_dir. That's why it's complaining about the input set cardinality-- it's expecting 2 items but getting 4. The old run_validate_PipeVal in "validate.nf" took in the output log dir as an input, but the submodule handles that with the options passed into "addParams" with the include statement at the top of the file.

For the second warning/error, it's caused by lines 91-94 in the same file. It seems like it's complaining about the cardinality again, but also the index file (I think it's talking about ich_reference_index_files?); PipeVal can't handle that type of file.

The line that Yash linked in call-gSNP basically sets up a channel with the correct cardinality, mapping the unique BAM files from the input csv into items that are tuples of the form [<PipeVal file type>, <path to BAM file>] (i.e. ['file-input', it]).

I think setting up both validate_output_file and run_validate_PipeVal to take in channels with the correct number/type of items should fix it or at least get it closer to running. However I might not be understanding what's going on fully because I'm confused about how/why the index files are passed into run_validate_PipeVal, because I thought PipeVal couldn't run on those; maybe @yashpatel6 can make sure everything I said is accurate and give more insight if I misunderstood anything.

Generally, you're correct! Some minor clarifications:

The process does expect each input to be a tuple with cardinality 2. So each emission in the input channel to the process must be a tuple of cardinality 2. The channel itself can contain as many emissions as you want. The cause for the error is that the input channel contains 4 emissions that each have a single item (cardinality 1). So the problem isn't that there are 4 items but that these 4 items have cardinality 1 when they need to have cardinality 2.

The second error/warning is actually caused because the input only had cardinality 1 so the second expected item (the actual path to the file for validating) is null and so you get that error.

PipeVal doesn't have explicit validation for index files but it is still able to verify that the file actually exists I believe. So passing ['file-input', <index_file>] should still work fine as a basic check that the file exists and if there's a checksum, I believe PipeVal will validate that (I could be wrong on the checksum part).

@aybeshlikyan , @yashpatel6 : thanks for the help so far. I understand the new run_validate_PipeVal requires 2 parameters, although Yash mentioned it can take a channel/set of many inputs, as long as each has the 2 required parameters. I don't know how to setup the inputs/channel to do this, but I'm reading more of the Nextflow documentation to try and figure it out.

Conceptually, I think I need something like the following:

For the input validation:

run_validate_PipeVal(
         // This needs to be a vector containing 'file-input' (or 'file-fasta'?), repeated X times, where X is the number of fasta files to validate
         [ file-input, file-input, ...., file-input]
         // I think this is a vector of paths to multiple fasta files? 
         ich_samples_validate.mix(.
         ich_reference_index_files
         )
         )

For the output validation:

validate_output_file(
         // This needs to be a vector containing 'file-input' (or 'file-bam'?), repeated X times, where X is the number of bam files to validate
         [ file-input, file-input, ...., file-input]
         // I think this is a vector of paths to multiple bam files?
         och_bam.mix(
            och_bam_index,
            Channel.from(params.work_dir, params.output_dir)
            )
         )

I don't know how to do this in groovy but here is a very rough guess based on here:

For the input validation:

ich_samples_validate.mix(.
         ich_reference_index_files
         ).flatMap{it -> ['file-input', it}.set{pipeval_input}
         // this assumes 'it' is every element of the ich_samples_validate.mix channel vector? 

For the output validation:

och_bam.mix(
            och_bam_index,
            Channel.from(params.work_dir, params.output_dir)
            ).flatMap{it -> ['file-input', it}.set{pipeval_output}

Then somehow I would have to send the pipeval_input and pipeval_output stuff to the corresponding calls to run_validate_PipeVal and validate_output_file ?

Generally, that's correct. You'll want to use map instead of flatMap though. flatMap is meant for doing a mapping operation on a channel and then flattening the result so if you use that, you'll end up with with each emission being a single element rather than a tuple of size 2. The reason there's a flatMap in call-gSNP is that the input channel contained values that didn't need to be validated so we map to keep only the things we want to validate -> flatten the result so each file to validate is a single emission -> map to ['file-input', it] to get it to the required input format. The flatMap operator handles the first two steps and the map handles the last step.

@yashpatel6 @aybeshlikyan : I think I fixed the input cardinality problem when validating the input files and output files, since I am no longer getting the cardinality warnings.

However, I am now getting the following error:
/hot/software/pipeline/pipeline-align-DNA/Nextflow/development/unreleased/jarbet-validate/BWA-MEM2-a_mini_n2.log

Error executing process > 'align_DNA_BWA_MEM2_workflow:run_validate_PipeVal (1)'

Caused by:
  Process `align_DNA_BWA_MEM2_workflow:run_validate_PipeVal (1)` terminated with an error exit status (1)

Command executed:

  set -euo pipefail
  validate -t file-input genome.fa > 'validation.txt'

Command exit status:
  1

Command output:
  (empty)

Work dir:
  /scratch/0a/231f887f90535270d8b01aa4749ea5

Although the cardinality is correct now, I think the contents of what is being passed to PipeVal seems wrong:

/hot/software/pipeline/pipeline-align-DNA/Nextflow/development/unreleased/jarbet-validate/align-DNA-8.1.0/a_mini_n2/BWA-MEM2-2.2.1/validation:

[[email protected] validation]$ cat input_validation.txt 
Input: HG002.N-n2_R2.fq.gz is valid file-fastq
Input: genome.fa.0123 is valid file-unknown
Input: genome.fa.fai is valid file-unknown
Input: genome.fa.bwt.2bit.64 is valid file-unknown
Input: genome.fa.ann is valid file-unknown

[[email protected] validation]$ cat output_validation.txt 
Input: scratch is valid file-unknown
Input: jarbet-validate is valid file-unknown

Any thoughts on how to fix this?
I'm guessing this line is no longer needed, but I don't understand the error from run_validate_PipeVal.

This seems to be because PipeVal got updated to throw an error for any file type that's not BAM, VCF, or FASTQ. Previous versions of PipeVal used to allow for other extensions and would check checksums if available and otherwise just pass the validation. @aybeshlikyan @tyamaguchi-ucla Any reason PipeVal was set up to only accept BAM, VCF, and FASTQ files in 3.0.0?

@aybeshlikyan @tyamaguchi-ucla Any reason PipeVal was set up to only accept BAM, VCF, and FASTQ files in 3.0.0?

I wasn't aware of the change. @aybeshlikyan thoughts?

Closing this PR as it's old and will be handled in a different PR