Intermediate file handling

[Faizal-Eeman]

Description

Enable save_intermediate_files in bcftools.nf and rtgtools.nf to publish intermediate files to intermediate dir when enabled.

PR also includes a few bug fixes from previous PR

• fix publish dir path in processes call_gSV_Delly and regenotype_gSV_Delly
• use params.variant_type instead of variant_type in main.nf

Closes #83

Testing Results

• Delly - gSV & gCNV
  • sample: CPCG0196-B1-downsampled-a-partial-sorted
  • input YAML: /hot/software/pipeline/pipeline-call-gSV/Nextflow/development/unreleased/mmootor-intermediate-file-handling/call-gSV-input-CPCG0196-B1-downsampled-a-partial-sorted.yaml
  • config: /hot/software/pipeline/pipeline-call-gSV/Nextflow/development/unreleased/mmootor-intermediate-file-handling/CPCG0196-B1-downsampled-a-partial-sorted.config
  • output: /hot/software/pipeline/pipeline-call-gSV/Nextflow/development/unreleased/mmootor-intermediate-file-handling/CPCG0196-B1-downsampled-a-partial-sorted/call-gSV-4.0.1/CPCG0196-B1/DELLY-1.1.3/output/
  • intermediate: /hot/software/pipeline/pipeline-call-gSV/Nextflow/development/unreleased/mmootor-intermediate-file-handling/CPCG0196-B1-downsampled-a-partial-sorted/call-gSV-4.0.1/CPCG0196-B1/DELLY-1.1.3/intermediate

Checklist

• I have read the code review guidelines and the code review best practice on GitHub check-list.

• I have reviewed the Nextflow pipeline standards.

• The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].

• I have set up or verified the branch protection rule following the github standards before opening this pull request.

• I have added my name to the contributors listings in the manifest block in the nextflow.config as part of this pull request, am listed
  already, or do not wish to be listed. (This acknowledgement is optional.)

• I have added the changes included in this pull request to the CHANGELOG.md under the next release version or unreleased, and updated the date.

• I have updated the version number in the metadata.yaml and manifest block of the nextflow.config file following semver, or the version number has already been updated. (Leave it unchecked if you are unsure about new version number and discuss it with the infrastructure team in this PR.)

• I have tested the pipeline on at least one A-mini sample with run_delly = true, run_manta = true, run_qc = true. For run_delly = true, I have tested 'variant_type' set to gSV, gCNV, and both. The paths to the test config files and output directories are captured above in the Testing Results section.

[yashpatel6]

Generally looks good though there are bugs being fixed here that should've been caught before. As a high priority, we'll want to add test cases to prevent issues like this. Anything else to add @sorelfitzgibbon @nwiltsie ?

[nwiltsie]

I think this looks good. @yashpatel6, agreed about the need for tooling to help catch these kinds of bugs.

[sorelfitzgibbon]

Nothing to add other than, I assume it would be rare for users to keep the intermediate files and so no need to compress the intermediate vcfs?

[Faizal-Eeman]

Nothing to add other than, I assume it would be rare for users to keep the intermediate files and so no need to compress the intermediate vcfs?

That's right, it would be a rare case and the intent of saving intermediate VCFs would only be temporary so for now I think uncompressed is fine. Besides sizes are substantially small.