Skip to content

scaffold_and_refine_multitaxa usability improvements: try obtaining e-mail address via Terra env introspection; allow taxid_to_ref_accessions_tsv to originate from an http[s] URL #591

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 27 commits into
base: master
Choose a base branch
from
Open

scaffold_and_refine_multitaxa usability improvements: try obtaining e-mail address via Terra env introspection; allow taxid_to_ref_accessions_tsv to originate from an http[s] URL #591

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
22c569a
scaffold_and_refine_multitaxa: make e-mail address optional; determin…
tomkinsc Mar 21, 2025
d5d28dc
task `download_from_url`: only download https[s]; pass through non-ht…
tomkinsc Mar 25, 2025
b892b32
viral-core 2.4.1->2.4.2
tomkinsc Mar 25, 2025
f3e35e1
`scaffold_and_refine_multitaxa`workflow: use the `download_file` task…
tomkinsc Mar 25, 2025
7be105c
call terra.check_terra_env
tomkinsc Mar 25, 2025
482ceb4
move null placeholders to input block of download_from_url to satisfy…
tomkinsc Mar 25, 2025
1250ed4
WDL (1.0) doesn't allow leading underscores in variable names
tomkinsc Mar 25, 2025
75414c8
import "../tasks/tasks_terra.wdl" as terra
tomkinsc Mar 25, 2025
ae324e4
fix typo in path_url
tomkinsc Mar 25, 2025
4f5ff52
debugging download_from_url delocalization on Terra
tomkinsc Mar 25, 2025
ccb1f0c
debug
tomkinsc Mar 25, 2025
524e9e0
debug delocalization: try relative rather than absolute path
tomkinsc Mar 25, 2025
f84f0e4
debug download_from_url
tomkinsc Mar 25, 2025
e3b4b01
debug continued
tomkinsc Mar 25, 2025
369b0ba
debug
tomkinsc Mar 26, 2025
f346b01
debug
tomkinsc Mar 26, 2025
7dffa37
touch FILE_LOCATION at start of task
tomkinsc Mar 26, 2025
5c5103f
debug
tomkinsc Mar 26, 2025
db03042
debug
tomkinsc Mar 26, 2025
50b226c
debug
tomkinsc Mar 26, 2025
b5dc3b0
cruft removal
tomkinsc Mar 26, 2025
e3dc5e3
empty string coersion to optional File? does not seem to work on Terra
tomkinsc Mar 26, 2025
b6fa0c3
fix conditional
tomkinsc Mar 26, 2025
34fb514
Merge branch 'master' into ct-scaffold_multitaxa_usability_improvements
tomkinsc Mar 26, 2025
c6acdc7
include an optional File? in flatten() call to appease dxWDL
tomkinsc Mar 26, 2025
4234f03
add comment about WDL 1.1 and "None"
tomkinsc Mar 26, 2025
54bc38f
comment
tomkinsc Mar 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .dockstore.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,9 @@ workflows:
- name: diff_genome_sets
subclass: WDL
primaryDescriptorPath: /pipes/WDL/workflows/diff_genome_sets.wdl
- name: download_file
subclass: WDL
primaryDescriptorPath: /pipes/WDL/workflows/download_file.wdl
- name: downsample
subclass: WDL
primaryDescriptorPath: /pipes/WDL/workflows/downsample.wdl
Expand Down
4 changes: 2 additions & 2 deletions pipes/WDL/tasks/tasks_assembly.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -583,7 +583,7 @@ task align_reads {
Boolean skip_mark_dupes = false

Int? machine_mem_gb
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"

String sample_name = basename(basename(basename(reads_unmapped_bam, ".bam"), ".taxfilt"), ".clean")
}
Expand Down Expand Up @@ -849,7 +849,7 @@ task run_discordance {
String out_basename = "run"
Int min_coverage = 4

String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}
parameter_meta {
reads_aligned_bam: {
Expand Down
4 changes: 2 additions & 2 deletions pipes/WDL/tasks/tasks_demux.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ task merge_tarballs {
String out_filename

Int? machine_mem_gb
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}

Int disk_size = 2625
Expand Down Expand Up @@ -163,7 +163,7 @@ task illumina_demux {

Int? machine_mem_gb
Int disk_size = 2625
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}

parameter_meta {
Expand Down
2 changes: 1 addition & 1 deletion pipes/WDL/tasks/tasks_interhost.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -351,7 +351,7 @@ task index_ref {
File? novocraft_license

Int? machine_mem_gb
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}

Int disk_size = 100
Expand Down
10 changes: 5 additions & 5 deletions pipes/WDL/tasks/tasks_ncbi.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ task sequencing_platform_from_bam {
input {
File bam

String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}

command <<<
Expand Down Expand Up @@ -188,7 +188,7 @@ task structured_comments {

File? filter_to_ids

String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}
String out_base = basename(assembly_stats_tsv, '.txt')
command <<<
Expand Down Expand Up @@ -241,7 +241,7 @@ task structured_comments_from_aligned_bam {
String out_basename = basename(aligned_bam, '.bam')
Boolean is_genome_assembly = true
Boolean sanitize_ids = true
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}
# see https://www.ncbi.nlm.nih.gov/genbank/structuredcomment/
command <<<
Expand Down Expand Up @@ -360,7 +360,7 @@ task rename_fasta_header {

String out_basename = basename(genome_fasta, ".fasta")

String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}
command {
set -e
Expand Down Expand Up @@ -525,7 +525,7 @@ task sra_meta_prep {
Boolean paired

String out_name = "sra_metadata.tsv"
String docker="quay.io/broadinstitute/viral-core:2.4.1"
String docker="quay.io/broadinstitute/viral-core:2.4.2"
}
Int disk_size = 100
parameter_meta {
Expand Down
4 changes: 2 additions & 2 deletions pipes/WDL/tasks/tasks_nextstrain.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,7 +332,7 @@ task derived_cols {
String? lab_highlight_loc
Array[File] table_map = []

String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
Int disk_size = 50
}
parameter_meta {
Expand Down Expand Up @@ -900,7 +900,7 @@ task filter_sequences_to_list {

String out_fname = sub(sub(basename(sequences, ".zst"), ".vcf", ".filtered.vcf"), ".fasta$", ".filtered.fasta")
# Prior docker image: "nextstrain/base:build-20240318T173028Z"
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
Int disk_size = 750
}
parameter_meta {
Expand Down
14 changes: 7 additions & 7 deletions pipes/WDL/tasks/tasks_read_utils.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ task group_bams_by_sample {
task get_bam_samplename {
input {
File bam
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}
Int disk_size = round(size(bam, "GB")) + 50
command <<<
Expand All @@ -111,7 +111,7 @@ task get_sample_meta {
input {
Array[File] samplesheets_extended

String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}
Int disk_size = 50
command <<<
Expand Down Expand Up @@ -172,7 +172,7 @@ task merge_and_reheader_bams {
File? reheader_table
String out_basename = basename(in_bams[0], ".bam")

String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
Int disk_size = 750
Int machine_mem_gb = 4
}
Expand Down Expand Up @@ -244,7 +244,7 @@ task rmdup_ubam {
String method = "mvicuna"

Int machine_mem_gb = 7
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}

Int disk_size = 375 + 2 * ceil(size(reads_unmapped_bam, "GB"))
Expand Down Expand Up @@ -303,7 +303,7 @@ task downsample_bams {
Boolean deduplicateAfter = false

Int? machine_mem_gb
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}

Int disk_size = 750
Expand Down Expand Up @@ -370,7 +370,7 @@ task FastqToUBAM {
Int cpus = 2
Int mem_gb = 4
Int disk_size = 750
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}
parameter_meta {
fastq_1: { description: "Unaligned read1 file in fastq format", patterns: ["*.fastq", "*.fastq.gz", "*.fq", "*.fq.gz"] }
Expand Down Expand Up @@ -424,7 +424,7 @@ task read_depths {
File aligned_bam

String out_basename = basename(aligned_bam, '.bam')
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}
Int disk_size = 200
command <<<
Expand Down
12 changes: 6 additions & 6 deletions pipes/WDL/tasks/tasks_reports.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ task alignment_metrics {
Int max_amplicons=500

Int machine_mem_gb=32
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}

String out_basename = basename(aligned_bam, ".bam")
Expand Down Expand Up @@ -142,7 +142,7 @@ task plot_coverage {
String? plotXLimits # of the form "min max" (ints, space between)
String? plotYLimits # of the form "min max" (ints, space between)

String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}

Int disk_size = 375
Expand Down Expand Up @@ -289,7 +289,7 @@ task coverage_report {
Array[File] mapped_bam_idx = [] # optional.. speeds it up if you provide it, otherwise we auto-index
String out_report_name = "coverage_report.txt"

String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}

Int disk_size = 375
Expand Down Expand Up @@ -364,7 +364,7 @@ task fastqc {
input {
File reads_bam

String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}
parameter_meta {
reads_bam:{
Expand Down Expand Up @@ -412,7 +412,7 @@ task align_and_count {
Boolean keep_duplicates_when_filtering = false

Int? machine_mem_gb
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}

String reads_basename=basename(reads_bam, ".bam")
Expand Down Expand Up @@ -535,7 +535,7 @@ task align_and_count_summary {

String output_prefix = "count_summary"

String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}

Int disk_size = 100
Expand Down
2 changes: 1 addition & 1 deletion pipes/WDL/tasks/tasks_taxon_filter.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ task merge_one_per_sample {
Boolean rmdup = false

Int machine_mem_gb = 7
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}

Int disk_size = 750
Expand Down
4 changes: 2 additions & 2 deletions pipes/WDL/tasks/tasks_terra.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ task gcs_copy {

task check_terra_env {
input {
String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}
meta {
description: "task for inspection of backend to determine whether the task is running on Terra and/or GCP"
Expand Down Expand Up @@ -440,7 +440,7 @@ task create_or_update_sample_tables {
String sample_table_name = "sample"
String library_table_name = "library"

String docker = "quay.io/broadinstitute/viral-core:2.4.1"
String docker = "quay.io/broadinstitute/viral-core:2.4.2"
}

meta {
Expand Down
Loading
Loading