How to restart an interrupted step-by-step task

[Blank921757]

Hello, I am currently performing alignment using 13 amphibian genomes. I was running the job step by step on a SLURM cluster, but due to some issues, my server was interrupted. I now need to restart the task. How should I proceed? I understand that the --restart command can be used, but I'm not sure which commands it should be added to. The task was interrupted while in Round 0, and under steps/logs/, the following log files are being generated: blast-Anc05.log, blast-Anc06.log, blast-Anc09.log, blast-Anc10.log, blast-Anc11.log (still incomplete), while preprocess-Xenopus_tropicalis.log has already been fully generated. Below is my main command:

#!/usr/bin/env bash

generated by : /home/cactus/cactus_env/bin/cactus-prepare evolverfrog.txt --outDir steps --outHal anura_13species.hal --jobStore ./jobstore --alignCores 72 --alignMemory 340G --cactusOptions '--workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local' --script

date : 2026-01-31 14:01:22.891076

cactus commit : 52bf4f9

set -xeo pipefail

Preprocessor

pids=()
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-preprocess ./jobstore/0 evolverfrog.txt steps/evolverfrog.txt --inputNames Xenopus_tropicalis Atelopus_laetissimus Pseudophryne_corroboree Mixophyes_fleayi Balebreviceps_hillmani Leptopelis_ragazzii Bufotes_viridis Fejervarya_limnocharis Nanorana_parkeri Rana_temporaria Aquarana_catesbeiana Bufo_bufo Bufo_ailaoanus --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/preprocess-Xenopus_tropicalis.log &
pids+=($!)
for pid in ${pids[*]}; do wait $pid; done

Alignment

Round 0

pids=()
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-blast ./jobstore/1 steps/evolverfrog.txt steps/Anc05.paf.gz --root Anc05 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/blast-Anc05.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-align ./jobstore/2 steps/evolverfrog.txt steps/Anc05.paf.gz steps/Anc05.hal --root Anc05 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress --maxCores 72 --consCores 72 --maxMemory 316.6Gi 2>&1 | tee steps/logs/align-Anc05.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-hal2fasta ./jobstore/3 steps/Anc05.hal Anc05 steps/Anc05.fa.gz --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/hal2fasta-Anc05.log &
pids+=($!)

singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-blast ./jobstore/4 steps/evolverfrog.txt steps/Anc06.paf.gz --root Anc06 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/blast-Anc06.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-align ./jobstore/5 steps/evolverfrog.txt steps/Anc06.paf.gz steps/Anc06.hal --root Anc06 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress --maxCores 72 --consCores 72 --maxMemory 316.6Gi 2>&1 | tee steps/logs/align-Anc06.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-hal2fasta ./jobstore/6 steps/Anc06.hal Anc06 steps/Anc06.fa.gz --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/hal2fasta-Anc06.log &
pids+=($!)

singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-blast ./jobstore/7 steps/evolverfrog.txt steps/Anc09.paf.gz --root Anc09 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/blast-Anc09.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-align ./jobstore/8 steps/evolverfrog.txt steps/Anc09.paf.gz steps/Anc09.hal --root Anc09 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress --maxCores 72 --consCores 72 --maxMemory 316.6Gi 2>&1 | tee steps/logs/align-Anc09.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-hal2fasta ./jobstore/9 steps/Anc09.hal Anc09 steps/Anc09.fa.gz --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/hal2fasta-Anc09.log &
pids+=($!)

singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-blast ./jobstore/10 steps/evolverfrog.txt steps/Anc10.paf.gz --root Anc10 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/blast-Anc10.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-align ./jobstore/11 steps/evolverfrog.txt steps/Anc10.paf.gz steps/Anc10.hal --root Anc10 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress --maxCores 72 --consCores 72 --maxMemory 316.6Gi 2>&1 | tee steps/logs/align-Anc10.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-hal2fasta ./jobstore/12 steps/Anc10.hal Anc10 steps/Anc10.fa.gz --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/hal2fasta-Anc10.log &
pids+=($!)

singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-blast ./jobstore/13 steps/evolverfrog.txt steps/Anc11.paf.gz --root Anc11 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/blast-Anc11.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-align ./jobstore/14 steps/evolverfrog.txt steps/Anc11.paf.gz steps/Anc11.hal --root Anc11 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress --maxCores 72 --consCores 72 --maxMemory 316.6Gi 2>&1 | tee steps/logs/align-Anc11.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-hal2fasta ./jobstore/15 steps/Anc11.hal Anc11 steps/Anc11.fa.gz --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/hal2fasta-Anc11.log &
pids+=($!)
for pid in ${pids[*]}; do wait $pid; done

Round 1

pids=()
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-blast ./jobstore/16 steps/evolverfrog.txt steps/Anc07.paf.gz --root Anc07 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/blast-Anc07.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-align ./jobstore/17 steps/evolverfrog.txt steps/Anc07.paf.gz steps/Anc07.hal --root Anc07 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress --maxCores 72 --consCores 72 --maxMemory 316.6Gi 2>&1 | tee steps/logs/align-Anc07.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-hal2fasta ./jobstore/18 steps/Anc07.hal Anc07 steps/Anc07.fa.gz --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/hal2fasta-Anc07.log &
pids+=($!)

singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-blast ./jobstore/19 steps/evolverfrog.txt steps/Anc08.paf.gz --root Anc08 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/blast-Anc08.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-align ./jobstore/20 steps/evolverfrog.txt steps/Anc08.paf.gz steps/Anc08.hal --root Anc08 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress --maxCores 72 --consCores 72 --maxMemory 316.6Gi 2>&1 | tee steps/logs/align-Anc08.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-hal2fasta ./jobstore/21 steps/Anc08.hal Anc08 steps/Anc08.fa.gz --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/hal2fasta-Anc08.log &
pids+=($!)
for pid in ${pids[*]}; do wait $pid; done

Round 2

pids=()
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-blast ./jobstore/22 steps/evolverfrog.txt steps/Anc03.paf.gz --root Anc03 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/blast-Anc03.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-align ./jobstore/23 steps/evolverfrog.txt steps/Anc03.paf.gz steps/Anc03.hal --root Anc03 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress --maxCores 72 --consCores 72 --maxMemory 316.6Gi 2>&1 | tee steps/logs/align-Anc03.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-hal2fasta ./jobstore/24 steps/Anc03.hal Anc03 steps/Anc03.fa.gz --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/hal2fasta-Anc03.log &
pids+=($!)

singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-blast ./jobstore/25 steps/evolverfrog.txt steps/Anc04.paf.gz --root Anc04 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/blast-Anc04.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-align ./jobstore/26 steps/evolverfrog.txt steps/Anc04.paf.gz steps/Anc04.hal --root Anc04 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress --maxCores 72 --consCores 72 --maxMemory 316.6Gi 2>&1 | tee steps/logs/align-Anc04.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-hal2fasta ./jobstore/27 steps/Anc04.hal Anc04 steps/Anc04.fa.gz --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/hal2fasta-Anc04.log &
pids+=($!)
for pid in ${pids[*]}; do wait $pid; done

Round 3

pids=()
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-blast ./jobstore/28 steps/evolverfrog.txt steps/Anc02.paf.gz --root Anc02 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/blast-Anc02.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-align ./jobstore/29 steps/evolverfrog.txt steps/Anc02.paf.gz steps/Anc02.hal --root Anc02 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress --maxCores 72 --consCores 72 --maxMemory 316.6Gi 2>&1 | tee steps/logs/align-Anc02.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-hal2fasta ./jobstore/30 steps/Anc02.hal Anc02 steps/Anc02.fa.gz --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/hal2fasta-Anc02.log &
pids+=($!)
for pid in ${pids[*]}; do wait $pid; done

Round 4

pids=()
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-blast ./jobstore/31 steps/evolverfrog.txt steps/Anc01.paf.gz --root Anc01 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/blast-Anc01.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-align ./jobstore/32 steps/evolverfrog.txt steps/Anc01.paf.gz steps/Anc01.hal --root Anc01 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress --maxCores 72 --consCores 72 --maxMemory 316.6Gi 2>&1 | tee steps/logs/align-Anc01.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-hal2fasta ./jobstore/33 steps/Anc01.hal Anc01 steps/Anc01.fa.gz --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/hal2fasta-Anc01.log &
pids+=($!)
for pid in ${pids[*]}; do wait $pid; done

Round 5

pids=()
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-blast ./jobstore/34 steps/evolverfrog.txt steps/Anc00.paf.gz --root Anc00 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/blast-Anc00.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-align ./jobstore/35 steps/evolverfrog.txt steps/Anc00.paf.gz steps/Anc00.hal --root Anc00 --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress --maxCores 72 --consCores 72 --maxMemory 316.6Gi 2>&1 | tee steps/logs/align-Anc00.log &&
singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-hal2fasta ./jobstore/36 steps/Anc00.hal Anc00 steps/Anc00.fa.gz --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/hal2fasta-Anc00.log &
pids+=($!)
for pid in ${pids[*]}; do wait $pid; done

HAL merging

singularity exec /share/home/name/software/cactus_v3.1.3.sif cactus-halAppendSubtrees ./jobstore/37 steps/Anc00.hal steps/Anc01.hal steps/Anc02.hal steps/Anc03.hal steps/Anc04.hal steps/Anc07.hal steps/Anc08.hal steps/Anc05.hal steps/Anc06.hal steps/Anc09.hal steps/Anc10.hal steps/Anc11.hal anura_13species.hal --workDir /tmp/cactus_29995 --coordinationDir /tmp/cactus_29995 --maxMemory 340G --binariesMode local --disableProgress 2>&1 | tee steps/logs/halAppend-Anc01.log

Thank you for your response.

[glennhickey]

When this happens, I copy the the script to a new script, then erase all the completed tasks from the new script. Then rerun the new script. You can edit the new script to add --restart to the failed command, or delete its jobstore and run again from scratch.