ColabFold process stuck in 'pending' state when using custom template and single sequence mode

[mavericb]

Expected Behavior

The ColabFold process should complete the analysis of the input sequence using the custom template and single sequence mode.

Current Behavior

The process gets stuck in a 'pending' state and doesn't progress beyond this point for over an hour.

Steps to Reproduce (for bugs)

1. Set up the environment with the following key configurations:
   • Custom template path: --custom-template-path ./templates
   • Single sequence mode: --msa-mode single_sequence
2. Run the following bash script:

#!/bin/bash
# Define the paths
DB_PATH='/home/lily/amelie/Workspace/ColabFold_old/MsaServer/databases'
INPUT_FASTA='/home/lily/amelie/Workspace/LigandMPNN/outputs/Cx43_Xenopus-laevis/RRNYRRNY-analogues/test_custom_templates_colabfold/seqs/Cx43_Xenopu_laevis_P16863-cropped_RRNYRRNY__id_3_relaxed_rank_001_alphafold2_multimer_v3_model_1_seed_043.fa'
OUTPUT_PATH="/home/lily/amelie/Workspace/ColabFold_old/work/outputs/Cx43-Xenopu_laevis/Cx43-RRNYRRNY-analogues-3_B"
LOG_PATH="/home/lily/amelie/Workspace/ColabFold_old/work/logs"
CACHE_PATH="/home/lily/amelie/Workspace/ColabFold_old/alphafold2_cache"
CUSTOM_TEMPLATE_PATH='/home/lily/amelie/Workspace/ColabFold_old/work/outputs/templates/1'

# Advanced settings
MSA_MODE="single_sequence"
PAIR_MODE="unpaired_paired"
PAIR_STRATEGY="greedy"
USE_TEMPLATES=True
NUM_RECYCLE=10
NUM_SEEDS=3
RANDOM_SEED=42
NUM_MODELS=1
MODEL_TYPE="auto"
USE_DROPOUT=False
MAX_SEQ=6000
NUM_RELAX=1
RELAX_MAX_ITERATIONS=200
USE_GPU_RELAX=True
RANK="multimer"

# GPU settings
NUM_GPUS=4

# Run Docker container
time docker run --gpus all \
  -v "${DB_PATH}:/database" \
  -v "${INPUT_FASTA}:/input.fasta" \
  -v "${OUTPUT_PATH}:/predictions" \
  -v "${CUSTOM_TEMPLATE_PATH}:/templates" \
  -v "${LOG_PATH}:/logs" \
  -v "${CACHE_PATH}:/cache" \
  ghcr.io/sokrypton/colabfold:1.5.5-cuda11.8.0 \
  /bin/bash -c "colabfold_batch \
    --msa-mode ${MSA_MODE} \
    --templates \
    --custom-template-path ./templates \
    --pair-mode ${PAIR_MODE} \
    --pair-strategy ${PAIR_STRATEGY} \
    --num-recycle ${NUM_RECYCLE} \
    --num-seeds ${NUM_SEEDS} \
    --random-seed ${RANDOM_SEED} \
    --num-models ${NUM_MODELS} \
    --model-type ${MODEL_TYPE} \
    $([[ $USE_DROPOUT == True ]] && echo '--use-dropout') \
    --max-seq ${MAX_SEQ} \
    --num-relax ${NUM_RELAX} \
    --relax-max-iterations ${RELAX_MAX_ITERATIONS} \
    $([[ $USE_GPU_RELAX == True ]] && echo '--use-gpu-relax') \
    --rank ${RANK} \
    /input.fasta /predictions && \
  echo 'Batch processing completed'"

ColabFold Output (for bugs)

Here's the output from running the bash script:

(base) lily@il-gpu04:~/amelie/Workspace/ColabFold_old/work$ bash predict_fast4.sh
SUBMIT:   0%|          | 0/300 [elapsed: 00:00 remaining: ?]2024-10-12 23:30:00,298 Running colabfold 1.5.5
2024-10-12 23:30:01,776 Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA Interpreter
2024-10-12 23:30:01,776 Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
2024-10-12 23:30:03,392 Running on GPU
2024-10-12 23:30:03,984 Found 6 citations for tools or databases
2024-10-12 23:30:04,344 Query 1/1: Cx43_Xenopu_laevis_P16863-cropped_RRNYRRNY__id_3_relaxed_rank_001_alphafold2_multimer_v3_model_1_seed_043__id_1__T_0.15__seed_42__overall_confidence_0.1588__ligand_confidence_1.0000__seq_rec_0.1250 (length 387)
PENDING:   0%|          | 0/300 [elapsed: 10:02 remaining: ?]

Additionally, here's the content of the log.txt file:

2024-10-12 22:20:14,280 Running colabfold 1.5.5
2024-10-12 22:20:15,403 Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA Interpreter
2024-10-12 22:20:15,403 Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
2024-10-12 22:20:17,293 Running on GPU
2024-10-12 22:20:17,967 Found 6 citations for tools or databases
2024-10-12 22:20:18,644 Query 1/1: Cx43_Xenopu_laevis_P16863-cropped_RRNYRRNY__id_3_relaxed_rank_001_alphafold2_multimer_v3_model_1_seed_043__id_1__T_0.15__seed_42__overall_confidence_0.1588__ligand_confidence_1.0000__seq_rec_0.1250 (length 387)
2024-10-12 22:20:19,523 Sleeping for 5s. Reason: PENDING
2024-10-12 22:20:25,395 Sleeping for 8s. Reason: PENDING
...

Context

We are trying to run ColabFold with a custom template and in single sequence mode. The process gets stuck in a 'pending' state and doesn't progress for over an hour. We're unsure if this is related to our specific configuration, an issue with ColabFold 1.5.5, or a problem with our input or environment.

Question

How can we determine why the process is stuck in a pending state, and what steps can we take to resolve this issue?