Cloud Computing profile Troubleshooting

[uelandte]

Hi pgsc_calc team!

Thanks for making this great tool. I have been struggling to get the pgsc_calc test profile up and running on our cloud computing setup. We work on Terra, which spins up Google Cloud VMs. Unfortunately, we do not have sudo permissions when interacting with this terminal (see Using the terminal and interactive analysis shell in Terra – Terra Support)) and so I have not been able to get Docker or Singularity to play nicely when running pgsc_calc.

We are able to run a startup script which allows you to preconfigure an environment with packages that would otherwise require sudo permissions to install (Preconfigure a Cloud Environment with a startup script – Terra Support), but then once opening the terminal I am unable to run docker with “docker run hello-world”.

We also have the ability to start up the VM with docker images from DockerHub or Google Artifact Registry (https://support.terra.bio/hc/en-us/articles/360037143432-Docker-tutorial-Custom-cloud-environments-for-Jupyter-Notebooks)

Attempt 1: Docker

With this, I tried installing docker using the startup script, and then running it in the background. When trying to run docker run hello-world, the specific error message I get is below.

Startup script

#!/usr/bin/env bash

for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do apt-get remove $pkg; done
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh
groupadd docker
usermod -aG docker $USER
dockerd &
echo "done"

Error message

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1ec31eb5944: Extracting [==================================================>]  2.459kB/2.459kB
docker: failed to register layer: unshare: operation not permitted.

Docker info

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 26.1.3
 Storage Driver: vfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.155+
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.276GiB
 Name: eb92884e012d
Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

I am on Ubuntu 20.04.6 (jammy). I have not been able to get the rootless Docker to work either unfortunately (Run the Docker daemon as a non-root user (Rootless mode) | Docker Docs).

Attempt 2: Offline mode

I tried following the tutorial at https://pgsc-calc.readthedocs.io/en/latest/how-to/offline.html . Unfortunately, I get similar error messages about the unshare operation not being permitted

~/pgsc_calc$ cat images.txt | xargs -I {} sh -c 'docker pull --platform linux/amd64 "$1"' - {}
pgscatalog-utils-1.0.2: Pulling from pgscatalog/pygscatalog
728328ac3bde: Extracting [==================================================>]  31.43MB/31.43MB
1b1ca9b4dc3e: Download complete
4574495dc7bc: Download complete
340c3a8c76a8: Download complete
02b48cf7294a: Download complete
4c06d7919e1e: Download complete
89c9525ee0c2: Download complete
failed to register layer: unshare: operation not permitted

Attempt 3: Conda

When trying to run the pgsc_calc test profile using conda, I get an error about not being able to find a pgscatalog Module. When trying to install pgscatalog-core, it mentions that I need Python >=3.11. Unfortunately, the python version in Terra is 3.10.12

NXF_VER=24.04.1 nextflow run pgscatalog/pgsc_calc -profile test,conda

Command error:
  Traceback (most recent call last):
    File "/home/jupyter/.local/bin/pgscatalog-combine", line 5, in <module>
      from pgscatalog.core.cli.combine_cli import run
  ModuleNotFoundError: No module named 'pgscatalog'

pip install pgscatalog-core
ERROR: Ignored the following versions that require a different python version: 0.1.0 Requires-Python <4.0,>=3.11; 0.1.1 Requires-Python <4.0,>=3.11; 0.1.2 Requires-Python <4.0,>=3.11
ERROR: Could not find a version that satisfies the requirement pgscatalog-core (from versions: none)

After installing a newer python version using user version of miniconda as mentioned in #293 , I get the same error

Command error:  Traceback (most recent call last):
    File "/home/jupyter/.local/bin/combine_scorefiles", line 5, in <module>
      from pgscatalog_utils.scorefile.combine_scorefiles import combine_scorefiles
  ModuleNotFoundError: No module named 'pgscatalog_utils'

Questions

My questions for you are 1) have you been able to successful set up pgsc_calc on any cloud computing setup (Terra, All of Us, UK Biobank Research Analysis Platform) and 2) do you have any specific guidance for troubleshooting on my end?

Thanks so much for your time!
Thomas

[nebfield]

You're right, I don't think singularity or docker is a good fit for this kind of situation where you're loading software into a VM without admin rights. On HPCs the admins install singularity for users, and when you have admin rights you can install docker with sudo.

Conda is normally the best backup choice in this situation. I have two suggestions:

1. Try $ nextflow run pgscatalog/pgsc_calc -r v2.0.0-alpha.5 -profile test,conda. There were some problems with alpha 6 so we've marked it as a pre-release for now.

2. In some situations the conda environments don't resolve properly, so I combined all of our dependencies in one big environment.yaml file:

name: pgsc-calc-2-alpha-5
channels:
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - python=3.10
  # plink2
  - bioconda::plink2==2.00a3.3
  # database
  - zstd
  # report
  - r-jsonlite
  - r-dplyr
  - r-tidyr
  - r-purrr
  - r-ggplot2
  - r-DT
  - r-tibble
  - r-forcats
  - r-readr
  - quarto=1.4.550
  - pip
  - pip:
    - pgscatalog_utils==0.5.2
    - fraposa-pgsc==0.1.0
    - pyyaml==6.0

Then run:

$ conda env create -f environment.yaml
$ conda activate pgsc-calc-2-alpha-5
(pgsc-calc-2-alpha-5) $ nextflow run pgscatalog/pgsc_calc -r v2.0.0-alpha.5 -profile test # note: no conda profile!

This isn't the best way to work with the calculator (containers are best), but it should work OK to get alpha 5 working. I was able to run the test profile in an interactive session on a HPC using this approach.

For cloud execution we normally use Nextflow's support for batch executors like Cloud Batch instead of a dedicated VM. I think some users have worked with the UK Biobank's RAP before but I have no experience with it.

Good luck! Please let me know if you have any problems.

[uelandte]

Thank you for the thoughtful reply here!

Still doesn't like the pgscatalog_utils module after setting up using the environment.yaml file. Interestingly, if I import pgscatalog_utils manually, I am able to run combine_scorefiles on its own just fine, just not within the pipeline.

Config and error message shown below, and hoping to loop in our cloud team to see if they have any other ideas! Really appreciate your willingness to share this troubleshooting advice, and I'll follow up if we are able to resolve. If anyone else has been successful running this pipeline in an interactive VM, we'd love to hear how you did it!

-Thomas

Setup and Profile Run

conda env create -f environment.yaml
conda activate pgsc-calc-2-alpha-5
(pgsc-calc-2-alpha-5) jupyter@XXXXXX:~$ NXF_VER=24.04.1 nextflow run pgscatalog/pgsc_calc -r v2.0.0-alpha.5 -profile test

[17/3d3c4b] PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)              [  0%] 0 of 1 ✔
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELBIM               -
[4b/78bd23] PGS…CCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR (cineca chromosome 22) [100%] 1 of 1 ✔
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_VCF                      -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MATCH:MATCH_VARIANTS                            -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MATCH:MATCH_COMBINE                             -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:PLINK2_SCORE                        -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:SCORE_AGGREGATE                     -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:REPORT:SCORE_REPORT                             -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:DUMPSOFTWAREVERSIONS                            -
ERROR ~ Error executing process > 'PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)'

Caused by:
  Process `PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)` terminated with an error exit status (1)


Command executed:
  combine_scorefiles -s PGS001229_22.txt             -t GRCh37             -o scorefiles.txt.gz             -l log_scorefiles.
json             -v

  cat <<-END_VERSIONS > versions.yml
  COMBINE_SCOREFILES:
      pgscatalog_utils: $(echo $(python -c 'import pgscatalog_utils; print(pgscatalog_utils.__version__)'))
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  Traceback (most recent call last):
    File "/home/jupyter/.local/bin/combine_scorefiles", line 5, in <module>
      from pgscatalog_utils.scorefile.combine_scorefiles import combine_scorefiles
  ModuleNotFoundError: No module named 'pgscatalog_utils'

Work dir:
  /home/jupyter/work/17/3d3c4b949c44246e1002168f068e58 

Configuration

------------------------------------------------------
  pgscatalog/pgsc_calc v2.0.0-alpha.5-g8bdf287
------------------------------------------------------
Core Nextflow options
  revision                  : v2.0.0-alpha.5
  runName                   : awesome_khorana
  launchDir                 : /home/jupyter
  workDir                   : /home/jupyter/work
  projectDir                : /home/jupyter/.nextflow/assets/pgscatalog/pgsc_calc
  userName                  : jupyter
  profile                   : test
  configFiles               :

Input/output options
  input                     : /home/jupyter/.nextflow/assets/pgscatalog/pgsc_calc/assets/examples/samplesheet.csv
  scorefile                 : /home/jupyter/.nextflow/assets/pgscatalog/pgsc_calc/assets/examples/scorefiles/PGS001229_22.txt
  outdir                    : /home/jupyter/.nextflow/assets/pgscatalog/pgsc_calc/results

Reference options
  ref_samplesheet           : /home/jupyter/.nextflow/assets/pgscatalog/pgsc_calc/assets/ancestry/reference.csv
  ld_grch37                 : /home/jupyter/.nextflow/assets/pgscatalog/pgsc_calc/assets/ancestry/high-LD-regions-hg19-GRCh37.txt
  ld_grch38                 : /home/jupyter/.nextflow/assets/pgscatalog/pgsc_calc/assets/ancestry/high-LD-regions-hg38-GRCh38.txt
  ancestry_checksums        : /home/jupyter/.nextflow/assets/pgscatalog/pgsc_calc/assets/ancestry/checksums.txt

Compatibility options
  target_build              : GRCh37

Max job request options
  max_cpus                  : 2
  max_memory                : 6.GB
  max_time                  : 6.h

Other parameters
  config_profile_name       : Test profile
  config_profile_description: Minimal test dataset to check pipeline function

[nebfield]

Something strange is happening with your local $PATH. combine_scorefiles shouldn't be found in /home/jupyter/.local/bin/combine_scorefiles:

$ conda activate pgsc-calc-2-alpha-5
(pgsc-calc-2-alpha-5) $ which combine_scorefiles
/path/to/conda/envs/pgsc-calc-2-alpha-5/bin/combine_scorefiles

It should point to the conda environment, not your local bin folder.

Have you previously tried installing pgscatalog_utils manually with pip? Perhaps it's worth trying to reset your VM to a fresh state.

[uelandte]

Appreciate your thoughts here! Still having issues with step-by-step below. Our cloud team suggested downloading everything to /home/jupyter/.local/bin/ - would it be possible to run pgsc_calc outside of conda without specifying a -profile? I'll keep playing around, potentially trying with system installs from the startup scripts that Terra allows when first booting up a new VM.

Step 1: Install and initialize miniconda

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh

 WARNING:
     You currently have a PYTHONPATH environment variable set. This may cause
     unexpected behavior when running the Python interpreter in Miniconda3.
     For best results, please verify that your PYTHONPATH only points to
     directories of packages that are compatible with the Python interpreter
     in Miniconda3: /home/jupyter/miniconda3

Not sure whether the default PYTHONPATH environment variable when spinning up the VM is messing up install locations?

echo $PYTHONPATH

 /etc/jupyter/custom:/usr/lib/spark/python:/home/jupyter/packages

# initialize miniconda
~/miniconda3/bin/conda init bash

After this, I exit the terminal, shut down the terminal, then open a new terminal to initialize. From here, my terminal starts with (base) jupyter@XXXXXX:$ and then to (pgsc-calc-2-alpha-5) jupyter@XXXXXX:$ after I activate the environment.

Step 2: Create PGSC_calc env

# This uses environment .yaml from pgsc_calc team 
conda env create -f pgsc_calc_env.yaml
conda activate pgsc-calc-2-alpha-5

Step 3: Attempt PGSC Calc test profile run

# have to specify nextflow version otherwise will download older version that won't work
NXF_VER=24.04.1 nextflow run pgscatalog/pgsc_calc -profile test -r v2.0.0-alpha.5

Unfortunately, this returns the same error as before with with ModuleNotFoundError: No module named 'pgscatalog_utils'

Step 4: Find combine_scorefiles
Appears that combine_scorefiles is my home directory (/home/jupyter) in the .local folder. This is from a fresh VM, so no previous manual installs of pgscatalog_utils

which combine_scorefiles
 /home/jupyter/.local/bin/combine_scorefiles

With the pgsc-calc-2-alpha-5 environment activated, my $PATH still points to the miniconda environment bin.

echo $PATH
 /home/jupyter/miniconda3/envs/pgsc-calc-2-alpha-5/bin:/home/jupyter/miniconda3/condabin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jupyter/.local/bin:/home/jupyter/packages/bin

I tried uninstalling the pgscatalog_utils and then reinstall to the conda environment directory

pip uninstall pgscatalog_utils

 Found existing installation: pgscatalog-utils 0.5.2
 Uninstalling pgscatalog-utils-0.5.2:
   Would remove:
     /home/jupyter/.local/bin/aggregate_scores
     /home/jupyter/.local/bin/ancestry_analysis
     /home/jupyter/.local/bin/combine_matches
     /home/jupyter/.local/bin/combine_scorefiles
     /home/jupyter/.local/bin/download_scorefiles
     /home/jupyter/.local/bin/match_variants
     /home/jupyter/.local/bin/relabel_ids
     /home/jupyter/.local/bin/validate_scorefiles
     /home/jupyter/.local/lib/python3.10/site-packages/pgscatalog_utils-0.5.2.dist-info/*
     /home/jupyter/.local/lib/python3.10/site-packages/pgscatalog_utils/*
 Proceed (Y/n)? Y
   Successfully uninstalled pgscatalog-utils-0.5.2

pip install --target /home/jupyter/miniconda3/envs/pgsc-calc-2-alpha-5/bin pgscatalog_utils

 ERROR: Can not combine '--user' and '--target'

Apparently --user is specified under the hood?

Other notes

• If I try to reinstall pgscatalog_utils (going to /home/jupyter/.local/bin) and then open up Python from the terminal, I am able to import the combine_scorefiles module just fine without issues.

Again, apologies for all the trouble here, and will follow up if able to resolve!

-Thomas

[nebfield]

Yes, if you have downloaded every dependency on your system you don't need to use a -profile, nextflow will just call system utilities. Installing these dependencies manually is a pain, which is why I suggested making the big conda environment.

The test profile is useful to make sure each process in the workflow can run and you're able to generate results, before you try working with your target genomes.

Before installing and using miniconda3, perhaps try unset PYTHONPATH. I have a feeling it may be interfering with the conda environment.

[uelandte]

Got it thank you! No luck when trying to unset PYTHONPATH or point it in the direction of the miniconda environment. It still wants to throw pgscatalog_utils in /home/jupyter/.local/bin. I think I'll unfortunately probably end up having to try to download every dependency manually.

If we are able to find a workaround, I'll reply here so that others might be able to benefit too.

Really appreciate your responsiveness so far in helping us out!
-Thomas