Download of test data fails

[hendrikweisser]

Check Documentation

I have checked the following places for your error:

• nf-core website: troubleshooting
• nf-core/rnaseq pipeline documentation - This gives a 404 error!

Description of the bug

Running the test pipeline fails because data files (e.g. "genome.fa") aren't getting downloaded. However, manual download of the same files (using wget) works.

Steps to reproduce

Steps to reproduce the behaviour:

1. Command line: nextflow run nf-core-rnaseq-3.0/workflow/ -profile test,singularity
2. See error: No such file: https://github.com/nf-core/test-datasets/raw/rnaseq/reference/genome.fa

The error in the log is:
java.nio.file.NoSuchFileException: https://github.com/nf-core/test-datasets/raw/rnaseq/reference/genome.fa

Expected behaviour

Test files get downloaded and pipeline runs successfully.

Log files

Have you provided the following extra information/files:

• The command used to run the pipeline
• The .nextflow.log file - see https://gist.github.com/hendrikweisser/cd21694e12c15fe0b2e00647e8486552

System

• Hardware: Server
• Executor: local?
• OS: Ubuntu
• Version: 20.04.1

Nextflow Installation

• Version: 20.12.0-edge build 5460

Container engine

• Engine: Singularity
• version: 3.6.3
• Image tag: ???

Additional context

I had to download the pipeline using the helper tool, because downloading it via nextflow run nf-core/rnaseq also failed.

[hendrikweisser]

Would there be a way to pre-stage the test data, since we can download it manually?

[drpatelh]

Hi @hendrikweisser ! Apologies for the late response. Unfortunately, it isn't possible to pre-stage the test data because we just provide direct download links for simplicity. This is really odd though because all of our CI tests run with the same test data. Could GitHub have been down at the time? I have seen weird network issues in the past too. Sorry, not sure what to suggest here. Have you tried running the pipeline on your own data directly?

[hendrikweisser]

Thanks for your comments, @drpatelh.

Could GitHub have been down at the time?

No. The problems persisted over several days and as I said, we could download the data manually without a problem.

Have you tried running the pipeline on your own data directly?

We have now, and luckily this seems to work.

If we are the only people experiencing these download issues, I guess the problem must be on our side (although our IT people assured me everything was working fine, and I didn't encounter connectivity problems outside of Nextflow).
I thought it might be useful to log this here in case others encounter the same problem.

[drpatelh]

Thanks for the update @hendrikweisser 👍🏽 Let's peg this one down to weird, local issues and if you manage to find a solution please update do let us know :)

[ashotmarg]

In case this is still relevant, I was having the same/similar issue with the remote file not working (while the local one was fine), which turned out to be due to an SSL handshake issue in my Java certificate store.
I ended up using nextflow -Djavax.net.debug=ssl run myNextflowFile.nf ... to troubleshoot the error.
After adding my local certificate file to the Java store, things worked fine.