Support streaming from cloud storage buckets

[SHuang-Broad]

Hi,

Given that mosdepth can be installed from source with independent htslib, and htslib supports streaming from cloud storage buckets, I'm wondering how difficult it would be, for mosdepth to support streaming from cloud storage directly.

E.g. something like the following (using Google bucket as example):

mosdepth -n --fast-mode --by 500 sample.wgs gs://path-to-my.bam

Thank you
Steve

[brentp]

Hi Steve,
yes, this is certainly possible. I tried to get a static build that would do this, but it was a bit fragile.
Instead, you can build mosdepth yourself with this as a guide for the nim stuff, then it will look for your libhts.so, so as long as you have built htslib with proper flags, then you'll be able to access, s3, gs, etc.

[brentp]

If you have a public gs url, I can test. I think I have this working.

[SHuang-Broad]

Thanks for the reply!

How about this public bam from Google itself?
gs://deepvariant/case-study-testdata/HG004.novaseq.pcr-free.35x.dedup.grch38_no_alt.chr20.bam
gs://deepvariant/case-study-testdata/HG004.novaseq.pcr-free.35x.dedup.grch38_no_alt.chr20.bam.bai

And there're bigger WGS files too in this public dataset, just unsure if it is requester-pays:
gs://deepvariant/case-study-testdata/

[brentp]

I am having trouble getting this into the static build. But, you can build htslib with --enable-gs and then build mosdepth yourself and get it to work (it works on my machine, just not with the static build).
I'll try a few more things, but I'd get the local build going. You can look at .github/workflows/build.yml in this repo (just add --enable-gcs) to the ./configure call , I don't use it in the github worfklow as it's not needed) to see how it's built.
You can ignore the d4 and rust stuff unless you need d4 support.

[brentp]

As it goes, right after posting that. I figured that if I do the static build without d4, then it works for gs:// urls. Here is that binary attached.
mosdepth.gz
you can gunzip, chmod +x and then use.