DAA merger?

[dportik]

Hi Benjamin,
I am an enthusiastic user of diamond, and have been using it in conjunction with MEGAN-LR for long-read metagenomics using PacBio HiFi data.

One key aspect of my pipeline design is reducing resource requirements and run times. My current strategy has been to split the HiFi reads file into four pieces and run each simultaneously, producing four output SAM files. Joining the results is easy with SAM format, and SAM can be converted to MEGAN-LR input formats. However, the frameshift characters are not compatible with MEGAN-LR when using sam2rma to convert the merged SAM. So, I set the frameshift penalty arbitrarily high to more or less turn off this option, but still take advantage of the range-culling feature for long reads.

Although eliminating frameshifts is not particularly problematic for HiFi reads due to their high accuracy (>99%), indels are the primary error profile so it would still be useful to allow frameshifts. If I can produce DAA outputs (instead of SAM) and merge them into a single DAA output, there is a clear path forward using daa-meganizer. This would also help speedup the conversion process for MEGAN-LR (sam2rma can take >24 hrs to run for some datasets), as daa-meganizer appears to be much faster. The only reason I designed the pipeline with SAM output is because merging DAA files was not possible, so I view it as a workaround rather than the optimal strategy.

Would it be possible to provide a DAA merging program? It would be extremely helpful in my case, and potentially for many others too.

Thanks,
Dan

[bbuchfink]

Yes, this has been on my todo list and I'll hopefully be able to get to it in the next weeks.

[dportik]

Just curious if there is any development on this topic? A DAA merging algorithm would still be exceptionally valuable in re-writing some of our larger workflows. Thanks!

[bbuchfink]

It is still on my todo list but I haven't had the capacity yet to implement it, sorry.

[blaizereal]

Dear Benjamin,

Will this planned merger module be able to handle results generated by long read and short read based algorithms? I think this hybrid solution would be pretty nice from the point of view of sample processing. Since instead of two (de novo assembled contigs + unassembled reads) daa files, we could see the taxonomic/functional profile in one output.

Thanks: Balázs

[bbuchfink]

Will this planned merger module be able to handle results generated by long read and short read based algorithms?

Yes that should be possible.

[dportik]

Just wanted to give this a nudge - still a feature that would be extremely useful!

[bbuchfink]

It's happening
dd49add

[dportik]

This is great news! This feature is going to result in some major speedups in a public workflow for shotgun metagenomics.

Will it make its way into a new release and on to bioconda soon?

Thanks again for taking the time to work on this, I really appreciate it.

[bbuchfink]

It's not yet finished but yes, that's the plan.

[bbuchfink]

You can access this feature now by compiling the latest master branch. The command line is diamond merge-daa --in ... -o out.daa with a list of files. I have not tested this well. Please test and let me know if everything works and if the performance is ok. Note that the sets of query sequences of the files are assumed to be disjoint.

[dportik]

Perfect. I'll set up some tests over the next week or two.