v2.3.0 PR

[BioWilko]

Reduces runtime and intermediate storage for large inputs, and fixes several taxonomy-parsing bugs surfaced by recent taxonomy versions.

Single-pass read extraction

Previously each output group meant its own process and its own full decompress and parse of the FASTQ. Now:

• extract_taxa: one process per sample covering all kreport splits, driven by a --report_config JSON (was one process per split).
• extract_fractions: 6 processes collapsed to 1. virus_and_unclassified, virus and human_filtered are extracted in one pass via --fraction_config.
• KrakenAssignments.get_read_maps() builds N independent read maps in a single pass over the assignment file.
• extract_utils: new TaxonWriter (LRU-capped filehandles plus buffered writes) bounds memory and fd use; per-taxon running sums replace per-read quality/length lists; faster mean_phred.

HCID

• check_hcid.py now takes read sequences from the SAM rather than re-decompressing the FASTQ, reverse-complementing flag-16 records to restore original orientation. The FASTQ is no longer staged into the process at all.
• minimap2 replaced with rammap (rammap_hcid), with a new dockerfiles/rammap/Dockerfile pulling the pinned v1.1.2 release binary. Verified byte-identical SAM output vs minimap2 2.26.

Compression and I/O

• bgzip/pigz replaced with crabz throughout (faster, lower CPU, better ratio). fastp now streams via FIFO/stdout straight into crabz instead of writing an uncompressed temp FASTQ.
• Deleted get_total_length.py and its two processes, an entire extra FASTQ pass. Total length now comes from the fastcat per-read stats and from the extraction pass itself.
• check_reads.py: bounded-window duplicate/interleave detection with early exit, plus a long-read fast path.
• concatenate_reads.py: batched writes; fixed gzip.open(..., mode="w") to "wt".

Taxonomy and report correctness

• report.py: domain detection by name (modern taxonomy uses R1/R2 rather than D); new simple_rank so numbered ranks (G1, D2, etc.) match correctly, previously these were missed in per-taxa read files.
• split_kraken_report.py: ancestor backfill ordering fix.
• taxonomy.py: children as list rather than set; load_entries_from_nodes now honours its taxon_ids filter instead of loading all ~2.6M nodes.

"Select domain" dropdown showed random mix of taxa for modern taxonomy

NCBI retired superkingdom in 2024: Bacteria/Archaea/Eukaryota are now rank domain, Viruses is acellular root, with realm clades beneath it. aggregate_lineages_bracken.py's hardcoded RANKS allowlist silently dropped any node whose rank wasn't listed, so the real domain nodes vanished and the next allowed rank — kingdom — became the top level of the lineage tree. The dropdown is populated from that tree's top-level keys, so it offered kingdoms and clades: on the test fixture, Pseudomonadati, Opisthokonta (a clade), Heunggongvirae. Now Bacteria, Eukaryota, Viruses, Unclassified.

• RANKS gains acellular root, domain and realm, and keeps superkingdom so pre-2024 dumps still resolve. cellular root is deliberately excluded — including it would collapse all three cellular domains under a single meaningless node.
• This also recovers counts that were being dropped outright: a read assigned directly to taxid 2 has a lineage ending at domain, so it previously contributed to no top-level node at all. Sankey percentages will shift slightly, and the top-level keys in *.kraken.json are renamed.
• New bin/ranks.py holds the vocabulary once (RANKS, DOMAIN_RANKS, UNCLASSIFIED_RANK, KNOWN_DOMAIN_NAMES), imported by aggregate_lineages_bracken.py and report.py and injected into the report's JS by make_report.py. Three copies previously had to agree by hand, and the JS copy drives both the #rank-select options and the sankey depth filter — where an unrecognised rank yields indexOf() == -1 and passes every filter, so drift showed up as nodes ignoring the rank selector.
• sankey.js: node domain_name was derived from val.rank == "superkingdom", which after the rename matched only the synthetic Unclassified node — leaving every tooltip's Domain: field null. Now tested against domainRanks.
• aggregate_lineages_bracken.py: the Unclassified node's rank was RANKS[0], incidentally superkingdom; now an explicit UNCLASSIFIED_RANK.
• aggregate_lineages_bracken.py: previous_rank was only assigned on the "node already exists" branch, so the species → subspecies fallback for off-allowlist ranks only fired when the parent species node happened to already be in the tree.
• report.py: get_domains() re-derived domains from simple_rank == "D", which the module's own comment explains is unreliable on modern dumps; it now returns the self.domains map already built at load time. Also dropped the stray entry.print() side effect from it and from get_tips()/get_rank_entries().

Containers and config

• Main container moved to a Wave build with pinned fastcat/fastp/jq plus crabz; get_versions reads conda-meta via jq (Wave images ship no package manager).
• New publish_dir_mode param (default copy) replacing hardcoded mode: 'copy'.
• Removed docker-build-push.yml; version bumped from v2.2.1 to v2.3.0.

Testing

• Full nf-test suite passes (12/12, including e2e).
• Extraction and HCID outputs were additionally diffed byte-for-byte against the pre-change baseline on real fixture data.