Add zlib-ng

[rhpvorderman]

Add zlib-ng, including its threaded interface.

This PR is much less extensive than #124.

Benchmarks!

import argparse

import xopen

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("input")
    parser.add_argument("output")
    parser.add_argument("--threads", type=int, default=1)
    parser.add_argument("--level", type=int, default=5)
    args = parser.parse_args()

    with open(args.input, "rb") as fin:
        with xopen.xopen(args.output, mode="wb", compresslevel=args.level,
                         threads=args.threads) as fout:
            while True:
                block = fin.read(128 * 1024)
                if block == b"":
                    break
                fout.write(block)

before: (4 threads + level 5 is current cutadapt default)

Benchmark 1: python benchmark_xopen.py --level 4 --threads 4  ~/test/5millionreads_R1.fastq ramdisk/out.fastq.gz
  Time (mean ± σ):      6.963 s ±  0.019 s    [User: 27.845 s, System: 2.475 s]
  Range (min … max):    6.941 s …  6.994 s    5 runs
370M    ramdisk/out.fastq.gz

Benchmark 1: python benchmark_xopen.py --level 5 --threads 4  ~/test/5millionreads_R1.fastq ramdisk/out.fastq.gz
  Time (mean ± σ):     13.101 s ±  0.420 s    [User: 52.276 s, System: 2.460 s]
  Range (min … max):   12.754 s … 13.724 s    5 runs
357M    ramdisk/out.fastq.gz

Benchmark 1: python benchmark_xopen.py --level 6 --threads 4  ~/test/5millionreads_R1.fastq ramdisk/out.fastq.gz
  Time (mean ± σ):     31.231 s ±  0.303 s    [User: 124.932 s, System: 2.215 s]
  Range (min … max):   31.024 s … 31.764 s    5 runs
340M    ramdisk/out.fastq.gz

after:

Benchmark 1: python benchmark_xopen.py --level 4 --threads 4  ~/test/5millionreads_R1.fastq ramdisk/out.fastq.gz
  Time (mean ± σ):      6.254 s ±  0.028 s    [User: 24.670 s, System: 0.601 s]
  Range (min … max):    6.227 s …  6.290 s    5 runs
352M    ramdisk/out.fastq.gz

Benchmark 1: python benchmark_xopen.py --level 5 --threads 4  ~/test/5millionreads_R1.fastq ramdisk/out.fastq.gz
  Time (mean ± σ):      7.656 s ±  0.021 s    [User: 30.262 s, System: 0.666 s]
  Range (min … max):    7.630 s …  7.683 s    5 runs
343M    ramdisk/out.fastq.gz

Benchmark 1: python benchmark_xopen.py --level 6 --threads 4  ~/test/5millionreads_R1.fastq ramdisk/out.fastq.gz
  Time (mean ± σ):     12.377 s ±  0.143 s    [User: 49.126 s, System: 0.630 s]
  Range (min … max):   12.278 s … 12.627 s    5 runs
332M    ramdisk/out.fastq.gz

Looks like zlib-ng level 5 uses much less time than zlib level 5, so this is a great improvement for cutadapt. Going back to level 4 cuts down 20% of compute time while going back to zlib level 5 size. Not really worth it if filesize is an important consideration. Level 6 suffers immensely from diminishing returns and is not worth it for such big files IMO.

Python-isal handles levels 1 to 3. These have similar size results. Just added level 1 for comparison. This is not affected by this PR.

Benchmark 1: python benchmark_xopen.py --level 1 --threads 4  ~/test/5millionreads_R1.fastq ramdisk/out.fastq.gz
  Time (mean ± σ):      1.087 s ±  0.063 s    [User: 4.037 s, System: 0.601 s]
  Range (min … max):    0.996 s …  1.150 s    5 runs
387M    ramdisk/out.fastq.gz

Benchmark 1: python benchmark_xopen.py --level 1 --threads 1  ~/test/5millionreads_R1.fastq ramdisk/out.fastq.gz
  Time (mean ± σ):      2.758 s ±  0.055 s    [User: 2.756 s, System: 0.323 s]
  Range (min … max):    2.668 s …  2.812 s    5 runs

[rhpvorderman]

Hi @marcelm, back from the holidays yet? This will have a nice performance impact on cutadapt with default settings. I guess the PR e-mail will be at the bottom of a long list of other e-mails by now, so this is just a friendly ping to bump it back into view.

[marcelm]

Hi, indeed, the GitHub notifications have piled up quite a bit ...

This looks good! Can you please add a chagelog entry?

I was considering changing the default compression level in Cutadapt from 5 to 4. Do I understand correctly that with this change, I would get a smaller file and that it would be twice as fast? (At least for this dataset.)

[rhpvorderman]

This looks good! Can you please add a chagelog entry?

Done.

I was considering changing the default compression level in Cutadapt from 5 to 4. Do I understand correctly that with this change, I would get a smaller file and that it would be twice as fast? (At least for this dataset.)

Yes and yes :-D. Though I still recommend level 2, that is almost an order of magnitude faster ;-).

[marcelm]

Yes and yes :-D.

Awesome!

Though I still recommend level 2, that is almost an order of magnitude faster ;-).

I may come around eventually ...