Fix fwrite length for gzip output

[philippechataignon]

Closes #6356. Closes #5506.

This PR is an attempt to create a better gzip file with fwrite. Its an important rewrite because it includes some refactoring of actual code.

zlib

• use Z_SYNC_FLUSH instead of Z_FINISH
• create manual heading and write crc and len in tail
• calc len and crc in thread and summarize in main thread
• len in gzip specification is 32 bits and then is modulo 2 ^ 32 for uncompressed size > 4GiB

C code

• simplify the implementation with only a #pragma omp parallel for for chunk loop and #pragma omp ordered for the writing and summarizing part.
• Matt Dowle introduces the use of pool of buffers : the idea is generalized. The pools are created at the beginning and then uses for writing headers and rows. All the malloc occur early and no need for an header buffer.
• Deobfuscate some part, especially if ( ) is followed by a new line, no =- or =*. Lot of work remains. Use of indent command ?
• Remove some old debug code (msg)

[github-actions]

Generated via commit a11f8b5

Download link for the artifact containing the test results: ↓ atime-results.zip

Task	Duration
R setup and installing dependencies	4 minutes and 36 seconds
Installing different package versions	8 minutes and 11 seconds
Running and plotting the test cases	2 minutes and 19 seconds

[oliverfoster]

len in gzip specification is 32 bits and then is false for uncompressed size > 4GiB

Is it not meant to be length modulo 4GiB rather than false? The remainder of dividing by 4GiB multiplied by 4GiB?

[philippechataignon]

You're right and this PR version stores the modulo 2**32 as requested but its not the right size.
Note that in new versions of gzip (after 1.12), the length stored in file is no more used. See note on gzip -l on https://savannah.gnu.org/news/?id=10121

7z l # (use header)
    Date      Time    Attr         Size   Compressed  Name
                     .....   1298954309   2212092276  mtcars.csv

% gzip --version                                                                                                                                                    
gzip 1.12
% gzip -lv mtcars.csv.gz  # (decompress and takes much more time)
method  crc     date  time           compressed        uncompressed  ratio uncompressed_name
defla 5cd79282 Aug 24 10:19          2212092276          9888888901  77.6% mtcars.csv

[philippechataignon]

Put PR #5513 in this PR with new param compressLevel.

[philippechataignon]

Hi, all checks are now OK for coverage. Review can continue. Thanks @MichaelChirico for advise.

[MichaelChirico]

Thanks so much for the extra effort!!

[MichaelChirico]

Thanks again! I found a few of the new # nocov extraneous and managed to come up with some simple tests to get coverage to 100% -- PTAL at my tail commits if you're interested.

[tdhock]

restoring branch so that atime tests can find the commit 3630413

[MichaelChirico]

FYI @philippechataignon, we need to retain the branch for any PR involved in an {atime} performance test:

data.table/.ci/atime/tests.R

Line 247 in f6dca2d

PR = "3630413ae493a5a61b06c50e80d166924d2ef89a"), # Close-to-last merge commit in the PR.

I'm not super jazzed about this setup, but nothing better has emerged as yet.