xopen.xopen(zst_file): results in OSError: b'/*stdin*\\ : Read error (39) : premature end \n' (exit code 1)

[gcflymoto]

Hi, with xopen/2.0.1 and zstandard/0.22.0 I see the following sometimes when opening a .zst file

with xopen.xopen(file_path) as fp:
  File "/usr/pkgs/python3/3.11.1/modules/r2/lib/python3.11/site-packages/xopen/__init__.py", line 352, in close
    self._raise_if_error(check_allowed_code_and_message, stderr_message)
  File "/usr/pkgs/python3/3.11.1/modules/r2/lib/python3.11/site-packages/xopen/__init__.py", line 413, in _raise_if_error
    raise OSError(f"{stderr_message!r} (exit code {retcode})")
OSError: b'/*stdin*\\ : Read error (39) : premature end \n' (exit code 1)

Do you know how I can decrypt this? I modified the script's invocation to call xopen and piped in /dev/null as the stdin, which seems to bypass it.

[marcelm]

Thanks! This can reliably be triggered with a compressed file that is larger than 128 KiB (131072 bytes).

As a workaround, you can add threads=0 to your xopen.xopen invocation for now.

[marcelm]

What we do in xopen is equivalent to something like this:

$ head -c 1000 largefile.txt.zst | zstd -dc -
/*stdin*\ : Read error (39) : premature end

That is, we stop sending input data to the zstd process, which it detects and complains about.

It appears this happens:

1. xopen.xopen() instantiates a _PipedCompressionProgram
   • which spawns an external zst process for decompression
   • to supply the zst process with data to decompress, a "feeder" thread is started that repeatedly reads 128 kiB chunks from the input file and sends them to the stdin of the process
2. No data or only some data is read by the program
3. _PipedCompressionProgram.close() is called before the end of the input file has been reached:
   • The feeder thread is told to stop doing its thing.
   • The feeder thread closes its output, that is, the stdin of the process.
   • This happens while zstd is in the middle of decoding compressed data
   • zstd complains as see above

[marcelm]

Fix merged in #161, so I think this can be closed now. Thanks again for reporting!