test_unar_tool test failure

[xiota]

Python 3.12

=================================== FAILURES ===================================
________________________________ test_unar_tool ________________________________

    @pytest.mark.skipif(sys.platform == "win32", reason="unar not available on Windows")
    @pytest.mark.skipif(not have_tool(rarfile.UNAR_TOOL), reason="unar not installed")
    def test_unar_tool():
        install_unar_tool()
        try:
>           test_read_rar3()

test/test_tool.py:93:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test/test_tool.py:56: in test_read_rar3
    rf.read(fn)
rarfile.py:838: in read
    return f.read()
rarfile.py:2310: in read
    self._check()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <rarfile.PipeReader object at 0x7f5d0024f520>

    def _check(self):
        """Check final CRC."""
        final = self._md_context.digest()
        exp = self._inf._md_expect
        if exp is None:
            return
        if final is None:
            return
        if self._returncode:
            check_returncode(self._returncode, "", tool_setup().get_errmap())
        if self._remain != 0:
            raise BadRarFile("Failed the read enough data")
        if final != exp:
>           raise BadRarFile("Corrupt file - CRC check failed: %s - exp=%r got=%r" % (
                self._inf.filename, exp, final))
E           rarfile.BadRarFile: Corrupt file - CRC check failed: stest1.txt - exp=3317163682 got=3067759956

rarfile.py:2326: BadRarFile
=========================== short test summary info ============================
FAILED test/test_tool.py::test_unar_tool - rarfile.BadRarFile: Corrupt file -...
=================== 1 failed, 150 passed, 6 skipped in 9.35s ===================

[markokr]

What is the version of unar you are running? And what OS are you on?

[xiota]

Arch Linux. unarchive 1.10.8

[CarloDePieri]

I can confirm this test failure.
I'm also running Arch, python 3.12.4 and the unar binary reports version v1.10.7 (although the unarchiver package is at 1.10.8-2).

People are also reporting this error on the aur page of python-rarfile (not that the aur package is to blame per se: I'm failing the test even by running pytest from this repo code directly).

[bgermann]

The version is known to be reported wrongly, see MacPaw/XADMaster#169

[jcfp]

I'm seeing similar CI failures in Debian:

• crc check failure in test_unar_tool;
• not enough data read failure in test_vols (and again via test_7zz_tool).

Those started showing up only recently, so I've been going through the CI logs (publicly available here) in an attempt to pinpoint what change or dependency triggered these failures but haven't managed to find the culprit.

Ubuntu was affected too and meanwhile added a patch that simply disables the failing tests in their 4.2-2ubuntu1 package revision.

[jcfp]

The patch made it to Debian testing.

@bgermann Unfortunately the crc error still happens with the unar debian package at revision 1.10.8+ds1-8, see this CI run. The failure looks identical to the one previously seen with revision 1.10.8+ds1-7 that did not include your updated patch. I'm still in the dark about what changed when the issues started showing up in the debian CI runs for rarfile (sometime late January - early February 2025).

[bgermann]

Actually, forget about the big endian CRC issue. Your test fails on amd64, so that cannot be the reason.

[bgermann]

Comparing the package versions between the last good and the first bad one, the only thing that comes to mind is the gnustep multiarch transition.

[bgermann]

Neither on Arch nor on Debian can I reproduce this manually. As the test runs fine in a non-autopkgtest environment, the reason has to be in the environment. The CRC is calculated by binascii.crc32 so the unar CRC code cannot be the reason.

My first guess by looking at the code would be USE_EXTRACT_HACK which writes to a temporary file. When that goes wrong and the error is not caught, the file might end up garbled and is then picked up to calculate the CRC.

[jcfp]

Put that first guess to the test but no luck... patching rarfile.py to have USE_EXTRACT_HACK = 0 (line ~126) results in the same crc-related test failures, plus two extra test failures with unrar-free only (in test/test_reading.py::test_reading_rar3_sfx and test/test_reading.py::test_reading_rar5_crc_sfx - both rarfile.BadRarFile). Full autopkgtest logs:

Result with USE_EXTRACT_HACK set to zero
Result without extra patch

The gnustep changes you mentioned look like a perfect match time-wise, but not exactly easy to test. I'm tempted to just skip the affected tests in the rarfile package, same as they did in ubuntu.

[bgermann]

The data that is CRCed is prepended by "dpkg-architecture: warning: cannot determine CC system type, falling back to default (native compilation)".

[bgermann]

So the Debian portion of this bug is actually a Debian-only bug. This does not explain the wrong CRC on Arch, which is indeed different from the one in Debian.

Hint: In the autopkgtest, you can work around the bug by export DEB_HOST_MULTIARCH=$(dpkg-architecture --query DEB_HOST_MULTIARCH) before calling pytest.