syncup总是发生MD5 hash比对不一致错误

[george-world]

Discussed in #710

^{Originally posted by george-world January 11, 2025}
各位前辈大拿

我用以下代码同步wordpress的备份文件，但是总是发生MD5比对错误，是因为上传的文件太大了吗？有什么解决方法？

python 代码片段

        # Perform the sync operation
        result = self.bypy.syncup(self.local_dir, self.remote_dir, deleteremote=True)

代码全文
baidupan_util.py.txt
baidupan_sync.py.txt

执行方式
pipenv run python3 baidupan_sync.py wp-content/updraft/ /updraft_tapioca_com_cn

系统环境

$ cat /etc/os-release
NAME="Alibaba Cloud Linux"
VERSION="3 (OpenAnolis Edition)"
ID="alinux"
ID_LIKE="rhel fedora centos anolis"
VERSION_ID="3"
VARIANT="OpenAnolis Edition"
VARIANT_ID="openanolis"
ALINUX_MINOR_ID="2104"
ALINUX_UPDATE_ID="10"
PLATFORM_ID="platform:al8"
PRETTY_NAME="Alibaba Cloud Linux 3.2104 U10 (OpenAnolis Edition)"
ANSI_COLOR="0;31"
HOME_URL="https://www.aliyun.com/"

$ pipenv run python3 --version
Python 3.12.0

错误对象上传文件

-rw-r--r-- 1 www www  29M Jan  9 18:27 backup_2025-01-09-1901_testtest_0b4b03972af9-themes.zip
-rw-r--r-- 1 www www  29M Jan  9 18:26 backup_2025-01-09-1901_testtest_0b4b03972af9-uploads.zip
-rw-r--r-- 1 www www  25M Jan  9 21:52 backup_2025-01-09-2152_wwwtesttestcomcn_825a20bccffb-themes.zip
-rw-r--r-- 1 www www  26M Jan  9 21:52 backup_2025-01-09-2152_wwwtesttestcomcn_825a20bccffb-uploads.zip
-rw-r--r-- 1 www www  25M Jan 10 15:40 backup_2025-01-10-1539_wwwtesttestcomcn_377ae4b6d3e2-themes.zip
-rw-r--r-- 1 www www  26M Jan 10 15:40 backup_2025-01-10-1539_wwwtesttestcomcn_377ae4b6d3e2-uploads.zip

log片段

Deletion request '247770017445518689' OK
Use 'list' command to confirm
Combining the following MD5 slices:
83c93365ac45e743b139196840b80f97
61c7812ebe793a174285c1168dc90f7f
<I> [16:23:36] Local and remote file hash DOESN'T match
<E> [16:23:36] '/www/wwwroot/testtest.com.cn/wp-content/updraft/backup_2025-01-09-1901_testtest_0b4b03972af9-uploads.zip' =C=> '/apps/bypy/updraft_testtest_com_cn/backup_2025-01-09-1901_testtest_0b4b03972af9-uploads.zip' FAILED.

log全文
20250111.log
另一次log全文
20250111_另一次.log

[george-world]

ByPy对象加了verify=False, verbose=True参数，上传后不比对MD5 hash了。
但是显示其他错误，并且重复上传远程已有的文件(已试验多次)。
self.bypy = ByPy(verify=False, verbose=True)

类代码全文
baidupan_util.py.txt

log片段

Loading Hash Cache File '/home/admin/.bypy/bypy.hashcache.json'...
Hash Cache File loaded.
Gathering local directory ...
Done
Gathering remote directory ...
Done
Comparing ...
Done
Deletion request '249765415727797151' OK
Use 'list' command to confirm
<E> [18:27:29] Error accessing 'https://pcs.baidu.com/rest/2.0/pcs/file'
<E> [18:27:29] Function: _rapidupload_file_act
<E> [18:27:29] Website parameters: {'method': 'rapidupload', 'path': '/apps/bypy/updraft_testtest_com_cn/backup_2025-01-09-1901_testtest_0b4b03972af9-themes.zip', 'content-length': 29917369, 'content-md5': '6735ae7a0k767eef6800b0bb7f9f5ab9', 'slice-md5': '77d2acaba01cdac3ac85eedb7f4673b6', 'content-crc32': '0xc3384fa7', 'ondup': 'overwrite'}
<E> [18:27:29] Full URL: https://pcs.baidu.com/rest/2.0/pcs/file?method=rapidupload&path=%2Fapps%2Fbypy%2Fupdraft_testtest_com_cn%2Fbackup_2025-01-09-1901_testtest_0b4b03972af9-themes.zip&content-length=29917369&content-md5=6735ae7a0k767eef6800b0bb7f9f5ab9&slice-md5=77d2acaba01cdac3ac85eedb7f4673b6&content-crc32=0xc3384fa7&ondup=overwrite&access_token=121.d5bbdff7ecfaac6d9aaf051a5e96a50d.Y_X4eBU9zSpsBsYGeZA5T_5Nhv3vJPtKqS4huB-.2osxcA
<E> [18:27:29] HTTP Response Status Code: 400
<E> [18:27:29] Error JSON returned:{'error_code': 31023, 'error_msg': 'param error', 'request_id': 249765628677262336}
Error code: 31023
Error Description: param error
<E> [18:27:29] Website returned: {"error_code":31023,"error_msg":"param error","request_id":249765628677262336}
'/www/wwwroot/testtest.com.cn/wp-content/updraft/backup_2025-01-09-1901_testtest_0b4b03972af9-themes.zip' >>==> '/apps/bypy/updraft_testtest_com_cn/backup_2025-01-09-1901_testtest_0b4b03972af9-themes.zip' OK.
[==============______] 70% (20.0MB/28.5MB) ETA: 18s (483kB/s, 42s gone) '/www/wwwroot/testtest.com.cn/wp-content/updraft/backup_2025-01-09-1901_testtest_0b4b03972af9-themes.zip' >>==> '/apps/bypy/updraft_testtest_com_cn/backup_2025-01-09-1901_testtest_0b4b03972af9-themes.zip' OK.
[====================] 100% (28.5MB/28.5MB) ETA:  (481kB/s, 1m gone) Combining the following MD5 slices:
2a89d466266def34fcfad5134db4d197
163333da62f35990f87263920d52487a
'/www/wwwroot/testtest.com.cn/wp-content/updraft/backup_2025-01-09-1901_testtest_0b4b03972af9-themes.zip' =C=> '/apps/bypy/updraft_testtest_com_cn/backup_2025-01-09-1901_testtest_0b4b03972af9-themes.zip' OK.

log全文
20250111_verify=False, verbose=True.log

[houtianze]

感谢反馈。

md5比较不成功是老问题了，百度服务器算的md5就是对不上，之前再上传一次好像就好了。。
看你第二个日志，感觉是百度改了秒传接口，具体参数我找不到文档，所以也不确定怎么改。
但是日志里有2025-01-11 18:35:03,463 - INFO - Successfully synced，应该是上传成功了，你用bypy compare --verify=False或者下载下来比较一下是不是本地和远端文件是一样的？

[george-world]

我比对了最新一次运行中同步到百度网盘的两个文件(各约26M大小)。

md5hash与源文件的相同。

百度盘文件: c4c399c15411e8e2759122dabc4c2764  backup_2025-01-13-1533_testtest_2abde194f88f-uploads.zip
源文件: c4c399c15411e8e2759122dabc4c2764  backup_2025-01-13-1533_testtest_2abde194f88f-uploads.zip

百度盘文件: b8f65b23bea376ca3d796432747f7e00  backup_2025-01-12-1532_testtest_7677b2bf33ae-uploads.zip
源文件: b8f65b23bea376ca3d796432747f7e00  backup_2025-01-12-1532_testtest_7677b2bf33ae-uploads.zip

log全文
20250114.log

[houtianze]

应该还是上传后第一次百度返回的MD5不对。后面你再传了一次？还是只是过了一些时间再比较了一下？

[george-world]

这个文件是第一次上传，比较时没有再次上传过。md5hash计算是先从Windows的百度盘软件里下载下来，在本地进行。

百度盘文件: c4c399c15411e8e2759122dabc4c2764  backup_2025-01-13-1533_testtest_2abde194f88f-uploads.zip
源文件: c4c399c15411e8e2759122dabc4c2764  backup_2025-01-13-1533_testtest_2abde194f88f-uploads.zip

[death4444]

我也发现这个问题，文件全被删了。哭死，重新传又要重跑进度

[houtianze]

改了一下syncup的比较，如果verify是False，跳过md5比较。