--ok-errors missing-old-version still errored out on a removed old version

[yarikoptic]

2025-07-27T12:49:23.86542539-04:00 ERROR process_item{url=s3://dandiarchive/dandisets/001077/draft/assets.jsonld?versionId=pSFPYBEPnGubf5gSdnVIIDdRiP.VKK2w}:download_item: s3invsync::syncer: Failed to download object error=failed downloading contents for s3://dandiarchive/dandisets/001077/draft/assets.jsonld?versionId=pSFPYBEPnGubf5gSdnVIIDdRiP.VKK2w

Caused by:
    0: streaming error
    1: error reading a body from connection: Connection reset by peer (os error 104)
    2: Connection reset by peer (os error 104)
2025-07-27T12:49:23.865979197-04:00 ERROR process_item{url=s3://dandiarchive/dandisets/001077/draft/assets.jsonld?versionId=UcSW6X3X58hWzGaumT3jPN2ZSH15bimQ}:download_item: s3invsync::syncer: Failed to download object error=failed downloading contents for s3://dandiarchive/dandisets/001077/draft/assets.jsonld?versionId=UcSW6X3X58hWzGaumT3jPN2ZSH15bimQ

Caused by:
    0: streaming error
    1: error reading a body from connection: Connection reset by peer (os error 104)
    2: Connection reset by peer (os error 104)
2025-07-27T12:49:23.890210115-04:00  INFO process_item{url=s3://dandiarchive/dandisets/001077/draft/assets.jsonld?versionId=.D_1PhGUGdNz7645Kq5rWuiEwNFektBg}: s3invsync::syncer: Backup path already exists; doing nothing path=/mnt/backup/dandi/dandiarchive-s3invsync-backups/manifests/dandisets/001077/draft/assets.jsonld.old..D_1PhGUGdNz7645Kq5rWuiEwNFektBg.959e04d175b7d15452ecdefdc748171a
2025-07-27T12:49:23.890241219-04:00  INFO process_item{url=s3://dandiarchive/dandisets/001077/draft/assets.jsonld?versionId=.D_1PhGUGdNz7645Kq5rWuiEwNFektBg}: s3invsync::syncer: Finished processing object
2025-07-27T12:49:23.89030822-04:00  INFO process_item{url=s3://dandiarchive/dandisets/001077/draft/assets.yaml?versionId=8.S03IMhNAKF.9Gt7rY.CwZ7sRTToNQr}: s3invsync::syncer: Processing object
2025-07-27T12:49:23.890349657-04:00  INFO process_item{url=s3://dandiarchive/dandisets/001077/draft/assets.yaml?versionId=8.S03IMhNAKF.9Gt7rY.CwZ7sRTToNQr}: s3invsync::syncer: Object is latest version of key
2025-07-27T12:49:23.890468988-04:00  INFO process_item{url=s3://dandiarchive/dandisets/001077/draft/assets.yaml?versionId=8.S03IMhNAKF.9Gt7rY.CwZ7sRTToNQr}: s3invsync::syncer: Backup path already exists but metadata does not match; renaming current file and downloading correct version path=/mnt/backup/dandi/dandiarchive-s3invsync-backups/manifests/dandisets/001077/draft/assets.yaml
2025-07-27T12:49:23.890790354-04:00 ERROR s3invsync::syncer: Error occurred error=failed downloading contents for s3://dandiarchive/dandisets/001077/draft/assets.jsonld?versionId=UcSW6X3X58hWzGaumT3jPN2ZSH15bimQ

Caused by:
    0: streaming error
    1: error reading a body from connection: Connection reset by peer (os error 104)
    2: Connection reset by peer (os error 104)
2025-07-27T12:49:23.890822577-04:00  INFO s3invsync::syncer: Shutting down in response to error
2025-07-27T12:49:23.893115821-04:00  INFO s3invsync::syncer: Process info version="0.1.0-alpha.2" git_commit="53d1542" manifest_date=2025-07-26T01-00Z elapsed=2962.344514801s physical_mem=39432192 virtual_mem=516001792
2025-07-27T12:49:23.893152711-04:00 ERROR s3invsync::syncer: Error occurred error=failed downloading contents for s3://dandiarchive/dandisets/001077/draft/assets.jsonld?versionId=pSFPYBEPnGubf5gSdnVIIDdRiP.VKK2w

Caused by:
    0: streaming error
    1: error reading a body from connection: Connection reset by peer (os error 104)
    2: Connection reset by peer (os error 104)
2025-07-27T12:49:23.893272176-04:00  INFO process_item{url=s3://dandiarchive/dandisets/001077/draft/assets.jsonld?versionId=LrYFy50Tx_gSfFy2kJG7Rg_7JHuCfrsQ}: s3invsync::syncer: Finished processing object
2025-07-27T12:49:23.893929166-04:00  INFO process_item{url=s3://dandiarchive/dandisets/001077/draft/assets.yaml?versionId=8.S03IMhNAKF.9Gt7rY.CwZ7sRTToNQr}: s3invsync::syncer: Finished processing object
Error: 2 ERRORS:
---
failed downloading contents for s3://dandiarchive/dandisets/001077/draft/assets.jsonld?versionId=UcSW6X3X58hWzGaumT3jPN2ZSH15bimQ

Caused by:
    0: streaming error
    1: error reading a body from connection: Connection reset by peer (os error 104)
    2: Connection reset by peer (os error 104)
---
failed downloading contents for s3://dandiarchive/dandisets/001077/draft/assets.jsonld?versionId=pSFPYBEPnGubf5gSdnVIIDdRiP.VKK2w

Caused by:
    0: streaming error
    1: error reading a body from connection: Connection reset by peer (os error 104)
    2: Connection reset by peer (os error 104)

which is indeed gone

dandi@drogon:/mnt/backup/tmp$ wget https://dandiarchive.s3.amazonaws.com/dandisets/001077/draft/assets.jsonld?versionId=pSFPYBEPnGubf5gSdnVIIDdRiP.VKK2w
--2025-07-28 08:46:24--  https://dandiarchive.s3.amazonaws.com/dandisets/001077/draft/assets.jsonld?versionId=pSFPYBEPnGubf5gSdnVIIDdRiP.VKK2w
Resolving dandiarchive.s3.amazonaws.com (dandiarchive.s3.amazonaws.com)... 3.5.129.13, 16.12.65.156, 16.12.65.52, ...
Connecting to dandiarchive.s3.amazonaws.com (dandiarchive.s3.amazonaws.com)|3.5.129.13|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2025-07-28 08:46:25 ERROR 404: Not Found.

here is current versions with mtimes ATM

aws s3api list-object-versions --bucket dandiarchive --prefix dandisets/001077/draft/assets.jsonld | jq -r '.Versions[] | [.VersionId, .LastModified] | @tsv'
923WUcSfBx.zlS2jFFUXOpQ2P6YLRW79        2025-07-28T09:59:39+00:00
OAlnNJRPBz34vdCzsGekD31CJJhFOwLv        2025-07-28T09:58:39+00:00
AAxUFayA5H2rrXRlSyVK7lDSHPxJwMdw        2025-07-28T09:57:39+00:00
1bZgQshW1whutvYKJastH4PI7vuizawV        2025-07-28T09:56:39+00:00
N7Qph2mM6d47wQwYg41XAEi8hdTIXrep        2025-07-28T09:55:39+00:00
XSt4R8YJmaN4w57rg.MY0sIA2Cq7Z6yS        2025-07-25T16:04:19+00:00
nQ.0fzKzUDNoCzt.UTPIYvwSahBU1c5G        2025-07-25T16:03:18+00:00