Test of PR #117

(Compare against earlier run for the same bucket)

This test was done by running s3invsync twice; the first run targeted the manifest from 2025-01-08, while the second run (against the backup directory created by the first run) targeted the manifest from 2025-01-01, thereby creating a situation where the backup contained some files that did not exist in the inventory and had to be deleted by the second run.

Commit: cd6507c (PR Delete files that don't match inventory items #117)
Host: typhon (Debian bookworm, 32 cores)
Inventory base: s3://dandiarchive-inventory/dandiarchive/dandiset-manifest/ (inventory for just the dandisets/ directory of the dandiarchive bucket; requires credentials)

Run 1:

Manifest: 2025-01-08T01-00Z
Path filter: none (490452 files, 1371 GB)
Start time: 2025-01-10 16:32:06 EDT
First (non-CSV) file download started: 2025-01-10 16:32:09 EDT
Last (non-CSV) file download finished: 2025-01-10 18:34:16 EDT
End time: 2025-01-10 18:34:16 EDT
Runtime: 2 hours, 2 minutes, 9.833 seconds
Maximum memory usage: 148 MB
Maximum CPU usage: 406%

info.json file produced by duct

{
  "command": "target/release/s3invsync -l TRACE --date 2025-01-08T01-00Z s3://dandiarchive-inventory/dandiarchive/dandiset-manifest/ /mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/backup",
  "system": {
    "cpu_total": 32,
    "memory_total": 1081801363456,
    "hostname": "typhon",
    "uid": 1012,
    "user": "jwodder"
  },
  "env": {},
  "gpu": [
    {
      "index": "0",
      "name": "NVIDIA A100-PCIE-40GB",
      "bus_id": "00000000:31:00.0",
      "driver_version": "560.35.05",
      "memory.total": "40960 MiB",
      "compute_mode": "Default"
    }
  ],
  "duct_version": "0.8.0",
  "schema_version": "0.2.0",
  "execution_summary": {
    "exit_code": 0,
    "command": "target/release/s3invsync -l TRACE --date 2025-01-08T01-00Z s3://dandiarchive-inventory/dandiarchive/dandiset-manifest/ /mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/backup",
    "logs_prefix": "/mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/01-",
    "wall_clock_time": 7329.833003282547,
    "peak_rss": 155242496,
    "average_rss": 123771345.11565346,
    "peak_vsz": 3009150976,
    "average_vsz": 3006609741.2086253,
    "peak_pmem": 0.0,
    "average_pmem": 0.0,
    "peak_pcpu": 406.0,
    "average_pcpu": 341.9033323998893,
    "num_samples": 7142,
    "num_reports": 124,
    "start_time": 1736544726.7271266,
    "end_time": 1736552056.56013
  },
  "output_paths": {
    "stdout": "/mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/01-stdout",
    "stderr": "/mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/01-stderr",
    "usage": "/mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/01-usage.json",
    "info": "/mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/01-info.json",
    "prefix": "/mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/01-"
  }
}

Run 2:

Manifest: 2025-01-01T01-00Z
Path filter: none (490182 files, 1371 GB)
Files downloaded: 45
Deleted: 7 directories, 240 files
Start time: 2025-01-10 18:34:16 EDT
First (non-CSV) file download started: 2025-01-10 18:34:57 EDT
Last (non-CSV) file download finished: 2025-01-10 18:34:58 EDT
End time: 2025-01-10 18:34:58 EDT
Runtime: 42 seconds
Maximum memory usage: 204 MB
Maximum CPU usage: 227%

info.json file produced by duct

{
  "command": "target/release/s3invsync -l TRACE --date 2025-01-01T01-00Z s3://dandiarchive-inventory/dandiarchive/dandiset-manifest/ /mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/backup",
  "system": {
    "cpu_total": 32,
    "memory_total": 1081801363456,
    "hostname": "typhon",
    "uid": 1012,
    "user": "jwodder"
  },
  "env": {},
  "gpu": [
    {
      "index": "0",
      "name": "NVIDIA A100-PCIE-40GB",
      "bus_id": "00000000:31:00.0",
      "driver_version": "560.35.05",
      "memory.total": "40960 MiB",
      "compute_mode": "Default"
    }
  ],
  "duct_version": "0.8.0",
  "schema_version": "0.2.0",
  "execution_summary": {
    "exit_code": 0,
    "command": "target/release/s3invsync -l TRACE --date 2025-01-01T01-00Z s3://dandiarchive-inventory/dandiarchive/dandiset-manifest/ /mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/backup",
    "logs_prefix": "/mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/02-",
    "wall_clock_time": 42.217392683029175,
    "peak_rss": 213995520,
    "average_rss": 129954474.66666669,
    "peak_vsz": 2800410624,
    "average_vsz": 2325391262.4761906,
    "peak_pmem": 0.0,
    "average_pmem": 0.0,
    "peak_pcpu": 227.0,
    "average_pcpu": 193.18333333333334,
    "num_samples": 42,
    "num_reports": 2,
    "start_time": 1736552056.6350358,
    "end_time": 1736552098.8524284
  },
  "output_paths": {
    "stdout": "/mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/02-stdout",
    "stderr": "/mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/02-stderr",
    "usage": "/mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/02-usage.json",
    "info": "/mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/02-info.json",
    "prefix": "/mnt/tmp/jwodder/s3invsync/rollback-2025.01.10.21.32.06Z-gh-33-cd6507c/02-"
  }
}

Run Reports #106

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions