Add ONEFLOW_DEBUG_CHECKPOINT to debug torch.load

[jackalcooper]

No description provided.

[github-actions]

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

[github-actions]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/10312/

[github-actions]

Speed stats:

GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 44.0ms (= 4397.9ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.6ms (= 5756.8ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.31 (= 57.6ms / 44.0ms)

OneFlow resnet50 time: 26.1ms (= 2605.5ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 38.1ms (= 3805.8ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.46 (= 38.1ms / 26.1ms)

OneFlow resnet50 time: 19.7ms (= 3935.7ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 34.9ms (= 6986.2ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.78 (= 34.9ms / 19.7ms)

OneFlow resnet50 time: 19.2ms (= 3835.6ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 30.9ms (= 6171.6ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.61 (= 30.9ms / 19.2ms)

OneFlow resnet50 time: 17.5ms (= 3492.1ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 31.1ms (= 6222.1ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.78 (= 31.1ms / 17.5ms)

OneFlow swin dataloader time: 0.201s (= 40.159s / 200, num_workers=1)
PyTorch swin dataloader time: 0.130s (= 25.927s / 200, num_workers=1)
Relative speed: 0.646 (= 0.130s / 0.201s)

OneFlow swin dataloader time: 0.056s (= 11.255s / 200, num_workers=4)
PyTorch swin dataloader time: 0.033s (= 6.587s / 200, num_workers=4)
Relative speed: 0.585 (= 0.033s / 0.056s)

OneFlow swin dataloader time: 0.030s (= 6.070s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.324s / 200, num_workers=8)
Relative speed: 0.548 (= 0.017s / 0.030s)

❌ OneFlow resnet50 time: 47.5ms (= 4747.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 64.3ms (= 6433.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.36 (= 64.3ms / 47.5ms)

OneFlow resnet50 time: 31.4ms (= 3141.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 44.4ms (= 4439.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.41 (= 44.4ms / 31.4ms)

OneFlow resnet50 time: 24.4ms (= 4873.7ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 40.2ms (= 8035.4ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.65 (= 40.2ms / 24.4ms)

OneFlow resnet50 time: 22.9ms (= 4578.9ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.7ms (= 7731.4ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.69 (= 38.7ms / 22.9ms)

OneFlow resnet50 time: 21.1ms (= 4211.7ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 34.2ms (= 6835.9ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.62 (= 34.2ms / 21.1ms)