train problem

[SunQiu88]

Hello!
why the train acc is always is 0.00 when i train the shelfnet50 ?can you tell me why?

[fmahoudeau]

Hello,
I looked back at my training logs and this is what I found. The training accuracy increases very slowly and the test AP is 0.000 at the first epoch, but then it increases pretty fast during the following epochs.
You might have wrongly configured your dataset. Check what comes into the model.
Also you need a large batch size. I've used 72.
Good luck.

2019-08-07 13:11:06,514 Epoch: [0][0/2081] Time 12.868s (12.868s) Speed 5.6 samples/s Data 4.801s (4.801s) Loss 0.30140 (0.30140) Accuracy 0.013 (0.013)
2019-08-07 13:14:15,316 Epoch: [0][200/2081] Time 0.933s (1.003s) Speed 77.2 samples/s Data 0.000s (0.037s) Loss 0.00129 (0.00550) Accuracy 0.014 (0.010)
2019-08-07 13:17:26,591 Epoch: [0][400/2081] Time 0.940s (0.980s) Speed 76.6 samples/s Data 0.000s (0.025s) Loss 0.00122 (0.00334) Accuracy 0.046 (0.018)
2019-08-07 13:20:38,990 Epoch: [0][600/2081] Time 0.952s (0.974s) Speed 75.6 samples/s Data 0.000s (0.020s) Loss 0.00103 (0.00261) Accuracy 0.059 (0.031)
2019-08-07 13:23:52,912 Epoch: [0][800/2081] Time 0.954s (0.973s) Speed 75.5 samples/s Data 0.000s (0.018s) Loss 0.00108 (0.00224) Accuracy 0.089 (0.045)
2019-08-07 13:27:05,744 Epoch: [0][1000/2081] Time 0.951s (0.971s) Speed 75.7 samples/s Data 0.000s (0.017s) Loss 0.00112 (0.00201) Accuracy 0.100 (0.056)
2019-08-07 13:30:18,466 Epoch: [0][1200/2081] Time 0.950s (0.970s) Speed 75.8 samples/s Data 0.000s (0.016s) Loss 0.00112 (0.00186) Accuracy 0.146 (0.066)
2019-08-07 13:33:31,523 Epoch: [0][1400/2081] Time 0.948s (0.969s) Speed 75.9 samples/s Data 0.000s (0.016s) Loss 0.00107 (0.00176) Accuracy 0.124 (0.074)
2019-08-07 13:36:46,397 Epoch: [0][1600/2081] Time 0.954s (0.970s) Speed 75.5 samples/s Data 0.000s (0.015s) Loss 0.00108 (0.00167) Accuracy 0.104 (0.080)
2019-08-07 13:40:00,635 Epoch: [0][1800/2081] Time 0.955s (0.970s) Speed 75.4 samples/s Data 0.000s (0.015s) Loss 0.00115 (0.00161) Accuracy 0.116 (0.084)
2019-08-07 13:43:16,970 Epoch: [0][2000/2081] Time 0.953s (0.971s) Speed 75.5 samples/s Data 0.000s (0.014s) Loss 0.00108 (0.00156) Accuracy 0.142 (0.088)
2019-08-07 13:44:47,822 Test: [0/89] Time 5.568 (5.568) Loss 0.0011 (0.0011) Accuracy 0.188 (0.188)
2019-08-07 13:45:53,815 => writing results json to output/coco/pose_shelfnet/shelf_384x288_adam_lr1e-3/results/keypoints_val2017_results_0.json
2019-08-07 13:45:59,271 | Arch | AP | Ap .5 | AP .75 | AP (M) | AP (L) | AR | AR .5 | AR .75 | AR (M) | AR (L) |
2019-08-07 13:45:59,272 |---|---|---|---|---|---|---|---|---|---|---|
2019-08-07 13:45:59,272 | pose_shelfnet | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.002 | 0.000 | 0.001 | 0.000 |
2019-08-07 13:45:59,276 => saving checkpoint to output/coco/pose_shelfnet/shelf_384x288_adam_lr1e-3
2019-08-07 13:46:05,278 Epoch: [1][0/2081] Time 5.589s (5.589s) Speed 12.9 samples/s Data 4.289s (4.289s) Loss 0.00111 (0.00111) Accuracy 0.138 (0.138)
2019-08-07 13:49:23,169 Epoch: [1][200/2081] Time 0.969s (1.012s) Speed 74.3 samples/s Data 0.000s (0.044s) Loss 0.00098 (0.00105) Accuracy 0.148 (0.147)
2019-08-07 13:52:38,113 Epoch: [1][400/2081] Time 0.956s (0.994s) Speed 75.3 samples/s Data 0.000s (0.028s) Loss 0.00117 (0.00104) Accuracy 0.154 (0.152)
2019-08-07 13:55:52,133 Epoch: [1][600/2081] Time 0.947s (0.986s) Speed 76.0 samples/s Data 0.000s (0.023s) Loss 0.00097 (0.00103) Accuracy 0.176 (0.160)
2019-08-07 13:59:06,414 Epoch: [1][800/2081] Time 0.952s (0.982s) Speed 75.6 samples/s Data 0.000s (0.020s) Loss 0.00110 (0.00102) Accuracy 0.234 (0.170)
2019-08-07 14:02:20,631 Epoch: [1][1000/2081] Time 0.954s (0.980s) Speed 75.5 samples/s Data 0.000s (0.018s) Loss 0.00095 (0.00101) Accuracy 0.224 (0.181)
2019-08-07 14:05:34,841 Epoch: [1][1200/2081] Time 0.957s (0.978s) Speed 75.2 samples/s Data 0.000s (0.017s) Loss 0.00097 (0.00100) Accuracy 0.284 (0.195)
2019-08-07 14:08:49,524 Epoch: [1][1400/2081] Time 0.955s (0.978s) Speed 75.4 samples/s Data 0.000s (0.017s) Loss 0.00087 (0.00099) Accuracy 0.294 (0.210)
2019-08-07 14:12:04,047 Epoch: [1][1600/2081] Time 0.957s (0.977s) Speed 75.2 samples/s Data 0.000s (0.016s) Loss 0.00091 (0.00097) Accuracy 0.326 (0.225)
2019-08-07 14:15:18,959 Epoch: [1][1800/2081] Time 0.998s (0.977s) Speed 72.2 samples/s Data 0.000s (0.016s) Loss 0.00081 (0.00096) Accuracy 0.403 (0.243)
2019-08-07 14:18:33,483 Epoch: [1][2000/2081] Time 0.955s (0.976s) Speed 75.4 samples/s Data 0.000s (0.015s) Loss 0.00082 (0.00094) Accuracy 0.424 (0.262)
2019-08-07 14:19:56,911 Test: [0/89] Time 4.398 (4.398) Loss 0.0008 (0.0008) Accuracy 0.550 (0.550)
2019-08-07 14:21:03,622 => writing results json to output/coco/pose_shelfnet/shelf_384x288_adam_lr1e-3/results/keypoints_val2017_results_0.json
2019-08-07 14:21:09,965 | Arch | AP | Ap .5 | AP .75 | AP (M) | AP (L) | AR | AR .5 | AR .75 | AR (M) | AR (L) |
2019-08-07 14:21:09,966 |---|---|---|---|---|---|---|---|---|---|---|
2019-08-07 14:21:09,966 | pose_shelfnet | 0.167 | 0.555 | 0.043 | 0.158 | 0.186 | 0.254 | 0.645 | 0.152 | 0.232 | 0.284 |
2019-08-07 14:21:09,966 => saving checkpoint to output/coco/pose_shelfnet/shelf_384x288_adam_lr1e-3
2019-08-07 14:21:16,837 Epoch: [2][0/2081] Time 4.943s (4.943s) Speed 14.6 samples/s Data 3.659s (3.659s) Loss 0.00080 (0.00080) Accuracy 0.443 (0.443)
2019-08-07 14:24:35,079 Epoch: [2][200/2081] Time 0.972s (1.011s) Speed 74.1 samples/s Data 0.000s (0.044s) Loss 0.00070 (0.00077) Accuracy 0.525 (0.470)
2019-08-07 14:27:49,903 Epoch: [2][400/2081] Time 0.954s (0.993s) Speed 75.5 samples/s Data 0.000s (0.028s) Loss 0.00073 (0.00075) Accuracy 0.514 (0.482)
2019-08-07 14:31:04,233 Epoch: [2][600/2081] Time 0.953s (0.986s) Speed 75.5 samples/s Data 0.000s (0.023s) Loss 0.00072 (0.00074) Accuracy 0.462 (0.492)
2019-08-07 14:34:17,834 Epoch: [2][800/2081] Time 0.953s (0.981s) Speed 75.5 samples/s Data 0.000s (0.020s) Loss 0.00069 (0.00073) Accuracy 0.573 (0.500)
2019-08-07 14:37:31,866 Epoch: [2][1000/2081] Time 0.955s (0.979s) Speed 75.4 samples/s Data 0.000s (0.018s) Loss 0.00070 (0.00072) Accuracy 0.500 (0.508)
2019-08-07 14:40:45,651 Epoch: [2][1200/2081] Time 0.954s (0.977s) Speed 75.5 samples/s Data 0.000s (0.017s) Loss 0.00067 (0.00071) Accuracy 0.574 (0.517)
2019-08-07 14:43:59,844 Epoch: [2][1400/2081] Time 0.955s (0.976s) Speed 75.4 samples/s Data 0.000s (0.017s) Loss 0.00060 (0.00070) Accuracy 0.587 (0.524)
2019-08-07 14:47:14,247 Epoch: [2][1600/2081] Time 0.952s (0.976s) Speed 75.6 samples/s Data 0.000s (0.016s) Loss 0.00064 (0.00069) Accuracy 0.583 (0.530)
2019-08-07 14:50:28,209 Epoch: [2][1800/2081] Time 0.949s (0.975s) Speed 75.9 samples/s Data 0.000s (0.016s) Loss 0.00065 (0.00069) Accuracy 0.574 (0.535)
2019-08-07 14:53:42,338 Epoch: [2][2000/2081] Time 0.954s (0.975s) Speed 75.5 samples/s Data 0.000s (0.015s) Loss 0.00063 (0.00068) Accuracy 0.575 (0.540)
2019-08-07 14:55:06,402 Test: [0/89] Time 5.054 (5.054) Loss 0.0006 (0.0006) Accuracy 0.692 (0.692)
2019-08-07 14:56:15,207 => writing results json to output/coco/pose_shelfnet/shelf_384x288_adam_lr1e-3/results/keypoints_val2017_results_0.json
2019-08-07 14:56:21,045 | Arch | AP | Ap .5 | AP .75 | AP (M) | AP (L) | AR | AR .5 | AR .75 | AR (M) | AR (L) |
2019-08-07 14:56:21,045 |---|---|---|---|---|---|---|---|---|---|---|
2019-08-07 14:56:21,046 | pose_shelfnet | 0.418 | 0.778 | 0.407 | 0.391 | 0.462 | 0.478 | 0.803 | 0.500 | 0.441 | 0.531 |