训练问题

[WangKun328]

2024-02-19 16:34:22,237 - mmdet - INFO - Iter [4950/120000] lr: 4.000e-04, eta: 6:13:37, time: 0.202, data_time: 0.020, memory: 4061, loss_mask: 0.1048, loss_texture: 0.1240, iou: 0.7859, loss: 0.2288
2024-02-19 16:34:31,688 - mmdet - INFO - Saving checkpoint at 5000 iterations
2024-02-19 16:34:35,605 - mmdet - INFO - Exp name: segrefiner_lr.py
2024-02-19 16:34:35,606 - mmdet - INFO - Iter [5000/120000] lr: 4.000e-04, eta: 6:14:51, time: 0.267, data_time: 0.007, memory: 4061, loss_mask: 0.1186, loss_texture: 0.1323, iou: 0.7998, loss: 0.2509
2024-02-19 16:34:46,323 - mmdet - INFO - Iter [5050/120000] lr: 4.000e-04, eta: 6:15:03, time: 0.214, data_time: 0.032, memory: 4061, loss_mask: 0.1113, loss_texture: 0.1355, iou: 0.8468, loss: 0.2468
2024-02-19 16:34:55,565 - mmdet - INFO - Iter [5100/120000] lr: 4.000e-04, eta: 6:14:40, time: 0.185, data_time: 0.006, memory: 4061, loss_mask: 0.1199, loss_texture: 0.1308, iou: 0.8084, loss: 0.2507
2024-02-19 16:35:06,691 - mmdet - INFO - Iter [5150/120000] lr: 4.000e-04, eta: 6:15:01, time: 0.223, data_time: 0.043, memory: 4061, loss_mask: 0.1118, loss_texture: 0.1262, iou: 0.7809, loss: 0.2380
2024-02-19 16:35:16,616 - mmdet - INFO - Iter [5200/120000] lr: 4.000e-04, eta: 6:14:54, time: 0.198, data_time: 0.015, memory: 4061, loss_mask: 0.1292, loss_texture: 0.1344, iou: 0.7996, loss: 0.2636
2024-02-19 16:35:26,801 - mmdet - INFO - Iter [5250/120000] lr: 4.000e-04, eta: 6:14:52, time: 0.204, data_time: 0.016, memory: 4061, loss_mask: 0.1406, loss_texture: 0.1342, iou: 0.8005, loss: 0.2748
2024-02-19 16:35:37,887 - mmdet - INFO - Iter [5300/120000] lr: 4.000e-04, eta: 6:15:10, time: 0.222, data_time: 0.037, memory: 4061, loss_mask: 0.1334, loss_texture: 0.1322, iou: 0.8054, loss: 0.2656
2024-02-19 16:35:46,770 - mmdet - INFO - Iter [5350/120000] lr: 4.000e-04, eta: 6:14:41, time: 0.178, data_time: 0.017, memory: 4061, loss_mask: 0.1028, loss_texture: 0.1307, iou: 0.8518, loss: 0.2335
2024-02-19 16:35:57,319 - mmdet - INFO - Iter [5400/120000] lr: 4.000e-04, eta: 6:14:47, time: 0.211, data_time: 0.016, memory: 4061, loss_mask: 0.2681, loss_texture: 0.1454, iou: 0.6773, loss: 0.4135
2024-02-19 16:36:07,540 - mmdet - INFO - Iter [5450/120000] lr: 4.000e-04, eta: 6:14:45, time: 0.204, data_time: 0.010, memory: 4061, loss_mask: 0.4376, loss_texture: 0.1406, iou: 0.3416, loss: 0.5782
2024-02-19 16:36:17,684 - mmdet - INFO - Iter [5500/120000] lr: 4.000e-04, eta: 6:14:43, time: 0.203, data_time: 0.023, memory: 4061, loss_mask: 0.5540, loss_texture: 0.1434, iou: 0.0000, loss: 0.6974
2024-02-19 16:36:27,192 - mmdet - INFO - Iter [5550/120000] lr: 4.000e-04, eta: 6:14:26, time: 0.190, data_time: 0.009, memory: 4061, loss_mask: 0.6086, loss_texture: 0.1315, iou: 0.0000, loss: 0.7401
2024-02-19 16:36:38,381 - mmdet - INFO - Iter [5600/120000] lr: 4.000e-04, eta: 6:14:45, time: 0.224, data_time: 0.038, memory: 4061, loss_mask: 0.5943, loss_texture: 0.1409, iou: 0.0000, loss: 0.7352
2024-02-19 16:36:48,669 - mmdet - INFO - Iter [5650/120000] lr: 4.000e-04, eta: 6:14:44, time: 0.206, data_time: 0.028, memory: 4061, loss_mask: 0.5827, loss_texture: 0.1452, iou: 0.0000, loss: 0.7279
2024-02-19 16:36:57,981 - mmdet - INFO - Iter [5700/120000] lr: 4.000e-04, eta: 6:14:24, time: 0.186, data_time: 0.008, memory: 4061, loss_mask: 0.5287, loss_texture: 0.1281, iou: 0.0000, loss: 0.6568
2024-02-19 16:37:07,785 - mmdet - INFO - Iter [5750/120000] lr: 4.000e-04, eta: 6:14:13, time: 0.196, data_time: 0.017, memory: 4061, loss_mask: 0.5875, loss_texture: 0.1300, iou: 0.0000, loss: 0.7175
2024-02-19 16:37:17,714 - mmdet - INFO - Iter [5800/120000] lr: 4.000e-04, eta: 6:14:06, time: 0.199, data_time: 0.017, memory: 4061, loss_mask: 0.5624, loss_texture: 0.1306, iou: 0.0000, loss: 0.6930
2024-02-19 16:37:27,840 - mmdet - INFO - Iter [5850/120000] lr: 4.000e-04, eta: 6:14:02, time: 0.202, data_time: 0.024, memory: 4061, loss_mask: 0.5655, loss_texture: 0.1239, iou: 0.0000, loss: 0.6894
2024-02-19 16:37:40,109 - mmdet - INFO - Iter [5900/120000] lr: 4.000e-04, eta: 6:14:39, time: 0.245, data_time: 0.066, memory: 4061, loss_mask: 0.6541, loss_texture: 0.1476, iou: 0.0000, loss: 0.8016
2024-02-19 16:37:50,496 - mmdet - INFO - Iter [5950/120000] lr: 4.000e-04, eta: 6:14:39, time: 0.208, data_time: 0.030, memory: 4061, loss_mask: 0.5569, loss_texture: 0.1273, iou: 0.0000, loss: 0.6842
从5500iter开始iou变成0是为什么呢

[WangKun328]

并且loss此时出现了增大的趋势 @MengyuWang826

[yanrihong]

我遇到了一样的问题，IOU会变成0 ，请问你解决了吗

[WangKun328]

我遇到了一样的问题，IOU会变成0 ，请问你解决了吗

解决了，适当调整学习率即可@ @yanrihong

[wfantastic]

Hello, excuse me, I see you have solved this problem, I also encountered it when training my own dataset. What is the basis for adjusting the learning rate appropriately?