Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Training process abnormal. #4

Open
lixiang007666 opened this issue Jan 7, 2024 · 14 comments
Open

Training process abnormal. #4

lixiang007666 opened this issue Jan 7, 2024 · 14 comments

Comments

@lixiang007666
Copy link

lixiang007666 commented Jan 7, 2024 •
edited
Loading

I have tried many different settings, but during the training of segrefiner, the IoU becomes zero after only a few steps. Is this a known issue?

Thanks.

image
@MengyuWang826
Copy link
Owner

I have tried many different settings, but during the training of segrefiner, the IoU becomes zero after only a few steps. Is this a known issue?

Thanks.

image

I haven't encountered this issue before. Please paste your complete settings and startup command here, and I'll find some time in the next few days to test it.

@yusuke-ai yusuke-ai mentioned this issue Jan 19, 2024
Open
@yusuke-ai
Copy link

I got an similar issue and the training configuration and the log is below.

Training Configuration

checkpoint_config = dict(
    interval=5000, by_epoch=False, save_last=True, max_keep_ckpts=20)
log_config = dict(
    interval=50, hooks=[dict(type='TextLoggerHook', by_epoch=False)])
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = 'segrefiner_hr_latest.pth'
workflow = [('train', 5000)]
opencv_num_threads = 0
mp_start_method = 'fork'
auto_scale_lr = dict(enable=False, base_batch_size=16)
object_size = 256
task = 'instance'
model = dict(
    type='SegRefiner',
    task='instance',
    step=6,
    denoise_model=dict(
        type='DenoiseUNet',
        in_channels=4,
        out_channels=1,
        model_channels=128,
        num_res_blocks=2,
        num_heads=4,
        num_heads_upsample=-1,
        attention_strides=(16, 32),
        learn_time_embd=True,
        channel_mult=(1, 1, 2, 2, 4, 4),
        dropout=0.0),
    diffusion_cfg=dict(
        betas=dict(type='linear', start=0.8, stop=0, num_timesteps=6),
        diff_iter=False),
    test_cfg=dict())
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='LoadAnnotations',
        with_bbox=False,
        with_label=False,
        with_mask=True),
    dict(type='LoadPatchData', object_size=256, patch_size=256),
    dict(type='Resize', img_scale=(256, 256), keep_ratio=False),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(
        type='Normalize',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        to_rgb=True),
    dict(type='DefaultFormatBundle'),
    dict(
        type='Collect',
        keys=[
            'object_img', 'object_gt_masks', 'object_coarse_masks',
            'patch_img', 'patch_gt_masks', 'patch_coarse_masks'
        ])
]
dataset_type = 'HRCollectionDataset'
img_root = '/share/project/datasets/MSCOCO/coco2017/'
ann_root = '/share/project/datasets/LVIS/'
train_dataloader = dict(samples_per_gpu=1, workers_per_gpu=1)
data = dict(
    train=dict(
        type='HRCollectionDataset',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='LoadAnnotations',
                with_bbox=False,
                with_label=False,
                with_mask=True),
            dict(type='LoadPatchData', object_size=256, patch_size=256),
            dict(type='Resize', img_scale=(256, 256), keep_ratio=False),
            dict(type='RandomFlip', flip_ratio=0.5),
            dict(
                type='Normalize',
                mean=[123.675, 116.28, 103.53],
                std=[58.395, 57.12, 57.375],
                to_rgb=True),
            dict(type='DefaultFormatBundle'),
            dict(
                type='Collect',
                keys=[
                    'object_img', 'object_gt_masks', 'object_coarse_masks',
                    'patch_img', 'patch_gt_masks', 'patch_coarse_masks'
                ])
        ],
        data_root='data/',
        collection_datasets=['thin', 'dis'],
        collection_json='data/collection_hr.json'),
    train_dataloader=dict(samples_per_gpu=1, workers_per_gpu=1),
    val=dict(),
    test=dict())
optimizer = dict(
    type='AdamW', lr=0.0004, weight_decay=0, eps=1e-08, betas=(0.9, 0.999))
optimizer_config = dict(grad_clip=None)
max_iters = 120000
runner = dict(type='IterBasedRunner', max_iters=120000)
lr_config = dict(
    policy='step',
    gamma=0.5,
    by_epoch=False,
    step=[80000, 100000],
    warmup='linear',
    warmup_by_epoch=False,
    warmup_ratio=1.0,
    warmup_iters=10)
interval = 5000
data_root = 'data/'
work_dir = './work_dirs/segrefiner_hr'
auto_resume = False
gpu_ids = [0]

The Log

2024-01-19 12:24:03,655 - mmdet - INFO - workflow: [('train', 5000)], max: 120000 iters
2024-01-19 12:24:03,655 - mmdet - INFO - Checkpoints will be saved to /home/yusuke/gitrepos/SegRefiner/work_dirs/segrefiner_hr by HardDiskBackend.
2024-01-19 12:24:15,210 - mmdet - INFO - Iter [50/120000]	lr: 4.000e-04, eta: 7:41:07, time: 0.231, data_time: 0.005, memory: 6267, loss_mask: 0.0749, loss_texture: 0.0805, iou: 0.9283, loss: 0.1554
2024-01-19 12:24:25,180 - mmdet - INFO - Iter [100/120000]	lr: 4.000e-04, eta: 7:09:42, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.0736, loss_texture: 0.0857, iou: 0.9084, loss: 0.1592
2024-01-19 12:24:35,107 - mmdet - INFO - Iter [150/120000]	lr: 4.000e-04, eta: 6:58:32, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.4370, loss_texture: 0.2199, iou: 0.5518, loss: 0.6569
2024-01-19 12:24:45,140 - mmdet - INFO - Iter [200/120000]	lr: 4.000e-04, eta: 6:53:55, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.2852, loss_texture: 0.2145, iou: 0.5794, loss: 0.4996
2024-01-19 12:24:55,056 - mmdet - INFO - Iter [250/120000]	lr: 4.000e-04, eta: 6:50:09, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.2346, loss_texture: 0.2139, iou: 0.6228, loss: 0.4484
2024-01-19 12:25:05,039 - mmdet - INFO - Iter [300/120000]	lr: 4.000e-04, eta: 6:48:02, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2050, loss_texture: 0.1940, iou: 0.6370, loss: 0.3990
2024-01-19 12:25:14,964 - mmdet - INFO - Iter [350/120000]	lr: 4.000e-04, eta: 6:46:09, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.2655, loss_texture: 0.2266, iou: 0.6330, loss: 0.4921
2024-01-19 12:25:24,949 - mmdet - INFO - Iter [400/120000]	lr: 4.000e-04, eta: 6:44:59, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2196, loss_texture: 0.2139, iou: 0.7239, loss: 0.4336
2024-01-19 12:25:34,935 - mmdet - INFO - Iter [450/120000]	lr: 4.000e-04, eta: 6:44:03, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1660, loss_texture: 0.1727, iou: 0.7408, loss: 0.3387
2024-01-19 12:25:44,947 - mmdet - INFO - Iter [500/120000]	lr: 4.000e-04, eta: 6:43:22, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2036, loss_texture: 0.2130, iou: 0.7069, loss: 0.4167
2024-01-19 12:25:54,943 - mmdet - INFO - Iter [550/120000]	lr: 4.000e-04, eta: 6:42:44, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1658, loss_texture: 0.1909, iou: 0.7922, loss: 0.3566
2024-01-19 12:26:04,933 - mmdet - INFO - Iter [600/120000]	lr: 4.000e-04, eta: 6:42:09, time: 0.200, data_time: 0.004, memory: 6267, loss_mask: 0.1983, loss_texture: 0.2182, iou: 0.7171, loss: 0.4165
2024-01-19 12:26:14,901 - mmdet - INFO - Iter [650/120000]	lr: 4.000e-04, eta: 6:41:33, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1579, loss_texture: 0.1868, iou: 0.7118, loss: 0.3447
2024-01-19 12:26:24,843 - mmdet - INFO - Iter [700/120000]	lr: 4.000e-04, eta: 6:40:57, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2049, loss_texture: 0.2065, iou: 0.7138, loss: 0.4114
2024-01-19 12:26:34,789 - mmdet - INFO - Iter [750/120000]	lr: 4.000e-04, eta: 6:40:25, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2391, loss_texture: 0.2162, iou: 0.6966, loss: 0.4553
2024-01-19 12:26:44,700 - mmdet - INFO - Iter [800/120000]	lr: 4.000e-04, eta: 6:39:51, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.1903, loss_texture: 0.2007, iou: 0.6805, loss: 0.3910
2024-01-19 12:26:54,823 - mmdet - INFO - Iter [850/120000]	lr: 4.000e-04, eta: 6:39:49, time: 0.202, data_time: 0.006, memory: 6267, loss_mask: 0.2275, loss_texture: 0.2216, iou: 0.6903, loss: 0.4491
2024-01-19 12:27:04,774 - mmdet - INFO - Iter [900/120000]	lr: 4.000e-04, eta: 6:39:24, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2924, loss_texture: 0.2216, iou: 0.6570, loss: 0.5139
2024-01-19 12:27:14,760 - mmdet - INFO - Iter [950/120000]	lr: 4.000e-04, eta: 6:39:04, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2299, loss_texture: 0.2123, iou: 0.6476, loss: 0.4421
2024-01-19 12:27:24,758 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:27:24,758 - mmdet - INFO - Iter [1000/120000]	lr: 4.000e-04, eta: 6:38:47, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2042, loss_texture: 0.1917, iou: 0.6918, loss: 0.3960
2024-01-19 12:27:34,786 - mmdet - INFO - Iter [1050/120000]	lr: 4.000e-04, eta: 6:38:34, time: 0.201, data_time: 0.004, memory: 6267, loss_mask: 0.2077, loss_texture: 0.1856, iou: 0.7152, loss: 0.3933
2024-01-19 12:27:44,821 - mmdet - INFO - Iter [1100/120000]	lr: 4.000e-04, eta: 6:38:22, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1809, loss_texture: 0.1987, iou: 0.7236, loss: 0.3796
2024-01-19 12:27:54,808 - mmdet - INFO - Iter [1150/120000]	lr: 4.000e-04, eta: 6:38:06, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2191, loss_texture: 0.2069, iou: 0.7322, loss: 0.4260
2024-01-19 12:28:04,823 - mmdet - INFO - Iter [1200/120000]	lr: 4.000e-04, eta: 6:37:52, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2450, loss_texture: 0.2021, iou: 0.7341, loss: 0.4471
2024-01-19 12:28:14,884 - mmdet - INFO - Iter [1250/120000]	lr: 4.000e-04, eta: 6:37:43, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1971, loss_texture: 0.2114, iou: 0.7124, loss: 0.4085
2024-01-19 12:28:24,896 - mmdet - INFO - Iter [1300/120000]	lr: 4.000e-04, eta: 6:37:30, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1347, loss_texture: 0.1755, iou: 0.8245, loss: 0.3102
2024-01-19 12:28:35,061 - mmdet - INFO - Iter [1350/120000]	lr: 4.000e-04, eta: 6:37:30, time: 0.203, data_time: 0.006, memory: 6267, loss_mask: 0.2002, loss_texture: 0.1978, iou: 0.7008, loss: 0.3980
2024-01-19 12:28:45,045 - mmdet - INFO - Iter [1400/120000]	lr: 4.000e-04, eta: 6:37:15, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1808, loss_texture: 0.1898, iou: 0.7335, loss: 0.3705
2024-01-19 12:28:54,994 - mmdet - INFO - Iter [1450/120000]	lr: 4.000e-04, eta: 6:36:56, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1796, loss_texture: 0.1766, iou: 0.7720, loss: 0.3561
2024-01-19 12:29:04,995 - mmdet - INFO - Iter [1500/120000]	lr: 4.000e-04, eta: 6:36:43, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1564, loss_texture: 0.1758, iou: 0.7233, loss: 0.3321
2024-01-19 12:29:14,931 - mmdet - INFO - Iter [1550/120000]	lr: 4.000e-04, eta: 6:36:25, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1863, loss_texture: 0.2102, iou: 0.7394, loss: 0.3965
2024-01-19 12:29:25,060 - mmdet - INFO - Iter [1600/120000]	lr: 4.000e-04, eta: 6:36:21, time: 0.203, data_time: 0.005, memory: 6267, loss_mask: 0.2034, loss_texture: 0.2099, iou: 0.7317, loss: 0.4134
2024-01-19 12:29:35,074 - mmdet - INFO - Iter [1650/120000]	lr: 4.000e-04, eta: 6:36:09, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1500, loss_texture: 0.1764, iou: 0.7995, loss: 0.3263
2024-01-19 12:29:45,054 - mmdet - INFO - Iter [1700/120000]	lr: 4.000e-04, eta: 6:35:55, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1477, loss_texture: 0.1792, iou: 0.7476, loss: 0.3270
2024-01-19 12:29:55,016 - mmdet - INFO - Iter [1750/120000]	lr: 4.000e-04, eta: 6:35:39, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1770, loss_texture: 0.1875, iou: 0.7434, loss: 0.3645
2024-01-19 12:30:04,943 - mmdet - INFO - Iter [1800/120000]	lr: 4.000e-04, eta: 6:35:22, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1464, loss_texture: 0.1856, iou: 0.7543, loss: 0.3320
2024-01-19 12:30:14,854 - mmdet - INFO - Iter [1850/120000]	lr: 4.000e-04, eta: 6:35:04, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.1349, loss_texture: 0.1745, iou: 0.7693, loss: 0.3094
2024-01-19 12:30:24,803 - mmdet - INFO - Iter [1900/120000]	lr: 4.000e-04, eta: 6:34:49, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1340, loss_texture: 0.1688, iou: 0.8084, loss: 0.3028
2024-01-19 12:30:34,767 - mmdet - INFO - Iter [1950/120000]	lr: 4.000e-04, eta: 6:34:35, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1249, loss_texture: 0.1673, iou: 0.7941, loss: 0.2922
2024-01-19 12:30:44,694 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:30:44,694 - mmdet - INFO - Iter [2000/120000]	lr: 4.000e-04, eta: 6:34:19, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2065, loss_texture: 0.1790, iou: 0.7811, loss: 0.3856
2024-01-19 12:30:54,604 - mmdet - INFO - Iter [2050/120000]	lr: 4.000e-04, eta: 6:34:02, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.1713, loss_texture: 0.2143, iou: 0.7463, loss: 0.3856
2024-01-19 12:31:04,618 - mmdet - INFO - Iter [2100/120000]	lr: 4.000e-04, eta: 6:33:52, time: 0.200, data_time: 0.004, memory: 6267, loss_mask: 0.2016, loss_texture: 0.2008, iou: 0.7280, loss: 0.4023
2024-01-19 12:31:14,605 - mmdet - INFO - Iter [2150/120000]	lr: 4.000e-04, eta: 6:33:40, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1621, loss_texture: 0.2051, iou: 0.7716, loss: 0.3672
2024-01-19 12:31:24,538 - mmdet - INFO - Iter [2200/120000]	lr: 4.000e-04, eta: 6:33:25, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1611, loss_texture: 0.1994, iou: 0.8068, loss: 0.3604
2024-01-19 12:31:34,484 - mmdet - INFO - Iter [2250/120000]	lr: 4.000e-04, eta: 6:33:11, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1697, loss_texture: 0.2121, iou: 0.6758, loss: 0.3818
2024-01-19 12:31:44,713 - mmdet - INFO - Iter [2300/120000]	lr: 4.000e-04, eta: 6:33:12, time: 0.205, data_time: 0.008, memory: 6267, loss_mask: 0.1664, loss_texture: 0.1805, iou: 0.7210, loss: 0.3469
2024-01-19 12:31:54,648 - mmdet - INFO - Iter [2350/120000]	lr: 4.000e-04, eta: 6:32:57, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1260, loss_texture: 0.1762, iou: 0.7685, loss: 0.3021
2024-01-19 12:32:04,631 - mmdet - INFO - Iter [2400/120000]	lr: 4.000e-04, eta: 6:32:45, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1444, loss_texture: 0.1815, iou: 0.8099, loss: 0.3260
2024-01-19 12:32:15,601 - mmdet - INFO - Iter [2450/120000]	lr: 4.000e-04, eta: 6:33:21, time: 0.219, data_time: 0.021, memory: 6267, loss_mask: 0.1563, loss_texture: 0.1941, iou: 0.7882, loss: 0.3505
2024-01-19 12:32:25,657 - mmdet - INFO - Iter [2500/120000]	lr: 4.000e-04, eta: 6:33:12, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1376, loss_texture: 0.1658, iou: 0.7853, loss: 0.3034
2024-01-19 12:32:35,804 - mmdet - INFO - Iter [2550/120000]	lr: 4.000e-04, eta: 6:33:07, time: 0.203, data_time: 0.003, memory: 6267, loss_mask: 0.1314, loss_texture: 0.1676, iou: 0.8216, loss: 0.2991
2024-01-19 12:32:45,851 - mmdet - INFO - Iter [2600/120000]	lr: 4.000e-04, eta: 6:32:57, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1733, loss_texture: 0.1933, iou: 0.7472, loss: 0.3666
2024-01-19 12:32:55,850 - mmdet - INFO - Iter [2650/120000]	lr: 4.000e-04, eta: 6:32:45, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1992, loss_texture: 0.2001, iou: 0.6920, loss: 0.3993
2024-01-19 12:33:05,933 - mmdet - INFO - Iter [2700/120000]	lr: 4.000e-04, eta: 6:32:37, time: 0.202, data_time: 0.003, memory: 6267, loss_mask: 0.1481, loss_texture: 0.1848, iou: 0.7880, loss: 0.3328
2024-01-19 12:33:15,942 - mmdet - INFO - Iter [2750/120000]	lr: 4.000e-04, eta: 6:32:25, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1818, loss_texture: 0.1817, iou: 0.7667, loss: 0.3635
2024-01-19 12:33:25,938 - mmdet - INFO - Iter [2800/120000]	lr: 4.000e-04, eta: 6:32:13, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1897, loss_texture: 0.1949, iou: 0.7399, loss: 0.3846
2024-01-19 12:33:36,020 - mmdet - INFO - Iter [2850/120000]	lr: 4.000e-04, eta: 6:32:05, time: 0.202, data_time: 0.003, memory: 6267, loss_mask: 0.1986, loss_texture: 0.1722, iou: 0.7496, loss: 0.3707
2024-01-19 12:33:46,018 - mmdet - INFO - Iter [2900/120000]	lr: 4.000e-04, eta: 6:31:53, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1509, loss_texture: 0.1899, iou: 0.7735, loss: 0.3408
2024-01-19 12:33:56,028 - mmdet - INFO - Iter [2950/120000]	lr: 4.000e-04, eta: 6:31:42, time: 0.200, data_time: 0.004, memory: 6267, loss_mask: 0.1364, loss_texture: 0.1691, iou: 0.7962, loss: 0.3055
2024-01-19 12:34:06,055 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:34:06,055 - mmdet - INFO - Iter [3000/120000]	lr: 4.000e-04, eta: 6:31:31, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1484, loss_texture: 0.1878, iou: 0.7773, loss: 0.3362
2024-01-19 12:34:16,073 - mmdet - INFO - Iter [3050/120000]	lr: 4.000e-04, eta: 6:31:21, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1766, loss_texture: 0.2010, iou: 0.7864, loss: 0.3776
2024-01-19 12:34:26,081 - mmdet - INFO - Iter [3100/120000]	lr: 4.000e-04, eta: 6:31:09, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1475, loss_texture: 0.1973, iou: 0.7756, loss: 0.3448
2024-01-19 12:34:36,077 - mmdet - INFO - Iter [3150/120000]	lr: 4.000e-04, eta: 6:30:58, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1222, loss_texture: 0.1656, iou: 0.8325, loss: 0.2878
2024-01-19 12:34:46,095 - mmdet - INFO - Iter [3200/120000]	lr: 4.000e-04, eta: 6:30:47, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1147, loss_texture: 0.1624, iou: 0.7541, loss: 0.2771
2024-01-19 12:34:56,057 - mmdet - INFO - Iter [3250/120000]	lr: 4.000e-04, eta: 6:30:34, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1926, loss_texture: 0.1761, iou: 0.7865, loss: 0.3687
2024-01-19 12:35:06,074 - mmdet - INFO - Iter [3300/120000]	lr: 4.000e-04, eta: 6:30:23, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1360, loss_texture: 0.1826, iou: 0.7763, loss: 0.3186
2024-01-19 12:35:16,077 - mmdet - INFO - Iter [3350/120000]	lr: 4.000e-04, eta: 6:30:12, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.3127, loss_texture: 0.1954, iou: 0.6610, loss: 0.5081
2024-01-19 12:35:26,089 - mmdet - INFO - Iter [3400/120000]	lr: 4.000e-04, eta: 6:30:01, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2593, loss_texture: 0.1952, iou: 0.6893, loss: 0.4545
2024-01-19 12:35:36,096 - mmdet - INFO - Iter [3450/120000]	lr: 4.000e-04, eta: 6:29:50, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1894, loss_texture: 0.2158, iou: 0.7296, loss: 0.4052
2024-01-19 12:35:46,117 - mmdet - INFO - Iter [3500/120000]	lr: 4.000e-04, eta: 6:29:40, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2522, loss_texture: 0.2290, iou: 0.6647, loss: 0.4813
2024-01-19 12:35:56,152 - mmdet - INFO - Iter [3550/120000]	lr: 4.000e-04, eta: 6:29:30, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1732, loss_texture: 0.1959, iou: 0.7273, loss: 0.3691
2024-01-19 12:36:06,232 - mmdet - INFO - Iter [3600/120000]	lr: 4.000e-04, eta: 6:29:21, time: 0.202, data_time: 0.003, memory: 6267, loss_mask: 0.1718, loss_texture: 0.1937, iou: 0.7334, loss: 0.3655
2024-01-19 12:36:16,231 - mmdet - INFO - Iter [3650/120000]	lr: 4.000e-04, eta: 6:29:10, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1612, loss_texture: 0.1986, iou: 0.7909, loss: 0.3598
2024-01-19 12:36:26,350 - mmdet - INFO - Iter [3700/120000]	lr: 4.000e-04, eta: 6:29:03, time: 0.202, data_time: 0.006, memory: 6267, loss_mask: 0.1699, loss_texture: 0.1903, iou: 0.7482, loss: 0.3603
2024-01-19 12:36:36,318 - mmdet - INFO - Iter [3750/120000]	lr: 4.000e-04, eta: 6:28:51, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1348, loss_texture: 0.1824, iou: 0.7632, loss: 0.3172
2024-01-19 12:36:47,253 - mmdet - INFO - Iter [3800/120000]	lr: 4.000e-04, eta: 6:29:08, time: 0.219, data_time: 0.022, memory: 6267, loss_mask: 0.2047, loss_texture: 0.2051, iou: 0.7482, loss: 0.4098
2024-01-19 12:36:57,272 - mmdet - INFO - Iter [3850/120000]	lr: 4.000e-04, eta: 6:28:57, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1410, loss_texture: 0.2001, iou: 0.7394, loss: 0.3412
2024-01-19 12:37:07,299 - mmdet - INFO - Iter [3900/120000]	lr: 4.000e-04, eta: 6:28:47, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1435, loss_texture: 0.1750, iou: 0.7415, loss: 0.3185
2024-01-19 12:37:17,327 - mmdet - INFO - Iter [3950/120000]	lr: 4.000e-04, eta: 6:28:36, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1412, loss_texture: 0.1831, iou: 0.7556, loss: 0.3242
2024-01-19 12:37:27,355 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:37:27,356 - mmdet - INFO - Iter [4000/120000]	lr: 4.000e-04, eta: 6:28:25, time: 0.201, data_time: 0.004, memory: 6267, loss_mask: 0.1276, loss_texture: 0.1654, iou: 0.7787, loss: 0.2930
2024-01-19 12:37:37,406 - mmdet - INFO - Iter [4050/120000]	lr: 4.000e-04, eta: 6:28:16, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1453, loss_texture: 0.1862, iou: 0.7722, loss: 0.3315
2024-01-19 12:37:47,268 - mmdet - INFO - Iter [4100/120000]	lr: 4.000e-04, eta: 6:28:00, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.5625, loss_texture: 0.2051, iou: 0.1537, loss: 0.7676
2024-01-19 12:37:57,120 - mmdet - INFO - Iter [4150/120000]	lr: 4.000e-04, eta: 6:27:45, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6593, loss_texture: 0.2000, iou: 0.0000, loss: 0.8593
2024-01-19 12:38:06,906 - mmdet - INFO - Iter [4200/120000]	lr: 4.000e-04, eta: 6:27:28, time: 0.196, data_time: 0.003, memory: 6267, loss_mask: 0.6285, loss_texture: 0.2061, iou: 0.0000, loss: 0.8345
2024-01-19 12:38:16,736 - mmdet - INFO - Iter [4250/120000]	lr: 4.000e-04, eta: 6:27:12, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6399, loss_texture: 0.1972, iou: 0.0000, loss: 0.8372
2024-01-19 12:38:26,524 - mmdet - INFO - Iter [4300/120000]	lr: 4.000e-04, eta: 6:26:55, time: 0.196, data_time: 0.003, memory: 6267, loss_mask: 0.6509, loss_texture: 0.1880, iou: 0.0000, loss: 0.8390
2024-01-19 12:38:36,357 - mmdet - INFO - Iter [4350/120000]	lr: 4.000e-04, eta: 6:26:40, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6114, loss_texture: 0.2054, iou: 0.0000, loss: 0.8168
2024-01-19 12:38:46,185 - mmdet - INFO - Iter [4400/120000]	lr: 4.000e-04, eta: 6:26:25, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6601, loss_texture: 0.1975, iou: 0.0000, loss: 0.8576
2024-01-19 12:38:55,983 - mmdet - INFO - Iter [4450/120000]	lr: 4.000e-04, eta: 6:26:09, time: 0.196, data_time: 0.003, memory: 6267, loss_mask: 0.6781, loss_texture: 0.2102, iou: 0.0000, loss: 0.8882
2024-01-19 12:39:06,238 - mmdet - INFO - Iter [4500/120000]	lr: 4.000e-04, eta: 6:26:04, time: 0.205, data_time: 0.012, memory: 6267, loss_mask: 0.6450, loss_texture: 0.1818, iou: 0.0000, loss: 0.8268
2024-01-19 12:39:16,085 - mmdet - INFO - Iter [4550/120000]	lr: 4.000e-04, eta: 6:25:50, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6331, loss_texture: 0.2153, iou: 0.0000, loss: 0.8485
2024-01-19 12:39:26,005 - mmdet - INFO - Iter [4600/120000]	lr: 4.000e-04, eta: 6:25:37, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.6359, loss_texture: 0.2007, iou: 0.0000, loss: 0.8366
2024-01-19 12:39:35,871 - mmdet - INFO - Iter [4650/120000]	lr: 4.000e-04, eta: 6:25:23, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6284, loss_texture: 0.1774, iou: 0.0000, loss: 0.8058
2024-01-19 12:39:45,796 - mmdet - INFO - Iter [4700/120000]	lr: 4.000e-04, eta: 6:25:11, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.6485, loss_texture: 0.2023, iou: 0.0000, loss: 0.8507
2024-01-19 12:39:55,629 - mmdet - INFO - Iter [4750/120000]	lr: 4.000e-04, eta: 6:24:56, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6554, loss_texture: 0.1779, iou: 0.0000, loss: 0.8333
2024-01-19 12:40:05,472 - mmdet - INFO - Iter [4800/120000]	lr: 4.000e-04, eta: 6:24:42, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6776, loss_texture: 0.2007, iou: 0.0000, loss: 0.8783

@yusuke-ai
Copy link

also environmental information

mmcv-full              1.7.1
torch                  1.12.1+cu116
torchvision            0.13.1+cu116
mmdet                  2.28.1       /home/yusuke/gitrepos/SegRefiner

@xiaolul2
Copy link

I also encounter the same problem....

@xiaolul2
Copy link

xiaolul2 commented Jan 19, 2024 •
edited
Loading

No decline in loss when iou becomes 0. Maybe it is caused by gradient exploration.
I use only 2 GPUs and change the learning rate to 1e-4 and solve this problem:)

@yusuke-ai
Copy link

@Liuliuliudalu
Thank you for the information!
Can I ask more questions?

  1. what is 14-e? 1e-14??
  2. Did you successfully trained the segmentation refiner with same quality as "segrefiner_hr_latest.pth"

@xiaolul2
Copy link

@Liuliuliudalu Thank you for the information! Can I ask more questions?

  1. what is 14-e? 1e-14??
  2. Did you successfully trained the segmentation refiner with same quality as "segrefiner_hr_latest.pth"

Sorry for my typo, it is 1e-4. I'm still training the segrefiner_lr. Maybe I can reply you later after evaluation.

@MengyuWang826
Copy link
Owner

Thank you all for your attention to this work and sorry for the delayed response. I believe the issue is due to training instability caused by a too-small batch size. The samples_per_gpu=1 setting in the config is only for convenient debugging, and the original results in the paper correspond to a global batch size of $4 \times 8=32$ . Therefore, please increase samples_per_gpu and the number of GPUs as much as possible. When the global batch size is small, a smaller learning rate can help stabilize training, but I'm not sure if this can reproduce the original results.

@MengyuWang826
Copy link
Owner

Thank you all for your attention to this work and sorry for the delayed response. I believe the issue is due to training instability caused by a too-small batch size. The samples_per_gpu=1 setting in the config is only for convenient debugging, and the original results in the paper correspond to a global batch size of 4×8=32 . Therefore, please increase samples_per_gpu and the number of GPUs as much as possible. When the global batch size is small, a smaller learning rate can help stabilize training, but I'm not sure if this can reproduce the original results.

And through testing, I have confirmed that for the initial learning rate of 4e-4 in HR-SegRefiner, the issue of IOU dropping to 0 does not occur when the global batch size reaches 4.

@MengyuWang826 MengyuWang826 mentioned this issue Jan 21, 2024
Closed
@yusuke-ai
Copy link

@Liuliuliudalu
Thank you for the answer!
I couldn't train the same perfromant model as the provided ones...

@MengyuWang826
Thank you for the answer!
I tried with 4e-4 and 2e-4 learning rate with batch size 16 * 4 = 64,
but in either learning rate, the loss often increased a lot like the log below(around per 3000 iterations).
Did you see such a trouble before?

2024-01-21 23:04:10,821 - mmdet - INFO - Iter [17450/120000]	lr: 2.000e-04, eta: 2 days, 1:07:27, time: 1.740, data_time: 0.378, memory: 70378, loss_mask: 0.0225, loss_texture: 0.0297, iou: 0.9646, loss: 0.0522
2024-01-21 23:05:35,286 - mmdet - INFO - Iter [17500/120000]	lr: 2.000e-04, eta: 2 days, 1:05:50, time: 1.689, data_time: 0.182, memory: 70378, loss_mask: 0.0204, loss_texture: 0.0279, iou: 0.9670, loss: 0.0483
2024-01-21 23:07:03,462 - mmdet - INFO - Iter [17550/120000]	lr: 2.000e-04, eta: 2 days, 1:04:36, time: 1.764, data_time: 0.461, memory: 70378, loss_mask: 0.0200, loss_texture: 0.0282, iou: 0.9661, loss: 0.0482
2024-01-21 23:08:29,160 - mmdet - INFO - Iter [17600/120000]	lr: 2.000e-04, eta: 2 days, 1:03:06, time: 1.714, data_time: 0.355, memory: 70378, loss_mask: 0.0224, loss_texture: 0.0303, iou: 0.9648, loss: 0.0527
2024-01-21 23:09:54,733 - mmdet - INFO - Iter [17650/120000]	lr: 2.000e-04, eta: 2 days, 1:01:36, time: 1.712, data_time: 0.475, memory: 70378, loss_mask: 0.0207, loss_texture: 0.0282, iou: 0.9687, loss: 0.0489
2024-01-21 23:11:20,855 - mmdet - INFO - Iter [17700/120000]	lr: 2.000e-04, eta: 2 days, 1:00:10, time: 1.722, data_time: 0.323, memory: 70378, loss_mask: 0.0250, loss_texture: 0.0313, iou: 0.9617, loss: 0.0564
2024-01-21 23:12:49,212 - mmdet - INFO - Iter [17750/120000]	lr: 2.000e-04, eta: 2 days, 0:58:56, time: 1.767, data_time: 0.080, memory: 70378, loss_mask: 0.4141, loss_texture: 0.1946, iou: 0.5304, loss: 0.6087
2024-01-21 23:14:12,939 - mmdet - INFO - Iter [17800/120000]	lr: 2.000e-04, eta: 2 days, 0:57:15, time: 1.675, data_time: 0.184, memory: 70378, loss_mask: 0.2365, loss_texture: 0.1971, iou: 0.6378, loss: 0.4336
2024-01-21 23:15:35,261 - mmdet - INFO - Iter [17850/120000]	lr: 2.000e-04, eta: 2 days, 0:55:27, time: 1.646, data_time: 0.153, memory: 70378, loss_mask: 0.2279, loss_texture: 0.1967, iou: 0.6453, loss: 0.4246
2024-01-21 23:17:04,512 - mmdet - INFO - Iter [17900/120000]	lr: 2.000e-04, eta: 2 days, 0:54:18, time: 1.785, data_time: 0.180, memory: 70378, loss_mask: 0.2043, loss_texture: 0.1948, iou: 0.6626, loss: 0.3991

@MengyuWang826
Copy link
Owner

@Liuliuliudalu Thank you for the answer! I couldn't train the same perfromant model as the provided ones...

@MengyuWang826 Thank you for the answer! I tried with 4e-4 and 2e-4 learning rate with batch size 16 * 4 = 64, but in either learning rate, the loss often increased a lot like the log below(around per 3000 iterations). Did you see such a trouble before?

2024-01-21 23:04:10,821 - mmdet - INFO - Iter [17450/120000]	lr: 2.000e-04, eta: 2 days, 1:07:27, time: 1.740, data_time: 0.378, memory: 70378, loss_mask: 0.0225, loss_texture: 0.0297, iou: 0.9646, loss: 0.0522
2024-01-21 23:05:35,286 - mmdet - INFO - Iter [17500/120000]	lr: 2.000e-04, eta: 2 days, 1:05:50, time: 1.689, data_time: 0.182, memory: 70378, loss_mask: 0.0204, loss_texture: 0.0279, iou: 0.9670, loss: 0.0483
2024-01-21 23:07:03,462 - mmdet - INFO - Iter [17550/120000]	lr: 2.000e-04, eta: 2 days, 1:04:36, time: 1.764, data_time: 0.461, memory: 70378, loss_mask: 0.0200, loss_texture: 0.0282, iou: 0.9661, loss: 0.0482
2024-01-21 23:08:29,160 - mmdet - INFO - Iter [17600/120000]	lr: 2.000e-04, eta: 2 days, 1:03:06, time: 1.714, data_time: 0.355, memory: 70378, loss_mask: 0.0224, loss_texture: 0.0303, iou: 0.9648, loss: 0.0527
2024-01-21 23:09:54,733 - mmdet - INFO - Iter [17650/120000]	lr: 2.000e-04, eta: 2 days, 1:01:36, time: 1.712, data_time: 0.475, memory: 70378, loss_mask: 0.0207, loss_texture: 0.0282, iou: 0.9687, loss: 0.0489
2024-01-21 23:11:20,855 - mmdet - INFO - Iter [17700/120000]	lr: 2.000e-04, eta: 2 days, 1:00:10, time: 1.722, data_time: 0.323, memory: 70378, loss_mask: 0.0250, loss_texture: 0.0313, iou: 0.9617, loss: 0.0564
2024-01-21 23:12:49,212 - mmdet - INFO - Iter [17750/120000]	lr: 2.000e-04, eta: 2 days, 0:58:56, time: 1.767, data_time: 0.080, memory: 70378, loss_mask: 0.4141, loss_texture: 0.1946, iou: 0.5304, loss: 0.6087
2024-01-21 23:14:12,939 - mmdet - INFO - Iter [17800/120000]	lr: 2.000e-04, eta: 2 days, 0:57:15, time: 1.675, data_time: 0.184, memory: 70378, loss_mask: 0.2365, loss_texture: 0.1971, iou: 0.6378, loss: 0.4336
2024-01-21 23:15:35,261 - mmdet - INFO - Iter [17850/120000]	lr: 2.000e-04, eta: 2 days, 0:55:27, time: 1.646, data_time: 0.153, memory: 70378, loss_mask: 0.2279, loss_texture: 0.1967, iou: 0.6453, loss: 0.4246
2024-01-21 23:17:04,512 - mmdet - INFO - Iter [17900/120000]	lr: 2.000e-04, eta: 2 days, 0:54:18, time: 1.785, data_time: 0.180, memory: 70378, loss_mask: 0.2043, loss_texture: 0.1948, iou: 0.6626, loss: 0.3991

This is a bit strange. When I have enough free GPUs available, I will check and retrain it.

@yusuke-ai
Copy link

@MengyuWang826 Thank you!

@Taocunguo
Copy link

还有环境信息

mmcv-full              1.7.1
torch                  1.12.1+cu116
torchvision            0.13.1+cu116
mmdet                  2.28.1       /home/yusuke/gitrepos/SegRefiner

我遇到了类似的问题，训练配置和日志如下。

训练配置

checkpoint_config = dict(
    interval=5000, by_epoch=False, save_last=True, max_keep_ckpts=20)
log_config = dict(
    interval=50, hooks=[dict(type='TextLoggerHook', by_epoch=False)])
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = 'segrefiner_hr_latest.pth'
workflow = [('train', 5000)]
opencv_num_threads = 0
mp_start_method = 'fork'
auto_scale_lr = dict(enable=False, base_batch_size=16)
object_size = 256
task = 'instance'
model = dict(
    type='SegRefiner',
    task='instance',
    step=6,
    denoise_model=dict(
        type='DenoiseUNet',
        in_channels=4,
        out_channels=1,
        model_channels=128,
        num_res_blocks=2,
        num_heads=4,
        num_heads_upsample=-1,
        attention_strides=(16, 32),
        learn_time_embd=True,
        channel_mult=(1, 1, 2, 2, 4, 4),
        dropout=0.0),
    diffusion_cfg=dict(
        betas=dict(type='linear', start=0.8, stop=0, num_timesteps=6),
        diff_iter=False),
    test_cfg=dict())
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='LoadAnnotations',
        with_bbox=False,
        with_label=False,
        with_mask=True),
    dict(type='LoadPatchData', object_size=256, patch_size=256),
    dict(type='Resize', img_scale=(256, 256), keep_ratio=False),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(
        type='Normalize',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        to_rgb=True),
    dict(type='DefaultFormatBundle'),
    dict(
        type='Collect',
        keys=[
            'object_img', 'object_gt_masks', 'object_coarse_masks',
            'patch_img', 'patch_gt_masks', 'patch_coarse_masks'
        ])
]
dataset_type = 'HRCollectionDataset'
img_root = '/share/project/datasets/MSCOCO/coco2017/'
ann_root = '/share/project/datasets/LVIS/'
train_dataloader = dict(samples_per_gpu=1, workers_per_gpu=1)
data = dict(
    train=dict(
        type='HRCollectionDataset',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='LoadAnnotations',
                with_bbox=False,
                with_label=False,
                with_mask=True),
            dict(type='LoadPatchData', object_size=256, patch_size=256),
            dict(type='Resize', img_scale=(256, 256), keep_ratio=False),
            dict(type='RandomFlip', flip_ratio=0.5),
            dict(
                type='Normalize',
                mean=[123.675, 116.28, 103.53],
                std=[58.395, 57.12, 57.375],
                to_rgb=True),
            dict(type='DefaultFormatBundle'),
            dict(
                type='Collect',
                keys=[
                    'object_img', 'object_gt_masks', 'object_coarse_masks',
                    'patch_img', 'patch_gt_masks', 'patch_coarse_masks'
                ])
        ],
        data_root='data/',
        collection_datasets=['thin', 'dis'],
        collection_json='data/collection_hr.json'),
    train_dataloader=dict(samples_per_gpu=1, workers_per_gpu=1),
    val=dict(),
    test=dict())
optimizer = dict(
    type='AdamW', lr=0.0004, weight_decay=0, eps=1e-08, betas=(0.9, 0.999))
optimizer_config = dict(grad_clip=None)
max_iters = 120000
runner = dict(type='IterBasedRunner', max_iters=120000)
lr_config = dict(
    policy='step',
    gamma=0.5,
    by_epoch=False,
    step=[80000, 100000],
    warmup='linear',
    warmup_by_epoch=False,
    warmup_ratio=1.0,
    warmup_iters=10)
interval = 5000
data_root = 'data/'
work_dir = './work_dirs/segrefiner_hr'
auto_resume = False
gpu_ids = [0]

日志

2024-01-19 12:24:03,655 - mmdet - INFO - workflow: [('train', 5000)], max: 120000 iters
2024-01-19 12:24:03,655 - mmdet - INFO - Checkpoints will be saved to /home/yusuke/gitrepos/SegRefiner/work_dirs/segrefiner_hr by HardDiskBackend.
2024-01-19 12:24:15,210 - mmdet - INFO - Iter [50/120000]	lr: 4.000e-04, eta: 7:41:07, time: 0.231, data_time: 0.005, memory: 6267, loss_mask: 0.0749, loss_texture: 0.0805, iou: 0.9283, loss: 0.1554
2024-01-19 12:24:25,180 - mmdet - INFO - Iter [100/120000]	lr: 4.000e-04, eta: 7:09:42, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.0736, loss_texture: 0.0857, iou: 0.9084, loss: 0.1592
2024-01-19 12:24:35,107 - mmdet - INFO - Iter [150/120000]	lr: 4.000e-04, eta: 6:58:32, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.4370, loss_texture: 0.2199, iou: 0.5518, loss: 0.6569
2024-01-19 12:24:45,140 - mmdet - INFO - Iter [200/120000]	lr: 4.000e-04, eta: 6:53:55, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.2852, loss_texture: 0.2145, iou: 0.5794, loss: 0.4996
2024-01-19 12:24:55,056 - mmdet - INFO - Iter [250/120000]	lr: 4.000e-04, eta: 6:50:09, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.2346, loss_texture: 0.2139, iou: 0.6228, loss: 0.4484
2024-01-19 12:25:05,039 - mmdet - INFO - Iter [300/120000]	lr: 4.000e-04, eta: 6:48:02, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2050, loss_texture: 0.1940, iou: 0.6370, loss: 0.3990
2024-01-19 12:25:14,964 - mmdet - INFO - Iter [350/120000]	lr: 4.000e-04, eta: 6:46:09, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.2655, loss_texture: 0.2266, iou: 0.6330, loss: 0.4921
2024-01-19 12:25:24,949 - mmdet - INFO - Iter [400/120000]	lr: 4.000e-04, eta: 6:44:59, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2196, loss_texture: 0.2139, iou: 0.7239, loss: 0.4336
2024-01-19 12:25:34,935 - mmdet - INFO - Iter [450/120000]	lr: 4.000e-04, eta: 6:44:03, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1660, loss_texture: 0.1727, iou: 0.7408, loss: 0.3387
2024-01-19 12:25:44,947 - mmdet - INFO - Iter [500/120000]	lr: 4.000e-04, eta: 6:43:22, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2036, loss_texture: 0.2130, iou: 0.7069, loss: 0.4167
2024-01-19 12:25:54,943 - mmdet - INFO - Iter [550/120000]	lr: 4.000e-04, eta: 6:42:44, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1658, loss_texture: 0.1909, iou: 0.7922, loss: 0.3566
2024-01-19 12:26:04,933 - mmdet - INFO - Iter [600/120000]	lr: 4.000e-04, eta: 6:42:09, time: 0.200, data_time: 0.004, memory: 6267, loss_mask: 0.1983, loss_texture: 0.2182, iou: 0.7171, loss: 0.4165
2024-01-19 12:26:14,901 - mmdet - INFO - Iter [650/120000]	lr: 4.000e-04, eta: 6:41:33, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1579, loss_texture: 0.1868, iou: 0.7118, loss: 0.3447
2024-01-19 12:26:24,843 - mmdet - INFO - Iter [700/120000]	lr: 4.000e-04, eta: 6:40:57, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2049, loss_texture: 0.2065, iou: 0.7138, loss: 0.4114
2024-01-19 12:26:34,789 - mmdet - INFO - Iter [750/120000]	lr: 4.000e-04, eta: 6:40:25, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2391, loss_texture: 0.2162, iou: 0.6966, loss: 0.4553
2024-01-19 12:26:44,700 - mmdet - INFO - Iter [800/120000]	lr: 4.000e-04, eta: 6:39:51, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.1903, loss_texture: 0.2007, iou: 0.6805, loss: 0.3910
2024-01-19 12:26:54,823 - mmdet - INFO - Iter [850/120000]	lr: 4.000e-04, eta: 6:39:49, time: 0.202, data_time: 0.006, memory: 6267, loss_mask: 0.2275, loss_texture: 0.2216, iou: 0.6903, loss: 0.4491
2024-01-19 12:27:04,774 - mmdet - INFO - Iter [900/120000]	lr: 4.000e-04, eta: 6:39:24, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2924, loss_texture: 0.2216, iou: 0.6570, loss: 0.5139
2024-01-19 12:27:14,760 - mmdet - INFO - Iter [950/120000]	lr: 4.000e-04, eta: 6:39:04, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2299, loss_texture: 0.2123, iou: 0.6476, loss: 0.4421
2024-01-19 12:27:24,758 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:27:24,758 - mmdet - INFO - Iter [1000/120000]	lr: 4.000e-04, eta: 6:38:47, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2042, loss_texture: 0.1917, iou: 0.6918, loss: 0.3960
2024-01-19 12:27:34,786 - mmdet - INFO - Iter [1050/120000]	lr: 4.000e-04, eta: 6:38:34, time: 0.201, data_time: 0.004, memory: 6267, loss_mask: 0.2077, loss_texture: 0.1856, iou: 0.7152, loss: 0.3933
2024-01-19 12:27:44,821 - mmdet - INFO - Iter [1100/120000]	lr: 4.000e-04, eta: 6:38:22, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1809, loss_texture: 0.1987, iou: 0.7236, loss: 0.3796
2024-01-19 12:27:54,808 - mmdet - INFO - Iter [1150/120000]	lr: 4.000e-04, eta: 6:38:06, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2191, loss_texture: 0.2069, iou: 0.7322, loss: 0.4260
2024-01-19 12:28:04,823 - mmdet - INFO - Iter [1200/120000]	lr: 4.000e-04, eta: 6:37:52, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2450, loss_texture: 0.2021, iou: 0.7341, loss: 0.4471
2024-01-19 12:28:14,884 - mmdet - INFO - Iter [1250/120000]	lr: 4.000e-04, eta: 6:37:43, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1971, loss_texture: 0.2114, iou: 0.7124, loss: 0.4085
2024-01-19 12:28:24,896 - mmdet - INFO - Iter [1300/120000]	lr: 4.000e-04, eta: 6:37:30, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1347, loss_texture: 0.1755, iou: 0.8245, loss: 0.3102
2024-01-19 12:28:35,061 - mmdet - INFO - Iter [1350/120000]	lr: 4.000e-04, eta: 6:37:30, time: 0.203, data_time: 0.006, memory: 6267, loss_mask: 0.2002, loss_texture: 0.1978, iou: 0.7008, loss: 0.3980
2024-01-19 12:28:45,045 - mmdet - INFO - Iter [1400/120000]	lr: 4.000e-04, eta: 6:37:15, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1808, loss_texture: 0.1898, iou: 0.7335, loss: 0.3705
2024-01-19 12:28:54,994 - mmdet - INFO - Iter [1450/120000]	lr: 4.000e-04, eta: 6:36:56, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1796, loss_texture: 0.1766, iou: 0.7720, loss: 0.3561
2024-01-19 12:29:04,995 - mmdet - INFO - Iter [1500/120000]	lr: 4.000e-04, eta: 6:36:43, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1564, loss_texture: 0.1758, iou: 0.7233, loss: 0.3321
2024-01-19 12:29:14,931 - mmdet - INFO - Iter [1550/120000]	lr: 4.000e-04, eta: 6:36:25, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1863, loss_texture: 0.2102, iou: 0.7394, loss: 0.3965
2024-01-19 12:29:25,060 - mmdet - INFO - Iter [1600/120000]	lr: 4.000e-04, eta: 6:36:21, time: 0.203, data_time: 0.005, memory: 6267, loss_mask: 0.2034, loss_texture: 0.2099, iou: 0.7317, loss: 0.4134
2024-01-19 12:29:35,074 - mmdet - INFO - Iter [1650/120000]	lr: 4.000e-04, eta: 6:36:09, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1500, loss_texture: 0.1764, iou: 0.7995, loss: 0.3263
2024-01-19 12:29:45,054 - mmdet - INFO - Iter [1700/120000]	lr: 4.000e-04, eta: 6:35:55, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1477, loss_texture: 0.1792, iou: 0.7476, loss: 0.3270
2024-01-19 12:29:55,016 - mmdet - INFO - Iter [1750/120000]	lr: 4.000e-04, eta: 6:35:39, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1770, loss_texture: 0.1875, iou: 0.7434, loss: 0.3645
2024-01-19 12:30:04,943 - mmdet - INFO - Iter [1800/120000]	lr: 4.000e-04, eta: 6:35:22, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1464, loss_texture: 0.1856, iou: 0.7543, loss: 0.3320
2024-01-19 12:30:14,854 - mmdet - INFO - Iter [1850/120000]	lr: 4.000e-04, eta: 6:35:04, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.1349, loss_texture: 0.1745, iou: 0.7693, loss: 0.3094
2024-01-19 12:30:24,803 - mmdet - INFO - Iter [1900/120000]	lr: 4.000e-04, eta: 6:34:49, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1340, loss_texture: 0.1688, iou: 0.8084, loss: 0.3028
2024-01-19 12:30:34,767 - mmdet - INFO - Iter [1950/120000]	lr: 4.000e-04, eta: 6:34:35, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1249, loss_texture: 0.1673, iou: 0.7941, loss: 0.2922
2024-01-19 12:30:44,694 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:30:44,694 - mmdet - INFO - Iter [2000/120000]	lr: 4.000e-04, eta: 6:34:19, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2065, loss_texture: 0.1790, iou: 0.7811, loss: 0.3856
2024-01-19 12:30:54,604 - mmdet - INFO - Iter [2050/120000]	lr: 4.000e-04, eta: 6:34:02, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.1713, loss_texture: 0.2143, iou: 0.7463, loss: 0.3856
2024-01-19 12:31:04,618 - mmdet - INFO - Iter [2100/120000]	lr: 4.000e-04, eta: 6:33:52, time: 0.200, data_time: 0.004, memory: 6267, loss_mask: 0.2016, loss_texture: 0.2008, iou: 0.7280, loss: 0.4023
2024-01-19 12:31:14,605 - mmdet - INFO - Iter [2150/120000]	lr: 4.000e-04, eta: 6:33:40, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1621, loss_texture: 0.2051, iou: 0.7716, loss: 0.3672
2024-01-19 12:31:24,538 - mmdet - INFO - Iter [2200/120000]	lr: 4.000e-04, eta: 6:33:25, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1611, loss_texture: 0.1994, iou: 0.8068, loss: 0.3604
2024-01-19 12:31:34,484 - mmdet - INFO - Iter [2250/120000]	lr: 4.000e-04, eta: 6:33:11, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1697, loss_texture: 0.2121, iou: 0.6758, loss: 0.3818
2024-01-19 12:31:44,713 - mmdet - INFO - Iter [2300/120000]	lr: 4.000e-04, eta: 6:33:12, time: 0.205, data_time: 0.008, memory: 6267, loss_mask: 0.1664, loss_texture: 0.1805, iou: 0.7210, loss: 0.3469
2024-01-19 12:31:54,648 - mmdet - INFO - Iter [2350/120000]	lr: 4.000e-04, eta: 6:32:57, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1260, loss_texture: 0.1762, iou: 0.7685, loss: 0.3021
2024-01-19 12:32:04,631 - mmdet - INFO - Iter [2400/120000]	lr: 4.000e-04, eta: 6:32:45, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1444, loss_texture: 0.1815, iou: 0.8099, loss: 0.3260
2024-01-19 12:32:15,601 - mmdet - INFO - Iter [2450/120000]	lr: 4.000e-04, eta: 6:33:21, time: 0.219, data_time: 0.021, memory: 6267, loss_mask: 0.1563, loss_texture: 0.1941, iou: 0.7882, loss: 0.3505
2024-01-19 12:32:25,657 - mmdet - INFO - Iter [2500/120000]	lr: 4.000e-04, eta: 6:33:12, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1376, loss_texture: 0.1658, iou: 0.7853, loss: 0.3034
2024-01-19 12:32:35,804 - mmdet - INFO - Iter [2550/120000]	lr: 4.000e-04, eta: 6:33:07, time: 0.203, data_time: 0.003, memory: 6267, loss_mask: 0.1314, loss_texture: 0.1676, iou: 0.8216, loss: 0.2991
2024-01-19 12:32:45,851 - mmdet - INFO - Iter [2600/120000]	lr: 4.000e-04, eta: 6:32:57, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1733, loss_texture: 0.1933, iou: 0.7472, loss: 0.3666
2024-01-19 12:32:55,850 - mmdet - INFO - Iter [2650/120000]	lr: 4.000e-04, eta: 6:32:45, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1992, loss_texture: 0.2001, iou: 0.6920, loss: 0.3993
2024-01-19 12:33:05,933 - mmdet - INFO - Iter [2700/120000]	lr: 4.000e-04, eta: 6:32:37, time: 0.202, data_time: 0.003, memory: 6267, loss_mask: 0.1481, loss_texture: 0.1848, iou: 0.7880, loss: 0.3328
2024-01-19 12:33:15,942 - mmdet - INFO - Iter [2750/120000]	lr: 4.000e-04, eta: 6:32:25, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1818, loss_texture: 0.1817, iou: 0.7667, loss: 0.3635
2024-01-19 12:33:25,938 - mmdet - INFO - Iter [2800/120000]	lr: 4.000e-04, eta: 6:32:13, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1897, loss_texture: 0.1949, iou: 0.7399, loss: 0.3846
2024-01-19 12:33:36,020 - mmdet - INFO - Iter [2850/120000]	lr: 4.000e-04, eta: 6:32:05, time: 0.202, data_time: 0.003, memory: 6267, loss_mask: 0.1986, loss_texture: 0.1722, iou: 0.7496, loss: 0.3707
2024-01-19 12:33:46,018 - mmdet - INFO - Iter [2900/120000]	lr: 4.000e-04, eta: 6:31:53, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1509, loss_texture: 0.1899, iou: 0.7735, loss: 0.3408
2024-01-19 12:33:56,028 - mmdet - INFO - Iter [2950/120000]	lr: 4.000e-04, eta: 6:31:42, time: 0.200, data_time: 0.004, memory: 6267, loss_mask: 0.1364, loss_texture: 0.1691, iou: 0.7962, loss: 0.3055
2024-01-19 12:34:06,055 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:34:06,055 - mmdet - INFO - Iter [3000/120000]	lr: 4.000e-04, eta: 6:31:31, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1484, loss_texture: 0.1878, iou: 0.7773, loss: 0.3362
2024-01-19 12:34:16,073 - mmdet - INFO - Iter [3050/120000]	lr: 4.000e-04, eta: 6:31:21, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1766, loss_texture: 0.2010, iou: 0.7864, loss: 0.3776
2024-01-19 12:34:26,081 - mmdet - INFO - Iter [3100/120000]	lr: 4.000e-04, eta: 6:31:09, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1475, loss_texture: 0.1973, iou: 0.7756, loss: 0.3448
2024-01-19 12:34:36,077 - mmdet - INFO - Iter [3150/120000]	lr: 4.000e-04, eta: 6:30:58, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1222, loss_texture: 0.1656, iou: 0.8325, loss: 0.2878
2024-01-19 12:34:46,095 - mmdet - INFO - Iter [3200/120000]	lr: 4.000e-04, eta: 6:30:47, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1147, loss_texture: 0.1624, iou: 0.7541, loss: 0.2771
2024-01-19 12:34:56,057 - mmdet - INFO - Iter [3250/120000]	lr: 4.000e-04, eta: 6:30:34, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1926, loss_texture: 0.1761, iou: 0.7865, loss: 0.3687
2024-01-19 12:35:06,074 - mmdet - INFO - Iter [3300/120000]	lr: 4.000e-04, eta: 6:30:23, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1360, loss_texture: 0.1826, iou: 0.7763, loss: 0.3186
2024-01-19 12:35:16,077 - mmdet - INFO - Iter [3350/120000]	lr: 4.000e-04, eta: 6:30:12, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.3127, loss_texture: 0.1954, iou: 0.6610, loss: 0.5081
2024-01-19 12:35:26,089 - mmdet - INFO - Iter [3400/120000]	lr: 4.000e-04, eta: 6:30:01, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2593, loss_texture: 0.1952, iou: 0.6893, loss: 0.4545
2024-01-19 12:35:36,096 - mmdet - INFO - Iter [3450/120000]	lr: 4.000e-04, eta: 6:29:50, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1894, loss_texture: 0.2158, iou: 0.7296, loss: 0.4052
2024-01-19 12:35:46,117 - mmdet - INFO - Iter [3500/120000]	lr: 4.000e-04, eta: 6:29:40, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2522, loss_texture: 0.2290, iou: 0.6647, loss: 0.4813
2024-01-19 12:35:56,152 - mmdet - INFO - Iter [3550/120000]	lr: 4.000e-04, eta: 6:29:30, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1732, loss_texture: 0.1959, iou: 0.7273, loss: 0.3691
2024-01-19 12:36:06,232 - mmdet - INFO - Iter [3600/120000]	lr: 4.000e-04, eta: 6:29:21, time: 0.202, data_time: 0.003, memory: 6267, loss_mask: 0.1718, loss_texture: 0.1937, iou: 0.7334, loss: 0.3655
2024-01-19 12:36:16,231 - mmdet - INFO - Iter [3650/120000]	lr: 4.000e-04, eta: 6:29:10, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1612, loss_texture: 0.1986, iou: 0.7909, loss: 0.3598
2024-01-19 12:36:26,350 - mmdet - INFO - Iter [3700/120000]	lr: 4.000e-04, eta: 6:29:03, time: 0.202, data_time: 0.006, memory: 6267, loss_mask: 0.1699, loss_texture: 0.1903, iou: 0.7482, loss: 0.3603
2024-01-19 12:36:36,318 - mmdet - INFO - Iter [3750/120000]	lr: 4.000e-04, eta: 6:28:51, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1348, loss_texture: 0.1824, iou: 0.7632, loss: 0.3172
2024-01-19 12:36:47,253 - mmdet - INFO - Iter [3800/120000]	lr: 4.000e-04, eta: 6:29:08, time: 0.219, data_time: 0.022, memory: 6267, loss_mask: 0.2047, loss_texture: 0.2051, iou: 0.7482, loss: 0.4098
2024-01-19 12:36:57,272 - mmdet - INFO - Iter [3850/120000]	lr: 4.000e-04, eta: 6:28:57, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1410, loss_texture: 0.2001, iou: 0.7394, loss: 0.3412
2024-01-19 12:37:07,299 - mmdet - INFO - Iter [3900/120000]	lr: 4.000e-04, eta: 6:28:47, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1435, loss_texture: 0.1750, iou: 0.7415, loss: 0.3185
2024-01-19 12:37:17,327 - mmdet - INFO - Iter [3950/120000]	lr: 4.000e-04, eta: 6:28:36, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1412, loss_texture: 0.1831, iou: 0.7556, loss: 0.3242
2024-01-19 12:37:27,355 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:37:27,356 - mmdet - INFO - Iter [4000/120000]	lr: 4.000e-04, eta: 6:28:25, time: 0.201, data_time: 0.004, memory: 6267, loss_mask: 0.1276, loss_texture: 0.1654, iou: 0.7787, loss: 0.2930
2024-01-19 12:37:37,406 - mmdet - INFO - Iter [4050/120000]	lr: 4.000e-04, eta: 6:28:16, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1453, loss_texture: 0.1862, iou: 0.7722, loss: 0.3315
2024-01-19 12:37:47,268 - mmdet - INFO - Iter [4100/120000]	lr: 4.000e-04, eta: 6:28:00, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.5625, loss_texture: 0.2051, iou: 0.1537, loss: 0.7676
2024-01-19 12:37:57,120 - mmdet - INFO - Iter [4150/120000]	lr: 4.000e-04, eta: 6:27:45, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6593, loss_texture: 0.2000, iou: 0.0000, loss: 0.8593
2024-01-19 12:38:06,906 - mmdet - INFO - Iter [4200/120000]	lr: 4.000e-04, eta: 6:27:28, time: 0.196, data_time: 0.003, memory: 6267, loss_mask: 0.6285, loss_texture: 0.2061, iou: 0.0000, loss: 0.8345
2024-01-19 12:38:16,736 - mmdet - INFO - Iter [4250/120000]	lr: 4.000e-04, eta: 6:27:12, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6399, loss_texture: 0.1972, iou: 0.0000, loss: 0.8372
2024-01-19 12:38:26,524 - mmdet - INFO - Iter [4300/120000]	lr: 4.000e-04, eta: 6:26:55, time: 0.196, data_time: 0.003, memory: 6267, loss_mask: 0.6509, loss_texture: 0.1880, iou: 0.0000, loss: 0.8390
2024-01-19 12:38:36,357 - mmdet - INFO - Iter [4350/120000]	lr: 4.000e-04, eta: 6:26:40, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6114, loss_texture: 0.2054, iou: 0.0000, loss: 0.8168
2024-01-19 12:38:46,185 - mmdet - INFO - Iter [4400/120000]	lr: 4.000e-04, eta: 6:26:25, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6601, loss_texture: 0.1975, iou: 0.0000, loss: 0.8576
2024-01-19 12:38:55,983 - mmdet - INFO - Iter [4450/120000]	lr: 4.000e-04, eta: 6:26:09, time: 0.196, data_time: 0.003, memory: 6267, loss_mask: 0.6781, loss_texture: 0.2102, iou: 0.0000, loss: 0.8882
2024-01-19 12:39:06,238 - mmdet - INFO - Iter [4500/120000]	lr: 4.000e-04, eta: 6:26:04, time: 0.205, data_time: 0.012, memory: 6267, loss_mask: 0.6450, loss_texture: 0.1818, iou: 0.0000, loss: 0.8268
2024-01-19 12:39:16,085 - mmdet - INFO - Iter [4550/120000]	lr: 4.000e-04, eta: 6:25:50, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6331, loss_texture: 0.2153, iou: 0.0000, loss: 0.8485
2024-01-19 12:39:26,005 - mmdet - INFO - Iter [4600/120000]	lr: 4.000e-04, eta: 6:25:37, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.6359, loss_texture: 0.2007, iou: 0.0000, loss: 0.8366
2024-01-19 12:39:35,871 - mmdet - INFO - Iter [4650/120000]	lr: 4.000e-04, eta: 6:25:23, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6284, loss_texture: 0.1774, iou: 0.0000, loss: 0.8058
2024-01-19 12:39:45,796 - mmdet - INFO - Iter [4700/120000]	lr: 4.000e-04, eta: 6:25:11, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.6485, loss_texture: 0.2023, iou: 0.0000, loss: 0.8507
2024-01-19 12:39:55,629 - mmdet - INFO - Iter [4750/120000]	lr: 4.000e-04, eta: 6:24:56, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6554, loss_texture: 0.1779, iou: 0.0000, loss: 0.8333
2024-01-19 12:40:05,472 - mmdet - INFO - Iter [4800/120000]	lr: 4.000e-04, eta: 6:24:42, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6776, loss_texture: 0.2007, iou: 0.0000, loss: 0.8783

Hello, I also encountered the same problem, may I ask you to solve it？

@yusuke-ai
Copy link

@MengyuWang826
Hi! Did you found out if there's the issue in the training process?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@xiaolul2 @lixiang007666 @Taocunguo @MengyuWang826 @yusuke-ai