How to save best checkpoints for more than 1 metric

[ChenJunzhi-buaa]

Is is only able to save best checkpoints for 1 metric?

[Ezra-Yu]

Yes, you can save the best checkpoints for more than 1 metric. save_best can accept a list.

default_hooks = dict(
    ......
    checkpoint=dict(type='CheckpointHook', interval=1,  save_best=['acc', 'mIoU'], rule='greater', max_keep_ckpts=3),
    ......
)

For more detail, you can refer to mmengine.hooks.CheckpointHook.

[ChenJunzhi-buaa]

write this in config file? I didn't succes.
I also write configs like below, which fails too.

Can you give more specific instructions please？
Thank you！

[Ezra-Yu]

Sorry to reply to you so late. I see you are on the master branch, the current master branch, the master branch needs to be viewed at https://github.com/open-mmlab/mmcv/blob/48ea88ab9f3f8eb1e799d1d711da0181d95e574d/mmcv/runner /hooks/evaluation.py#L86 source code. The example I gave is how to use the 1.x branch.

What does your error reporting look like?

[ChenJunzhi-buaa]

    assert isinstance(save_best, str) or save_best is None, \
AssertionError: ""save_best"" should be a str or None rather than <class 'list'>
    assert isinstance(save_best, str) or save_best is None, \
AssertionError: ""save_best"" should be a str or None rather than <class 'list'>
    assert isinstance(save_best, str) or save_best is None, \
    assert isinstance(save_best, str) or save_best is None, \
AssertionError: ""save_best"" should be a str or None rather than <class 'list'>
AssertionError: ""save_best"" should be a str or None rather than <class 'list'>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 5651) of binary: /home/chenjunzhi/miniconda3/envs/open-mmlab/bin/python3
Traceback (most recent call last):
  File "/home/chenjunzhi/miniconda3/envs/open-mmlab/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/chenjunzhi/miniconda3/envs/open-mmlab/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/chenjunzhi/miniconda3/envs/open-mmlab/lib/python3.10/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/home/chenjunzhi/miniconda3/envs/open-mmlab/lib/python3.10/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/home/chenjunzhi/miniconda3/envs/open-mmlab/lib/python3.10/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/home/chenjunzhi/miniconda3/envs/open-mmlab/lib/python3.10/site-packages/torch/distributed/run.py", line 752, in run
    elastic_launch(
  File "/home/chenjunzhi/miniconda3/envs/open-mmlab/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/chenjunzhi/miniconda3/envs/open-mmlab/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
tools/train.py FAILED

[Ezra-Yu]

save_best should be an str (a metric), refer to https://github.com/open-mmlab/mmcv/blob/48ea88ab9f3f8eb1e799d1d711da0181d95e574d/mmcv/runner/hooks/evaluation.py#L34.

Sorry that save best checkpoints for more than 1 metric do not support in the master brach.

[ChenJunzhi-buaa]

ok， thank you!