New save_strategy
option called "best" to save when a new best performance is achieved.
#31626
Labels
Feature request
Request for a new feature
Feature request
Introduce a new option for the
save_strategy
argument called"best"
which would save the model once a new best performance is achieved.Motivation
The
save_strategy
argument was first introduced a few years ago in #10286. Currently the supported options are"no"
,"epoch"
, and"steps"
. I'm assuming that this is to match theIntervalStrategy
that's used by evaluation as well.Judging by a conversation on a HuggingFace Discussion Forum topic, the best model is always kept by default and therefore if saving occurs at any time during the process then the best model is saved (ref: https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442). If the user deems that saving often is too burdensome then they may set
save_strategy = "no"
which would save the best model at the end of training.I believe that introducing some flexibility for saving would be beneficial so that users don't have to perform saving so often but also don't have to wait until the end for a checkpoint.
Your contribution
If this feature is deemed worth it by the core maintainers then I'd be willing to take this on myself and open a PR. There are some aspects that I believe might warrant further discussion (e.g., which metric should be used to determine "best," how to override
IntervalStrategy
, etc.).The text was updated successfully, but these errors were encountered: