Autoresume Validation with Max Duration

[mvpatel2000]

What does this PR do?

Additional checks based on #3357:

• Autoresume requires max_duration be specified on init and cannot accept duration in fit. Otherwise, Composer will autoresume from last checkpoint, and then the specified value of max_duration from fit will be incremented to the loaded max duration. In short, fit(duration=...) is not idempotent with autoresume.
• reset_time in fit does not work with autoresume as it will force a restart in training. Instead, time should be ignored on load, which is used in initial load and not on autoresume load
• Implicitly prevents multiple calls to fit -- if you cannot increment duration in fit or reset time, it is not possible to call fit twice.

[eracah]

Do we want to keep the incremented functionality? If I load in a 10ba checkpoint with Trainer(..., max_duration='15ba') and I say fit(duration='12ba'), wouldn't I want it to train for 2 batches instead of 5. Wouldn't what I specify in fit override what I did in the constructor?

[mvpatel2000]

Do we want to keep the incremented functionality? If I load in a 10ba checkpoint with Trainer(..., max_duration='15ba') and I say fit(duration='12ba'), wouldn't I want it to train for 2 batches instead of 5. Wouldn't what I specify in fit override what I did in the constructor?

Iirc the original behavior was modeled after what other training libraries do... but I agree it's perhaps not ideal. We should revisit this in a follow-on PR

[eracah]

Do we want to keep the incremented functionality? If I load in a 10ba checkpoint with Trainer(..., max_duration='15ba') and I say fit(duration='12ba'), wouldn't I want it to train for 2 batches instead of 5. Wouldn't what I specify in fit override what I did in the constructor?

Iirc the original behavior was modeled after what other training libraries do... but I agree it's perhaps not ideal. We should revisit this in a follow-on PR

yeah we should this is very counter-intuitive

[mvpatel2000]

Do we want to keep the incremented functionality? If I load in a 10ba checkpoint with Trainer(..., max_duration='15ba') and I say fit(duration='12ba'), wouldn't I want it to train for 2 batches instead of 5. Wouldn't what I specify in fit override what I did in the constructor?

@antoinebrl curious if you have any input given your recent github issue referencing multiple fit calls

[antoinebrl]

I must say this was a very counter intuitive bug to track down. This PR implements some fences which push the user towards the expected behavior but I agree the notion of training duration can be improved.

Regarding the multiple calls to .fit, we currently don't have that use-case. However, I remember reading a LLM paper where they do 2.1 epochs: 2 epochs with a very large dataset, and 0.1 for fine-tuning on a high quality dataset tailored for the task.