A way to continue training

[Garshishka]

Considering that this program can use CPU and low VRam cards to train, how about adding a way or parameter to continue training from saved splat.ply? Is this even feasible?

[pierotofy]

I don't see why not.

1. Modify savePly to store the current step count (in a comment PLY header value, maybe)
2. Read PLY back into the tensors (reverse of savePly), read step count.
3. Resume from the previous step count.

For a numerically correct resume, one should also dump the optimizer state but I don't think that would actually matter too much for the end result.

We'd welcome a pull request for this. Interested?

[Garshishka]

I would if I could :(
But cpp and ML are an unknown to me

[stefvfx]

I think it would be very useful.

[Itox001]

+1 for this feature. Currently I can only reasonably train ~3000 iterations before RAM consumption exhausts my resources because of the memory leak on MPS devices. I am hoping that stopping and resuming the training would reset this, allowing me to train for longer.

[eloquentarduino]

+1. I'm not a C++ guy so I can't help here.

[AsherJingkongChen]

+1 for this feature. Currently I can only reasonably train ~3000 iterations before RAM consumption exhausts my resources because of the memory leak on MPS devices. I am hoping that stopping and resuming the training would reset this, allowing me to train for longer.

So sad. I hope there would be a solution for you 😊.

[elliotmarks06]

I would also love to see this added!