Training on CPU produces division by zero error when computing progress value to display.

[sztejkat]

Describe the bug

During training when progress is slow following error is reported:

    (....)
      File "***removed****/modules/training.py", line 717, in do_train
        timer_info = f"`{1.0/its:.2f}` s/it"
                         ~~~^~~~
    ZeroDivisionError: float division by zero
    (...)

Code inspection shows that it isn't immune to zero value.

As a work around I changed in above file lines like following:

            line 713: in its = tracked.current_steps / time_elapsed
                            if its > 0:
                               if its > 1:
                                   timer_info = f"`{its:.2f}` it/s"
                               else:
                                    timer_info = f"`{1.0/its:.2f}` s/it"
            
                               total_time_estimate = (1.0 / its) * (tracked.max_steps)     

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

Hard to do, depends on hardware, software setup, system load and etc.

Screenshot

No response

Logs

See description.

System Info

Commit hash: 4d466d5c80eb83892b7dfb76fa4ab69efd6d6989
version tag: 2.0

Os: Linux, some old ubuntu distro. CPU only.