Training Time

[Huster-Hq]

Hi, do you only include 'main_training part' in the '30 hours' here?

Please ask the total number of frames in your training data set, as well as your total number of training iterations is 125000 and batch_size is 16? (followed your train_config.yaml)

[hkchengrex]

It is for the entire training process.

I don't have the total number of training frames off the top of my head, but you can just count from the training dataset.

The default configuration in the code is correct.

[Huster-Hq]

Thanks!

I currently face a problem: the training time is too long when I run train.py on our custom dataset (two 3090 GPUs).

My custom dataset: 19544 frames in total, and I set the 'iteration=125000', and 'batch_size=14'.

The following figures are my training record and GPU. Is this reasonable?

[hkchengrex]

1. The first printed "time" is not reliable as it includes warm-up and initialization.
2. In general, we want a high GPU utilization (see Training low perf #58 (comment)) -- your GPUs are sitting at 18C and 29C so I don't think they are working hard enough unless you have really good cooling. This usually indicates a bottleneck in CPU/disk IO.

[Huster-Hq]

At the later 'time', the 'avg_time' is still 12-13.

[hkchengrex]

Your CPUs and GPUs are not being used as much as they should be. Potentially related: pytorch/pytorch#99625

[hkchengrex]

Please feel free to re-open if there are any updates.