Optimizing GPU and CPU usage

[alexracape]

The metrics from weights and biases show:
GPU usage = 55%
CPU usage = 2.5%
CPU per core usage varies but all are well below 100%

I think this must be due to an I/O where the workers need to wait for the GPU / main process to handle each request.

[alexracape]

Issue is the waiting during the queue and repeatedly loading small batches onto the GPU.
Next, need to time how long it takes for a single batch to be processed. Larger batches could help, but then the processes need to wait longer. Look into if there are papers on this buffering problem