[Solved?]Training is taking too long

[Sxwako]

What we want to solve

1. the time it seems to take for training is too long. I would like to train faster.
2. should I change my settings to do this or should I use Google Colab?

When training, I noticed that Epoch was progressing very slowly.
My machine was performing reasonably well and I looked in the task manager and found the following

However, it is taking on average about 3 minutes to finish 1Epoch. At this rate, it would take almost 20 days to finish 9999Epoch.
In this case, should I use Google Colab?
By the way, the detaset folder contains only 89 audio files.

My environment is as follows

CPU:Ryzen9-5900HS
RAM:32GB
GPU:RTX1080(laptop)

Python:3.8
numpy1.23

I am waiting for an answer.

[jiaxinf]

Simply put, I think you are likely running the project on CPU rather than GPU. I had this issue as well and discovered a solution here in another discussion #779 (comment)
After I switch to GPU training, it takes <1min to complete 1 epoch for me.

[Sxwako]

When I checked, it was not using the GPU, so I reinstalled pytorch, which allowed the GPU to be used, but did not change the speed nearly as much.

[nikich340]

You have 32 GB of RAM, but 42 GB used on screenshot?
It seems that your OS may use swap file on HDD to extend RAM, while it's quite slow.
So loading and unloading data to swap may be the bottleneck. Try to disable swap file and reduce num_workers in config.json if your real RAM is not enough.
Also, you have only 6 iterations per epoch - are you using few big audios? Make sure you have sliced them into pieces of 10s duration (max).

[Sxwako]

I changed the setting to not use virtual memory, but on the contrary, the speed was reduced (´;ω;｀).
I also checked all wav files and found that they were generally cut to about 10 seconds.

[nikich340]

Check in python if torch.cuda.is_available() returns True.

[Sxwako]

In the end, speed was greatly improved by using paperspace.
After using it, we were able to finish in 4~5 seconds per Epoch. Great! (^▽^)/