Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem when using run_pt.py #4

Open
Melonoyk opened this issue Apr 23, 2024 · 3 comments
Open

Problem when using run_pt.py #4

Melonoyk opened this issue Apr 23, 2024 · 3 comments

Comments

@Melonoyk
Copy link

I met a problem when i testing the pre-train part of your code. I use: bash script/run_pt.sh. to follow your Start Training part in README, and find that the process is blocked at the Epoch 1 /100. Eventually the process will be forcibly killed. I also tried to interrupt the process and found that it stuck at reading the length of dataloader. I wonder if this is due to hardware requirements that don't support pre-training(using RTX 3090), and looking forward to your reply very much.

The output is as follow:
image

image

@tyang816
Copy link
Owner

Hi, Ruiwen,
Sorry for the late reply.
I have tested the pre-train code just now (RTX 3090), but I didn't meet any wrong, can you provide more information?
Thx
image

@Melonoyk
Copy link
Author

Hi, Yang,
Some of the packages i use are not in the same version mentioned in enviroment.yaml which may cause the occurrence of this issue. When i using: conda env create -f environment.yaml, this process will shutdown in the middle. I fix this problem by rewriting the BatchSampler function, cuz i found that the loading of dataset is stuck at loading the first data of dataset into sampler while i dont konw how this issue occurs. At last, thank u for your testing!

image

@tyang816
Copy link
Owner

Great! I will check the environment file soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants