-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue with SliRec and DKN OOM #2076
Conversation
Signed-off-by: miguelgfierro <[email protected]>
Signed-off-by: miguelgfierro <[email protected]>
Signed-off-by: miguelgfierro <[email protected]>
Signed-off-by: miguelgfierro <[email protected]>
Signed-off-by: miguelgfierro <[email protected]>
If I comment all tests except SLiRec, then it works, and it is quick:
See https://github.com/recommenders-team/recommenders/actions/runs/8422556385/job/23062115526 It works with:
https://github.com/recommenders-team/recommenders/actions/runs/8424096579 If I have lightgcn, SLiRec and DKN 4a484e8, I get an OOM https://github.com/recommenders-team/recommenders/actions/runs/8427004517/job/23076600230#step:3:8023 |
Signed-off-by: miguelgfierro <[email protected]>
Signed-off-by: miguelgfierro <[email protected]>
Signed-off-by: miguelgfierro <[email protected]>
Signed-off-by: miguelgfierro <[email protected]>
Signed-off-by: miguelgfierro <[email protected]>
Signed-off-by: miguelgfierro <[email protected]>
Signed-off-by: miguelgfierro <[email protected]>
@SimonYansenZhao feel free to continue in this branch #2076 The problem I'm seeing is that each test for some reason keeps the memory so the next test get an OOM. I tried |
Yeah, I ran each tests individually today, and all passed. I'll try to run the three tests together. |
@miguelgfierro all tests including nightly builds passed after #2077 was merged. Not sure what's going on with the OOM issue. But I think we can merge staging into main now. |
@SimonYansenZhao I´m freaking out. |
Let me run again the nightlys |
Description
Fix #2063
I run both tests in a local V100 and they pass. In the nightly tests I get: https://github.com/recommenders-team/recommenders/actions/workflows/azureml-gpu-nightly.yml?query=branch%3Amiguel%2Fnightly_oom
The DKN test is already with 5 epochs and BS=200. I don´t know if both tests are conflicting because TF takes the whole memory of the GPU by default.
This shouldn't happen because in the BaseModel class we have
gpu_options = tf.compat.v1.GPUOptions(allow_growth=True)
. However, when I run the code in local, I see TF getting all the memory.Related Issues
References
Checklist:
git commit -s -m "your commit message"
.staging branch
AND NOT TOmain branch
.