You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What if I want to use fp16 true, but with a loss scaler? This is closer to DeepSpeed's default settings. With FSDP, 16-true, no loss scaler my model doesn't converge. However, with FSDP, 16-true, and a loss scaler (commented out the assert and fixed the typo'ed return scaler instead of return none line) my model converges.
I came here to open this issue, and you already did.
I second this issue.
I fixed the package itself by adding
if scaler is not None and self.precision not in ["16-mixed", "16-true"]:
raise ValueError(f"`precision={precision!r}` does not use a scaler, found {scaler}.")
but it has to be fixed naturally.
zaptrem
changed the title
FSDPPrecision should support 16-true with a loss scalar
FSDPPrecision should support 16-true with a loss scaler
Jun 13, 2024
Description & Motivation
pytorch-lightning/src/lightning/fabric/plugins/precision/fsdp.py
Line 61 in f6fd046
What if I want to use fp16 true, but with a loss scaler? This is closer to DeepSpeed's default settings. With FSDP, 16-true, no loss scaler my model doesn't converge. However, with FSDP, 16-true, and a loss scaler (commented out the assert and fixed the typo'ed return scaler instead of return none line) my model converges.
Pitch
No response
Alternatives
No response
Additional context
No response
cc @Borda
The text was updated successfully, but these errors were encountered: