Skip to content

Commit

Permalink
fix dilocoo
Browse files Browse the repository at this point in the history
  • Loading branch information
samsja committed Oct 3, 2024
1 parent 3684133 commit ba51a7f
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/zeroband/checkpoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ def load(self, resume_ckpt_path: str) -> None:
dcp.load(self.states, checkpoint_id=resume_ckpt_path)
# since we don't load the param list from the state dict as its the same as the model one we just copy
if self.diloco_offloaded_param_list is not None:
for param_offloaded, param_model in zip(self.diloco_offloaded_param_list, self.model.model.parameters()):
for param_offloaded, param_model in zip(self.diloco_offloaded_param_list, self.model.parameters()):
param_offloaded.data.copy_(param_model.data)

## the next part is a fix so that each rank save a different dataloader rank. It not efficient because it reads the state two times from disk
Expand Down

0 comments on commit ba51a7f

Please sign in to comment.