Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: Computation of Total Number of Parameters #186

Merged
merged 6 commits into from
Jul 15, 2024
Merged

Conversation

mali-git
Copy link
Member

@mali-git mali-git commented Jul 15, 2024

What does this PR do?

This PR updates the computation of the total number parameters in a (sharded) model.
Previously, the parameter count only reflected the number of parameters in a single shard wher the model was sharded.

General Changes

  • All ranks collectively reduce the total number of parameters.

Breaking Changes

  • None

Checklist before submitting final PR

  • My PR is minimal and addresses one issue in isolation
  • I have merged the latest version of the target branch into this feature branch
  • I have reviewed my own code w.r.t. correct implementation, missing type hints, proper documentation, etc.
  • I have run a sample config for model training
  • I have checked that all tests run through (python tests/tests.py)

@mali-git mali-git requested a review from le1nux July 15, 2024 09:36
Copy link
Member

@le1nux le1nux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :)

@mali-git mali-git merged commit 8e79acc into main Jul 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants