Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Small tidying #1222

Merged
merged 5 commits into from
May 21, 2024
Merged

Small tidying #1222

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
17b061f
Tolerate no fused kernels
yang May 17, 2024
fdc395f
Fix requirements file syntax
yang May 17, 2024
ae7e849
Update NeoXArgs docs automatically
invalid-email-address May 17, 2024
1b85a2f
Merge branch 'main' into tidy
Quentin-Anthony May 21, 2024
0b61f53
Update NeoXArgs docs automatically
invalid-email-address May 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion configs/neox_arguments.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ Logging Arguments

- **git_hash**: str

Default = b68ba6d
Default = 1b85a2f

current git hash of repository

Expand Down
7 changes: 5 additions & 2 deletions megatron/model/norms.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@

import torch
from torch.nn import LayerNorm as LayerNorm
from .fused_layer_norm import MixedFusedLayerNorm


def get_norm(neox_args):
Expand All @@ -23,7 +22,11 @@ def get_norm(neox_args):
eps = neox_args.rms_norm_epsilon
elif neox_args.norm == "layernorm":
eps = neox_args.layernorm_epsilon
norm = MixedFusedLayerNorm if neox_args.layernorm_fusion else LayerNorm
if neox_args.layernorm_fusion:
from .fused_layer_norm import MixedFusedLayerNorm
norm = MixedFusedLayerNorm
else:
norm = LayerNorm
elif neox_args.norm == "scalenorm":
eps = neox_args.scalenorm_epsilon
norm = ScaleNorm
Expand Down
4 changes: 2 additions & 2 deletions requirements/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
git+https://github.com/EleutherAI/DeeperSpeed.git@02e2ebf7dee6aaab3d89094ed470a4609763c742#egg=deepspeed
deepspeed@git+https://github.com/EleutherAI/DeeperSpeed.git@02e2ebf7dee6aaab3d89094ed470a4609763c742#egg=deepspeed
ftfy>=6.0.1
git+https://github.com/EleutherAI/lm_dataformat.git@4eec05349977071bf67fc072290b95e31c8dd836
lm_dataformat@git+https://github.com/EleutherAI/lm_dataformat.git@4eec05349977071bf67fc072290b95e31c8dd836
huggingface_hub>=0.11.0
jinja2==3.1.4
lm_eval>=0.4.0,<=0.4.1
Expand Down