Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add other Norm to choose from #9

Open
notprime opened this issue Feb 23, 2024 · 0 comments
Open

Add other Norm to choose from #9

notprime opened this issue Feb 23, 2024 · 0 comments

Comments

@notprime
Copy link

smp only used BatchNorm as normalization layer.
We should add the option of choosing other normalization layers, namely:

  • GroupNorm
  • LayerNorm
  • InstanceNorm

and their variations.

One small problem arise: while BatchNorm and InstanceNorm require parameters which can be easily inferred from the specific used architecture, GroupNorm and LayerNorm requires specific parameters to be defined, namely num_groups for GroupNorm, and normalized_shape for LayerNorm.
Probably normalized_shape can somewhat be inferred from the input size with a series of precautions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant