Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Draft: Feat/initialization component #168

Merged
merged 42 commits into from
Jul 3, 2024
Merged

Draft: Feat/initialization component #168

merged 42 commits into from
Jul 3, 2024

Commits on Jun 30, 2024

  1. feat: added Weight Initialization Factory

    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    53616e0 View commit details
    Browse the repository at this point in the history

  2. feat: implemented model-wise and named-parameter-wise initalization c… 

    …lasses
    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    03b7a7f View commit details
    Browse the repository at this point in the history

  3. feat: drafted the init configs

    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    d2ecf0d View commit details
    Browse the repository at this point in the history

  4. chore: added missing init file

    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    0efd32d View commit details
    Browse the repository at this point in the history

  5. refactor: removed previous initialization code in the abstract model … 

    …class
    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    dcafa38 View commit details
    Browse the repository at this point in the history

  6. feat: added initialization config classes

    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    83ce570 View commit details
    Browse the repository at this point in the history

  7. refactor: moved WeightInitializationIF to separate file to prevent ci… 

    …rcular imports
    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    ce48a57 View commit details
    Browse the repository at this point in the history

  8. feat: added WeightInitializerWrapperConfig to config

    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    a08d4d9 View commit details
    Browse the repository at this point in the history

  9. feat: added WeightInitializerWrapper

    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    c39eabd View commit details
    Browse the repository at this point in the history

  10. feat: wired up all the weight initializers as components

    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    2af2625 View commit details
    Browse the repository at this point in the history

  11. feat: added functionaliy to initalize model weights

    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    a950746 View commit details
    Browse the repository at this point in the history

  12. feat: added the weight init to config lorem ipsum

    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    416831a View commit details
    Browse the repository at this point in the history

  13. refactor: removed raising an excpetion when module is not covered for… 

    … init. Need to check how we want to handle this case
    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    86308d4 View commit details
    Browse the repository at this point in the history

  14. refactor: fixed pydantic model_validator in PlainWeightInitialization… 

    …Config
    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    d2f2e15 View commit details
    Browse the repository at this point in the history

  15. refactor: removed all weight init code from the models

    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    2dc37e2 View commit details
    Browse the repository at this point in the history

  16. chore: fixed typo

    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    f7d6d0d View commit details
    Browse the repository at this point in the history

  17. refactor: finalized config lorem ipsum for weight initialization

    @le1nux
    le1nux committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    8fcaf08 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. refactor: moved configs back to the init factory and removed not need… 

    …ed init method
    @le1nux
    le1nux committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    fe2fb49 View commit details
    Browse the repository at this point in the history

  2. feat: added HighLevelWeightInitializationFactory

    @le1nux
    le1nux committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    7164482 View commit details
    Browse the repository at this point in the history

  3. feat: wired up HighLevelWeightInitializationFactory

    @le1nux
    le1nux committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    327dbd8 View commit details
    Browse the repository at this point in the history

  4. refactor: config_lorem_ipsum.yaml now suppports HighLevelWeightInitia… 

    …lizationFactory
    @le1nux
    le1nux committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    89d5a29 View commit details
    Browse the repository at this point in the history

  5. refactor: mean of weight init constraint is now relaxed to float-only

    @le1nux
    le1nux committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    010f362 View commit details
    Browse the repository at this point in the history

  6. chore: added reference citation to the weigh init for gpt2

    @le1nux
    le1nux committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    c0a9aa5 View commit details
    Browse the repository at this point in the history

  7. fix: replaced std of 0.4 by math.sqrt(0.4) for scaled_embed

    @le1nux
    le1nux committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    7d26675 View commit details
    Browse the repository at this point in the history

  8. chore: added citation

    @le1nux
    le1nux committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    bb4cd36 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. refactor: added edge case handling when module is not of type linear … 

    …or embedding during initialization
    @le1nux
    le1nux committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    48dced9 View commit details
    Browse the repository at this point in the history

  2. refactor: removed ModulewiseNormalInitialization and extended NamedPa… 

    …rameterwiseNormalInitialization to support also plain initialisation
    @le1nux
    le1nux committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    93a0cb4 View commit details
    Browse the repository at this point in the history

  3. refactor: removed old weigh_init from coca config

    @le1nux
    le1nux committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    c2817f7 View commit details
    Browse the repository at this point in the history

  4. refactor: all weight init code is now based on regex matching on name… 

    …d parameters
    @le1nux
    le1nux committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    98fe633 View commit details
    Browse the repository at this point in the history

  5. feat: added plain weight init for CoCa

    @le1nux
    le1nux committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    b8e42c1 View commit details
    Browse the repository at this point in the history

  6. refactor: renamed plain_std -> std

    @le1nux
    le1nux committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    054d6a2 View commit details
    Browse the repository at this point in the history

  7. refactor: renamed plain_std -> std (missed one instance)

    @le1nux
    le1nux committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    602336c View commit details
    Browse the repository at this point in the history

  8. refactor: some renamings and passing now the calculated std from plai… 

    …n to scaled.
    @le1nux
    le1nux committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    824da6a View commit details
    Browse the repository at this point in the history

  9. refactor: removed legacy NamedParameterwiseNormalInitializationConfig

    @le1nux
    le1nux committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    4573364 View commit details
    Browse the repository at this point in the history

  10. refactor: simplified the initialization structure and improved naming

    @le1nux
    le1nux committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    0e29447 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. refactor: for plain init we do not allow hidde_dim to be specified wh… 

    …en std is of type float
    @le1nux
    le1nux committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    eea99bc View commit details
    Browse the repository at this point in the history

  2. refactor: removed currently not needed init routines from component r… 

    …egistry.
    @le1nux
    le1nux committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    d7c0825 View commit details
    Browse the repository at this point in the history

  3. test: initialization unit test adjustments (first steps)

    @flxst
    flxst committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    2c50e06 View commit details
    Browse the repository at this point in the history

  4. test: initialization unit test adjustments (config fix)

    @flxst
    flxst committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    96c0657 View commit details
    Browse the repository at this point in the history

  5. test: fix coca config

    @flxst
    flxst committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    d3c1683 View commit details
    Browse the repository at this point in the history

  6. feat: added plain filters for coca

    @le1nux
    le1nux committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    1a61bf6 View commit details
    Browse the repository at this point in the history

  7. test: readd test for initialization

    @flxst
    flxst committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    003eb99 View commit details
    Browse the repository at this point in the history