Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Voltron v0 #1

Open
wants to merge 42 commits into
base: main
Choose a base branch
from
Open

[WIP] Voltron v0 #1

wants to merge 42 commits into from

Commits on Apr 13, 2024

  1. wip for v0 for testing

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    f43824f View commit details
    Browse the repository at this point in the history

  2. mini fixes

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    b7cd6ea View commit details
    Browse the repository at this point in the history

  3. fix install and train

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    9b56525 View commit details
    Browse the repository at this point in the history

  4. fix config and dataset

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    a357c7f View commit details
    Browse the repository at this point in the history

  5. fix lr

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    1bfbdb5 View commit details
    Browse the repository at this point in the history

  6. fix text field of dataset

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    072b74a View commit details
    Browse the repository at this point in the history

  7. improve data handling

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    38a872d View commit details
    Browse the repository at this point in the history

  8. flesh out the model w/ attn

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    6c0e92b View commit details
    Browse the repository at this point in the history

  9. fix args/kwargs ordering

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    91c4fa2 View commit details
    Browse the repository at this point in the history

  10. use LlamaBitMGQA

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    c298db1 View commit details
    Browse the repository at this point in the history

  11. fix order of init for module

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    b5487fd View commit details
    Browse the repository at this point in the history

  12. make tinier and fix dataset map

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    a7854a2 View commit details
    Browse the repository at this point in the history

  13. fix back to use dataset.data.columns

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    bc4ce8a View commit details
    Browse the repository at this point in the history

  14. make sure to remove extra columns

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    122316e View commit details
    Browse the repository at this point in the history

  15. fix data loop and make tinier

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    a1c7b2a View commit details
    Browse the repository at this point in the history

  16. use generic collator to pad equally

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    0d76c4e View commit details
    Browse the repository at this point in the history

  17. accont for position_ids in mod block

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    91d4a11 View commit details
    Browse the repository at this point in the history

  18. flesh out rotary embeddigs

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    81e18b9 View commit details
    Browse the repository at this point in the history

  19. misc fixes

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    bc5ac07 View commit details
    Browse the repository at this point in the history

  20. fix tokenizer and activation checkpointing

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    c5a4c71 View commit details
    Browse the repository at this point in the history

  21. more fixes

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    c493b65 View commit details
    Browse the repository at this point in the history

  22. remove hard dependencies from axolotl

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    8aa8d81 View commit details
    Browse the repository at this point in the history

  23. remove more hard deps

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    e6edf93 View commit details
    Browse the repository at this point in the history

  24. re-enable DWA again

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    270150f View commit details
    Browse the repository at this point in the history

  25. actually check for dwa

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    49cc04a View commit details
    Browse the repository at this point in the history

  26. wandb on main rank only

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    4437672 View commit details
    Browse the repository at this point in the history

  27. fix modulo for log steps

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    c2f804c View commit details
    Browse the repository at this point in the history

  28. attempt to use accelerator loop

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    88f25a9 View commit details
    Browse the repository at this point in the history

  29. update configuration

    @winglian
    winglian committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    a10c31a View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2024

  1. wip rms norm

    @winglian
    winglian committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    8239c6e View commit details
    Browse the repository at this point in the history

  2. use apex rms norm optim

    @winglian
    winglian committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    2b2f332 View commit details
    Browse the repository at this point in the history

  3. queued dataloader and gradient norm

    @winglian
    winglian committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    cba6e66 View commit details
    Browse the repository at this point in the history

  4. fixes for loss calc, grad accum, dataloader for dispatch_batches

    @winglian
    winglian committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    ffafd7a View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2024

  1. tweak size names

    @winglian
    winglian committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    0255ffe View commit details
    Browse the repository at this point in the history

  2. upcast/downcast

    @winglian
    winglian committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    d289d98 View commit details
    Browse the repository at this point in the history

  3. integrate infini-attention

    @winglian
    winglian committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    84d755a View commit details
    Browse the repository at this point in the history

  4. handle position_id if passed, throw it on the floor

    @winglian
    winglian committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    665530b View commit details
    Browse the repository at this point in the history

  5. match infini-attention segment len to mixture of depth

    @winglian
    winglian committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    2439688 View commit details
    Browse the repository at this point in the history

  6. misc fixes for integrations

    @winglian
    winglian committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    a77e6ae View commit details
    Browse the repository at this point in the history

  7. make infini-attention work

    @winglian
    winglian committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    6a49def View commit details
    Browse the repository at this point in the history

  8. fix dimensions passed to infini-attention

    @winglian
    winglian committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    555087a View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. fix perplexity calculation and add quick instructions

    @winglian
    winglian committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    b117faa View commit details
    Browse the repository at this point in the history