Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wip add new proposed message structure #1904

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

wip add new proposed message structure #1904

wants to merge 16 commits into from

Commits on Oct 11, 2024

  1. wip add new proposed message structure

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    6e8ebe2 View commit details
    Browse the repository at this point in the history

  2. tokenization

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    5f42c4b View commit details
    Browse the repository at this point in the history

  3. wip

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    df604b9 View commit details
    Browse the repository at this point in the history

  4. wip transform builder

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    a8944f5 View commit details
    Browse the repository at this point in the history

  5. wip make the chat dataset loadable

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    b31e099 View commit details
    Browse the repository at this point in the history

  6. wip chatml + llama 3 new chat objects

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    7755ab1 View commit details
    Browse the repository at this point in the history

  7. chore: lint

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    ed1cf57 View commit details
    Browse the repository at this point in the history

  8. chore: lint

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    c29fa30 View commit details
    Browse the repository at this point in the history

  9. fix tokenization

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    87d91aa View commit details
    Browse the repository at this point in the history

  10. remove dacite dependency since we're using pydantic now

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    9455427 View commit details
    Browse the repository at this point in the history

  11. fix handling when already correctly split in messages

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    f7b124a View commit details
    Browse the repository at this point in the history

  12. make sure to remove chat features from tokenized ds

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    a362c6f View commit details
    Browse the repository at this point in the history

  13. move chat to be a input transform for messages

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    5336500 View commit details
    Browse the repository at this point in the history

  14. make sure llama3 has the bos token

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    9717d0b View commit details
    Browse the repository at this point in the history

  15. remove non-working special token code

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    14d4b16 View commit details
    Browse the repository at this point in the history

  16. fix messages strat loader

    @winglian
    winglian committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    0e08627 View commit details
    Browse the repository at this point in the history