Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Generic huggingface transformer support #30

Merged
merged 13 commits into from
Feb 9, 2024
Merged

feat: Generic huggingface transformer support #30

merged 13 commits into from
Feb 9, 2024

Commits on Jan 27, 2024

  1. feat: towards generic huggingface transformer support

    @le1nux
    le1nux committed Jan 27, 2024
    Configuration menu
    Copy the full SHA
    0992d21 View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2024

  1. feat: renamed redpajama memmap datasets (added tokenizer info)

    @le1nux
    le1nux committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    95e67a0 View commit details
    Browse the repository at this point in the history

  2. refactor: renamed tokenizer.json to tokenizer_gpt2.json

    @le1nux
    le1nux committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    3b51117 View commit details
    Browse the repository at this point in the history

  3. feat: implemented automatic FSDP wrapping

    @le1nux
    le1nux committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    88c4fdb View commit details
    Browse the repository at this point in the history

  4. feat: fully implemented HuggingFacePretrainedModel with respective co… 

    …nfiguration
    @le1nux
    le1nux committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    4d6a5ff View commit details
    Browse the repository at this point in the history

  5. chore: renamed Block to GPT2Block

    @le1nux
    le1nux committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    9309505 View commit details
    Browse the repository at this point in the history

  6. feat: wired up huggingface transformer models

    @le1nux
    le1nux committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    f83da11 View commit details
    Browse the repository at this point in the history

  7. refactor: fixed configs due to latest changes

    @le1nux
    le1nux committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    abb63aa View commit details
    Browse the repository at this point in the history

  8. chore: added documentation

    @le1nux
    le1nux committed Jan 28, 2024
    Configuration menu
    Copy the full SHA
    8e30e00 View commit details
    Browse the repository at this point in the history

Commits on Feb 9, 2024

  1. refactor: moved lookup_enum to dedicated file.

    @le1nux
    le1nux committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    e6f4aac View commit details
    Browse the repository at this point in the history

  2. feat: introduced huggingface_prediction_subscription_key to HuggingFa… 

    …cePretrainedModelConfig to support different output formats
    @le1nux
    le1nux committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    f435fc8 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into huggingface_models_support

    @le1nux
    le1nux authored Feb 9, 2024
    Configuration menu
    Copy the full SHA
    e93e767 View commit details
    Browse the repository at this point in the history

  4. Update preprocess_dataset.py

    @le1nux
    le1nux authored Feb 9, 2024
    Configuration menu
    Copy the full SHA
    00e10ae View commit details
    Browse the repository at this point in the history