Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Aligner/nemotron5 #11264

Draft
wants to merge 80 commits into
base: main
Choose a base branch
from
Draft

Aligner/nemotron5 #11264

wants to merge 80 commits into from

Commits on Nov 4, 2024

  1. add nemotron5 conversion

    @JRD971000
    JRD971000 committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    1343bee View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2024

  1. Apply isort and black reformatting 

    Signed-off-by: JRD971000 <[email protected]>
    @JRD971000
    JRD971000 committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    ada4b90 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2024

  1. guard cuda access

    @JRD971000
    JRD971000 committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    627a40d View commit details
    Browse the repository at this point in the history

  2. Apply isort and black reformatting 

    Signed-off-by: JRD971000 <[email protected]>
    @JRD971000
    JRD971000 committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    75d8854 View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2024

  1. add nemotron5 conversion

    @JRD971000
    JRD971000 authored and Ali Taghibakhshi committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    0d9bb4f View commit details
    Browse the repository at this point in the history

  2. guard cuda access

    @JRD971000
    JRD971000 authored and Ali Taghibakhshi committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    aa0fafb View commit details
    Browse the repository at this point in the history

  3. Apply isort and black reformatting 

    Signed-off-by: JRD971000 <[email protected]>
    @JRD971000
    JRD971000 authored and Ali Taghibakhshi committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    57008da View commit details
    Browse the repository at this point in the history

  4. cleanup

    Ali Taghibakhshi committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    c26bd22 View commit details
    Browse the repository at this point in the history

  5. cleanup

    Ali Taghibakhshi committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    6e4bd6f View commit details
    Browse the repository at this point in the history

  6. Apply isort and black reformatting 

    Signed-off-by: JRD971000 <[email protected]>
    @JRD971000
    JRD971000 committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    3b63b4c View commit details
    Browse the repository at this point in the history

  7. a long overdue tiktoken special tokens fix -- Tkonuk 

    Signed-off-by: adithyare <[email protected]>
    @arendu
    arendu committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    19e5049 View commit details
    Browse the repository at this point in the history

  8. Apply isort and black reformatting 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    8231cde View commit details
    Browse the repository at this point in the history

  9. pad to mult is not available in chat dataset 

    Signed-off-by: adithyare <[email protected]>
    @arendu
    arendu committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    be7f996 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'aligner/nemotron5' of https://github.com/NVIDIA/NeMo in… 

    …to aligner/nemotron5
    @arendu
    arendu committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    88ec5e0 View commit details
    Browse the repository at this point in the history

  11. merged 

    Signed-off-by: adithyare <[email protected]>
    @arendu
    arendu committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    023dfbe View commit details
    Browse the repository at this point in the history

  12. Apply isort and black reformatting 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    e415b65 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2024

  1. disable vocab padding 

    Signed-off-by: Gerald Shen <[email protected]>
    @gshennvm
    gshennvm committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    ecb2bf6 View commit details
    Browse the repository at this point in the history

  2. fix for torch empty 

    Signed-off-by: adithyare <[email protected]>
    @arendu
    arendu committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    20e251c View commit details
    Browse the repository at this point in the history

  3. Minor changes to conversion script 

    Signed-off-by: Ali Taghibakhshi <[email protected]>
    @JRD971000
    JRD971000 authored Nov 13, 2024
    Configuration menu
    Copy the full SHA
    7c78ef4 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'aligner/nemotron5' of https://github.com/NVIDIA/NeMo in… 

    …to aligner/nemotron5
    @arendu
    arendu committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    acfde95 View commit details
    Browse the repository at this point in the history

  5. dtype fix in mamba 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    6c2ce66 View commit details
    Browse the repository at this point in the history

  6. Apply isort and black reformatting 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    985e0cf View commit details
    Browse the repository at this point in the history

  7. resolved conflict for dtype 

    Signed-off-by: adithyare <[email protected]>
    @arendu
    arendu committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    cf3bf49 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'aligner/nemotron5' of https://github.com/NVIDIA/NeMo in… 

    …to aligner/nemotron5
    @arendu
    arendu committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    b08f3eb View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. Set skip_special_tokens to False by default in tiktoken_tokenizer.py

    @ertkonuk
    ertkonuk committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    d6a014f View commit details
    Browse the repository at this point in the history

  2. removing redundant params_dtype attr in mamba yaml 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    a4134ef View commit details
    Browse the repository at this point in the history

  3. Apply isort and black reformatting 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    2cdd1a9 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2024

  1. add nemo intermediate ckpt

    Ali Taghibakhshi committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    85a1c9c View commit details
    Browse the repository at this point in the history

  2. Apply isort and black reformatting 

    Signed-off-by: JRD971000 <[email protected]>
    @JRD971000
    JRD971000 committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    ae07158 View commit details
    Browse the repository at this point in the history

Commits on Nov 19, 2024

  1. added timing logs 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    e4b2259 View commit details
    Browse the repository at this point in the history

  2. debug slowness 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    3b284af View commit details
    Browse the repository at this point in the history

  3. Workaround for crash due to bytes tokens in Tiktoken tokenizer

    @odelalleau
    odelalleau committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    6723809 View commit details
    Browse the repository at this point in the history

  4. debug Nones 

    Signed-off-by: Adi Renduchintala <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    3b2e00f View commit details
    Browse the repository at this point in the history

  5. Merge branch 'aligner/nemotron5' of https://github.com/NVIDIA/NeMo in… 

    …to aligner/nemotron5
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    54fba29 View commit details
    Browse the repository at this point in the history

  6. Apply isort and black reformatting 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    849ff34 View commit details
    Browse the repository at this point in the history

  7. debugging args to generate 

    Signed-off-by: adithyare <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    fdd8005 View commit details
    Browse the repository at this point in the history

  8. Apply isort and black reformatting 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    44d1e9d View commit details
    Browse the repository at this point in the history

  9. remove end_strings 

    Signed-off-by: Jiaqi Zeng <[email protected]>
    @HeyyyyyyG
    HeyyyyyyG committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    c4c7de6 View commit details
    Browse the repository at this point in the history

  10. Apply isort and black reformatting 

    Signed-off-by: HeyyyyyyG <[email protected]>
    @HeyyyyyyG
    HeyyyyyyG committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    3ab0d2c View commit details
    Browse the repository at this point in the history

  11. remove end_strings and end_of_turn 

    Signed-off-by: Jiaqi Zeng <[email protected]>
    @HeyyyyyyG
    HeyyyyyyG committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    23812c3 View commit details
    Browse the repository at this point in the history

  12. removed timing/debug logs 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    e504655 View commit details
    Browse the repository at this point in the history

  13. remove logs resolve conflicts 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    7730d5f View commit details
    Browse the repository at this point in the history

  14. removed logs in server, added a single timer 

    Signed-off-by: adithyare <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    405889e View commit details
    Browse the repository at this point in the history

  15. Apply isort and black reformatting 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    2db23a7 View commit details
    Browse the repository at this point in the history

  16. debug 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    d27c4a5 View commit details
    Browse the repository at this point in the history

  17. Apply isort and black reformatting 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    15fdf8a View commit details
    Browse the repository at this point in the history

  18. debug eval script times 

    Signed-off-by: adithyare <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    ca902fd View commit details
    Browse the repository at this point in the history

  19. Apply isort and black reformatting 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    aee8a89 View commit details
    Browse the repository at this point in the history

  20. added import 

    Signed-off-by: adithyare <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    3958925 View commit details
    Browse the repository at this point in the history

  21. Merge branch 'aligner/nemotron5' of https://github.com/NVIDIA/NeMo in… 

    …to aligner/nemotron5
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    6aa111f View commit details
    Browse the repository at this point in the history

  22. Apply isort and black reformatting 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    0a63807 View commit details
    Browse the repository at this point in the history

  23. time generate method 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    0278a01 View commit details
    Browse the repository at this point in the history

  24. Merge branch 'aligner/nemotron5' of https://github.com/NVIDIA/NeMo in… 

    …to aligner/nemotron5
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    744839c View commit details
    Browse the repository at this point in the history

  25. loop once in server mode 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 19, 2024
    Configuration menu
    Copy the full SHA
    2b44faf View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2024

  1. TRT-LLM FIX FOR NEMOTRON5, THIS BREAKS TRT FOR ALL OTHER MODELS 

    Signed-off-by: Gerald Shen <[email protected]>
    @gshennvm
    gshennvm committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    a923f76 View commit details
    Browse the repository at this point in the history

  2. fix(export): update API for disabling device reassignment in TRTLLM f… 

    …or Aligner (#10863)

* fix(export): update API for disabling device reassignment in TRTLLM for Aligner

[feat] Upgrade nemo-export path for aligner to TRTLLM-v12 and use python runtime

Signed-off-by: Terry Kong <[email protected]>

fix: forgot to always set _disable_torch_cuda_device_set

Signed-off-by: Terry Kong <[email protected]>

Signed-off-by: Terry Kong <[email protected]>

Apply isort and black reformatting

Signed-off-by: terrykong <[email protected]>

invert torch device set

Signed-off-by: Terry Kong <[email protected]>

* remove comment

Signed-off-by: Terry Kong <[email protected]>

---------

Signed-off-by: Terry Kong <[email protected]>
    @terrykong @gshennvm
    terrykong authored and gshennvm committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    df9374f View commit details
    Browse the repository at this point in the history

  3. removed logs and debugging code 

    Signed-off-by: adithyare <[email protected]>
    @arendu
    arendu committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    daf406b View commit details
    Browse the repository at this point in the history

  4. Merge branch 'aligner/nemotron5' of https://github.com/NVIDIA/NeMo in… 

    …to aligner/nemotron5
    @arendu
    arendu committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    9581135 View commit details
    Browse the repository at this point in the history

  5. Apply isort and black reformatting 

    Signed-off-by: arendu <[email protected]>
    @arendu
    arendu committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    551bf41 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2024

  1. add batching support in inference server

    Haifeng Qian committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    9853c30 View commit details
    Browse the repository at this point in the history

  2. Apply isort and black reformatting 

    Signed-off-by: haifengqian <[email protected]>
    @haifengqian
    haifengqian committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    b912e92 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2024

  1. hack to remove trailing </s> 

    Signed-off-by: Jiaqi Zeng <[email protected]>
    @HeyyyyyyG
    HeyyyyyyG committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    287ab7f View commit details
    Browse the repository at this point in the history

  2. Apply isort and black reformatting 

    Signed-off-by: HeyyyyyyG <[email protected]>
    @HeyyyyyyG
    HeyyyyyyG committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    c9b6c60 View commit details
    Browse the repository at this point in the history

  3. enforce tokens_to_generate as max number of generated tokens for each… 

    … sequence in a batch
    Haifeng Qian committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    65f0a3b View commit details
    Browse the repository at this point in the history

  4. Avoid potential race conditions with batching 

    In theory, with the previous implementation it would have been possible
for a thread to re-use the output from a previous batch, if it happened
to grab the lock before the thread with queryid == 0.
    @odelalleau
    odelalleau committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    17e148c View commit details
    Browse the repository at this point in the history

  5. Slightly reduce sleep time when batching queries 

    This can give a small speedup for free, since usually batched queries
all come in within <0.5s
    @odelalleau
    odelalleau committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    61f999a View commit details
    Browse the repository at this point in the history

  6. add checkpoint fix 

    Signed-off-by: Gerald Shen <[email protected]>
    @gshennvm
    gshennvm committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    23923fe View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2024

  1. only save untarred nemo files 

    Signed-off-by: Gerald Shen <[email protected]>
    @gshennvm
    gshennvm committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    cee062f View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2024

  1. Attempt to fix the nemotron5 tokenizer

    @ertkonuk
    ertkonuk committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    d07a17c View commit details
    Browse the repository at this point in the history

  2. Simplified the text_to_tokens method

    @ertkonuk
    ertkonuk committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    30cef20 View commit details
    Browse the repository at this point in the history

  3. Fixed tokenization of special characters.

    @ertkonuk
    ertkonuk committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    413e736 View commit details
    Browse the repository at this point in the history

  4. Use decode_with_offsets

    @ertkonuk
    ertkonuk committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    a77dc9f View commit details
    Browse the repository at this point in the history

  5. Skip BOS/EOS tokens in ids_to_text() by default 

    This is because those tokens are typically added in the code (e.g. for
padding purpose) and we do not want them to be part of the response.
    @odelalleau
    odelalleau committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    4b71c0f View commit details
    Browse the repository at this point in the history

  6. Ensure tokens_to_text() is consistent with ids_to_text()

    @odelalleau
    odelalleau committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    52ec872 View commit details
    Browse the repository at this point in the history

  7. Fix ids_to_tokens() to handle tokens associated to multiple token IDs

    @odelalleau
    odelalleau committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    ab699a5 View commit details
    Browse the repository at this point in the history

  8. Simplify implementation of token_to_id()

    @odelalleau
    odelalleau committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    c23db69 View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2024

  1. Fix for when ids_to_tokens() is unable to return a valid token

    @odelalleau
    odelalleau committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    9387c74 View commit details
    Browse the repository at this point in the history

  2. remove eos hack given the fix in 4b71c0f 

    Signed-off-by: Jiaqi Zeng <[email protected]>
    @HeyyyyyyG
    HeyyyyyyG committed Nov 27, 2024
    Configuration menu
    Copy the full SHA
    33564d4 View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2024

  1. change dist ckpt to zarr 

    Signed-off-by: Ali Taghibakhshi <[email protected]>
    @JRD971000
    JRD971000 authored Nov 28, 2024
    Configuration menu
    Copy the full SHA
    6076b60 View commit details
    Browse the repository at this point in the history

  2. Fully remove hack that was adding "</s>" to end_strings

    @odelalleau
    odelalleau committed Nov 28, 2024
    Configuration menu
    Copy the full SHA
    57ef506 View commit details
    Browse the repository at this point in the history