Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Openai triton server #104

Draft
wants to merge 66 commits into
base: main
Choose a base branch
from
Draft

Openai triton server #104

wants to merge 66 commits into from

Commits on Mar 5, 2024

  1. updating with torch compile option for baseline

    @nnshah1
    nnshah1 committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    acdaac1 View commit details
    Browse the repository at this point in the history

  2. updated to build model on initialization

    @nnshah1
    nnshah1 committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    452c70f View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. moving torch tensorrt import

    @nnshah1
    nnshah1 committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    e518923 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2024

  1. updating dependencies

    @nnshah1
    nnshah1 committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    90a2266 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. updates for meetup tutorials

    @nnshah1
    nnshah1 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    c25db8f View commit details
    Browse the repository at this point in the history

  2. updated with tutorial repl

    @nnshah1
    nnshah1 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    3885dbf View commit details
    Browse the repository at this point in the history

  3. updating to volume mount local directory

    @nnshah1
    nnshah1 committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    4bec62b View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. update to install 24.03 version of genai-perf

    @nnshah1
    nnshah1 committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    9b5ce1f View commit details
    Browse the repository at this point in the history

  2. updates to make versions configurable

    @nnshah1
    nnshah1 committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    25692bd View commit details
    Browse the repository at this point in the history

  3. update with repl

    @nnshah1
    nnshah1 committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    49b526e View commit details
    Browse the repository at this point in the history

Commits on May 4, 2024

  1. updates with partial openai api support

    @nnshah1
    nnshah1 committed May 4, 2024
    Configuration menu
    Copy the full SHA
    53e12b9 View commit details
    Browse the repository at this point in the history

  2. more elaborate openai server

    @nnshah1
    nnshah1 committed May 4, 2024
    Configuration menu
    Copy the full SHA
    2b27882 View commit details
    Browse the repository at this point in the history

  3. adding in initial chat completions

    @nnshah1
    nnshah1 committed May 4, 2024
    Configuration menu
    Copy the full SHA
    0c8ee72 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2024

  1. updated to exclude openai.yml taken directly https://raw.githubuserco… 

    …ntent.com/openai/openai-openapi/master/openapi.yaml
    @nnshah1
    nnshah1 committed May 5, 2024
    Configuration menu
    Copy the full SHA
    884278f View commit details
    Browse the repository at this point in the history

  2. updated to exclude external unmodified api spec from pre-commit

    @nnshah1
    nnshah1 committed May 5, 2024
    Configuration menu
    Copy the full SHA
    72135a5 View commit details
    Browse the repository at this point in the history

  3. moving generated files into subdirectories

    @nnshah1
    nnshah1 committed May 5, 2024
    Configuration menu
    Copy the full SHA
    85c9c06 View commit details
    Browse the repository at this point in the history

  4. initial readme

    @nnshah1
    nnshah1 committed May 5, 2024
    Configuration menu
    Copy the full SHA
    058a6cb View commit details
    Browse the repository at this point in the history

  5. updates

    @nnshah1
    nnshah1 committed May 5, 2024
    Configuration menu
    Copy the full SHA
    0414fa1 View commit details
    Browse the repository at this point in the history

  6. update to make repl installation optional

    @nnshah1
    nnshah1 committed May 5, 2024
    Configuration menu
    Copy the full SHA
    f96e550 View commit details
    Browse the repository at this point in the history

  7. updating to 24.04 base image for stable diffusion

    @nnshah1
    nnshah1 committed May 5, 2024
    Configuration menu
    Copy the full SHA
    32b9a7a View commit details
    Browse the repository at this point in the history

  8. update with script for fastapi code gen

    @nnshah1
    nnshah1 committed May 5, 2024
    Configuration menu
    Copy the full SHA
    c78767d View commit details
    Browse the repository at this point in the history

  9. initial check in for codegen. use this as base for showing steps to m… 

    …odify
    @nnshah1
    nnshah1 committed May 5, 2024
    Configuration menu
    Copy the full SHA
    7a061ca View commit details
    Browse the repository at this point in the history

  10. added in models api

    @nnshah1
    nnshah1 committed May 5, 2024
    Configuration menu
    Copy the full SHA
    9b98479 View commit details
    Browse the repository at this point in the history

  11. updated with model info call

    @nnshah1
    nnshah1 committed May 5, 2024
    Configuration menu
    Copy the full SHA
    5c9acd2 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. remove delete

    @nnshah1
    nnshah1 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    b051b5a View commit details
    Browse the repository at this point in the history

  2. remove delete

    @nnshah1
    nnshah1 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    67b1ab6 View commit details
    Browse the repository at this point in the history

  3. adding support for completions api

    @nnshah1
    nnshah1 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    e952195 View commit details
    Browse the repository at this point in the history

  4. disabled logging

    @nnshah1
    nnshah1 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    680499c View commit details
    Browse the repository at this point in the history

Commits on May 7, 2024

  1. incremental updates for chat completions

    @nnshah1
    nnshah1 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    de62259 View commit details
    Browse the repository at this point in the history

  2. updates for chat completion api

    @nnshah1
    nnshah1 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    2509c34 View commit details
    Browse the repository at this point in the history

  3. update with assitant strings

    @nnshah1
    nnshah1 committed May 7, 2024
    Configuration menu
    Copy the full SHA
    73b3ce0 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. updates for trt-llm - note code is not portable between vllm and 

    trt-llm yet
    @nnshah1
    nnshah1 committed May 8, 2024
    Configuration menu
    Copy the full SHA
    57b5ac2 View commit details
    Browse the repository at this point in the history

  2. adding stop words

    @nnshah1
    nnshah1 committed May 8, 2024
    Configuration menu
    Copy the full SHA
    b1ec52f View commit details
    Browse the repository at this point in the history

  3. updates to make vllm and trt-llm work in same code base

    @nnshah1
    nnshah1 committed May 8, 2024
    Configuration menu
    Copy the full SHA
    c1dfd0a View commit details
    Browse the repository at this point in the history

  4. updates to support both trtlm and vllm

    @nnshah1
    nnshah1 committed May 8, 2024
    Configuration menu
    Copy the full SHA
    e759096 View commit details
    Browse the repository at this point in the history

  5. adding transformer utilities to get tokenizer - may need further tweaks

    @nnshah1
    nnshah1 committed May 8, 2024
    Configuration menu
    Copy the full SHA
    25de682 View commit details
    Browse the repository at this point in the history

  6. updated to return string unless decode exception

    @nnshah1
    nnshah1 committed May 8, 2024
    Configuration menu
    Copy the full SHA
    52dc3f4 View commit details
    Browse the repository at this point in the history

  7. updated with metrics

    @nnshah1
    nnshah1 committed May 8, 2024
    Configuration menu
    Copy the full SHA
    2b42bee View commit details
    Browse the repository at this point in the history

  8. deleting unused files

    @nnshah1
    nnshah1 committed May 8, 2024
    Configuration menu
    Copy the full SHA
    e53e763 View commit details
    Browse the repository at this point in the history

  9. adding basic support for arg parsing

    @nnshah1
    nnshah1 committed May 8, 2024
    Configuration menu
    Copy the full SHA
    0774e84 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. update to list both triton id and source id

    @nnshah1
    nnshah1 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    7228d9e View commit details
    Browse the repository at this point in the history

  2. matched version in route

    @nnshah1
    nnshah1 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    f7181a0 View commit details
    Browse the repository at this point in the history

  3. renaming application

    @nnshah1
    nnshah1 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    e0109e6 View commit details
    Browse the repository at this point in the history

  4. adding developer tool

    @nnshah1
    nnshah1 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    20e14a9 View commit details
    Browse the repository at this point in the history

  5. update typo

    @nnshah1
    nnshah1 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    7237a79 View commit details
    Browse the repository at this point in the history

  6. updated

    @nnshah1
    nnshah1 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    31a5765 View commit details
    Browse the repository at this point in the history

  7. updated

    @nnshah1
    nnshah1 committed May 9, 2024
    Configuration menu
    Copy the full SHA
    afec851 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. updated tag for genaiperf

    @nnshah1
    nnshah1 committed May 14, 2024
    Configuration menu
    Copy the full SHA
    cc72ee1 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'nnshah1-meetup-04-2024' of https://github.com/triton-in… 

    …ference-server/tutorials into nnshah1-meetup-04-2024
    @nnshah1
    nnshah1 committed May 14, 2024
    Configuration menu
    Copy the full SHA
    3d96c2d View commit details
    Browse the repository at this point in the history

  3. updated readme with known issues / limitations

    @nnshah1
    nnshah1 committed May 14, 2024
    Configuration menu
    Copy the full SHA
    f1ed67f View commit details
    Browse the repository at this point in the history

Commits on May 19, 2024

  1. updating with pytorch base example

    @nnshah1
    nnshah1 committed May 19, 2024
    Configuration menu
    Copy the full SHA
    be5bb9d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'nnshah1-meetup-04-2024' of https://github.com/triton-in… 

    …ference-server/tutorials into nnshah1-meetup-04-2024
    @nnshah1
    nnshah1 committed May 19, 2024
    Configuration menu
    Copy the full SHA
    73244cd View commit details
    Browse the repository at this point in the history

  3. updating genai perf tag

    @nnshah1
    nnshah1 committed May 19, 2024
    Configuration menu
    Copy the full SHA
    cc4d886 View commit details
    Browse the repository at this point in the history

Commits on May 22, 2024

  1. updated to build pytorch without torchvision

    @nnshah1
    nnshah1 committed May 22, 2024
    Configuration menu
    Copy the full SHA
    41ef140 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. upgrade to 24.05

    @nnshah1
    nnshah1 committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    3573aaa View commit details
    Browse the repository at this point in the history

  2. update run to 24.05

    @nnshah1
    nnshah1 committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    3556d99 View commit details
    Browse the repository at this point in the history

  3. update to exclude none in settng vllm request

    @nnshah1
    nnshah1 committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    246a237 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. Fix the parameter to tensor conversion in TRTLLM FastAPI implementati… 

    …on (#98)

* Fix the parameter to tensor conversion in TRTLLM FastAPI implementation

* Fix format
    @tanmayv25
    tanmayv25 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    b1653d7 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. moving to 24.06 as base

    @nnshah1
    nnshah1 committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    0eb2bd4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'nnshah1-meetup-04-2024' of https://github.com/triton-in… 

    …ference-server/tutorials into nnshah1-meetup-04-2024
    @nnshah1
    nnshah1 committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    ce39d40 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. Merge branch 'nnshah1-meetup-04-2024' into openai-triton-server

    @nnshah1
    nnshah1 committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    d70d377 View commit details
    Browse the repository at this point in the history

  2. updating to remove unnecessary parts

    @nnshah1
    nnshah1 committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    db86092 View commit details
    Browse the repository at this point in the history

  3. updated to not rely on triton cli internals

    @nnshah1
    nnshah1 committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    45bf9a6 View commit details
    Browse the repository at this point in the history

  4. typo

    @nnshah1
    nnshah1 committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    1cb3932 View commit details
    Browse the repository at this point in the history

  5. updated to remove pip install

    @nnshah1
    nnshah1 committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    8402989 View commit details
    Browse the repository at this point in the history

  6. updated for style

    @nnshah1
    nnshah1 committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    c00bf15 View commit details
    Browse the repository at this point in the history