Skip to content

Commit

Permalink
Added vLLM model type to docs
Browse files Browse the repository at this point in the history
  • Loading branch information
mstfbl committed Sep 10, 2024
1 parent 82fc146 commit 02e28da
Showing 1 changed file with 41 additions and 0 deletions.
41 changes: 41 additions & 0 deletions docs/api-reference/model.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -162,3 +162,44 @@ results = experiment.run(
best_result = experiment.experiment_result.best_run_result
```
</Accordion>

<Accordion title="VLLMModel">

### Overview

The `VLLMModel` class is a subclass of `Model` that is implemented to work with models served through vLLM. It handles the setup and execution, allowing you to easily integrate vLLM-served models them with your applications.

### Class Variables

| Parameter | Type | Default | Description | Required |
|----------------------|----------------------------|--------------------------------|----------------------------------------------------------------------------------------------|----------|
| `api_url` | `Optional[str]` | N/A | URL of model being served through vLLM. | No |
| `required_api_keys` | `ClassVar[Set[str]]` | `{}` | The set of expected API keys for model being served through vLLM. | No |
| `hyperparameters` | `ClassVar[Dict]` | A default hyperparameter search space with `temperature`, `max_tokens`, `top_p`. | A dictionary defining the hyperparameters that can be tuned for the model being served on vLLM. | No |


### Example Usage

```python
from nomadic.model import VLLMModel
from nomadic.tuner import tune

experiment = Experiment(
model=VLLMModel(api_url="http://localhost:8000/generate"),
params={"temperature", "max_tokens"},
evaluation_dataset=json.loads(
requests.get(
"https://example.com/evaluation_dataset.json"
).content
),
evaluator=SemanticSimilarityEvaluator(embed_model=OpenAIEmbedding()),
)

results = experiment.run(
param_dict={
"temperature": tune.choice([0.1, 0.5, 0.9]),
"max_tokens": tune.choice([50, 100, 200]),
})
best_result = experiment.experiment_result.best_run_result
```
</Accordion>

0 comments on commit 02e28da

Please sign in to comment.