Skip to content

Commit

Permalink
fixes --library-name parameter in example for sentence_transformers.m…
Browse files Browse the repository at this point in the history
…dx (#703)

fixes --library-name paramater in example for sentence_transformers.mdx

Looks like we stopped passing in --library-name at some point.  This is the only place I could find it in the docs.  We infer it from the model type.  

If I run the example:
```
optimum-cli export neuron -m Snowflake/snowflake-arctic-embed-l --library-name sentence_transformers --sequence_length 1024 --batch_size 1 --task feature-extraction snowflake-arctic-embed-l-inf2
/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: `Transformer2DModelOutput` is deprecated and will be removed in version 1.0.0. Importing `Transformer2DModelOutput` from `diffusers.models.transformer_2d` is deprecated and this will be removed in a future version. Please use `from diffusers.models.modeling_outputs import Transformer2DModelOutput`, instead.
  deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message)
usage: optimum-cli
Optimum CLI tool: error: unrecognized arguments: --library-name snowflake-arctic-embed-l-inf2

```

If I remove --library-name, the example compiles.

In the code example, they show it being passed in.  I assume that still works because I see it being used in the tests.
  • Loading branch information
jimburtoft authored Sep 30, 2024
1 parent 7180d48 commit d0a621f
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions docs/source/inference_tutorials/sentence_transformers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ model.save_pretrained("bge_emb_inf2/")
Here we will use the `optimum-cli` to convert the model. Similar to the `NeuronModelForSentenceTransformers` we need to define our input shape and batch size. The input shape is defined by the `sequence_length` and the batch size by `batch_size`. The `optimum-cli` will automatically convert the model to a format compatible with AWS Inferentia2 and save it to the specified output directory.

```bash
optimum-cli export neuron -m BAAI/bge-small-en-v1.5 --library-name sentence_transformers --sequence_length 384 --batch_size 1 --task feature-extraction bge_emb_inf2/
optimum-cli export neuron -m BAAI/bge-small-en-v1.5 --sequence_length 384 --batch_size 1 --task feature-extraction bge_emb_inf2/
```

### Load compiled Sentence Transformers model and run inference
Expand Down Expand Up @@ -94,7 +94,7 @@ You can compile CLIP models with Optimum Neuron either by using the `optimum-cli
* With the Optimum CLI
```bash
optimum-cli export neuron -m sentence-transformers/clip-ViT-B-32 --sequence_length 64 --text_batch_size 3 --image_batch_size 1 --num_channels 3 --height 224 --width 224 --task feature-extraction --library-name sentence_transformers --subfolder 0_CLIPModel clip_emb/
optimum-cli export neuron -m sentence-transformers/clip-ViT-B-32 --sequence_length 64 --text_batch_size 3 --image_batch_size 1 --num_channels 3 --height 224 --width 224 --task feature-extraction --subfolder 0_CLIPModel clip_emb/
```
* With the `NeuronModelForSentenceTransformers` class
Expand Down Expand Up @@ -158,4 +158,4 @@ Since compiled models with dynamic batching enabled only accept input tensors wi

eg. if you want to encode 3 or 4 or 5 texts and 1 image, you could set `text_batch_size = 5 = max(3, 4, 5)` and `image_batch_size = 1` during the compilation.

</Tip>
</Tip>

0 comments on commit d0a621f

Please sign in to comment.