Not able to load VLM #141
Comments
|
Can you give me more details? Not sure how to reproduce. You can also use the more recent engines based on litellm by installing the source |
Updated. @vinid |
|
Unfortunately set_backward_engine does not directly support vllm models. If you want to use VLLM you need to import the ChatVLLM interface: import os
import logging
import pytest
from typing import Union, List
from textgrad import Variable, BlackboxLLM, TextLoss
from textgrad.optimizer import TextualGradientDescent
from textgrad.engine.vllm import ChatVLLM
vllm_engine = ChatVLLM(model_string="meta-llama/Meta-Llama-3-8B-Instruct")
def test_simple_forward_pass_engine():
text = Variable("Hello", role_description="A variable")
engine = BlackboxLLM(engine=vllm_engine)
response = engine(text)
assert response
def test_primitives():
"""
Test the basic functionality of the Variable class.
"""
x = Variable("A sntence with a typo", role_description="The input sentence", requires_grad=True)
system_prompt = Variable("Evaluate the correctness of this sentence", role_description="The system prompt")
loss = TextLoss(system_prompt, engine=vllm_engine)
optimizer = TextualGradientDescent(parameters=[x], engine=vllm_engine)
l = loss(x)
l.backward(vllm_engine)
optimizer.step()
assert x.value == "A sentence with a typo" |
|
Thanks for the quick response. Getting memory error. I ran this in a A100 machine. Error: In the tutorial, its mentioned the below way to load the image. Since you mentioned the set_backward_engine is not supported for VLM, how to load the image and run the VLM model on it. |
|
Memory error is likely due to the vllm, can you send the strack trace of the actual call? |
|
INFO 11-13 11:07:47 model_runner.py:1067] Loading model weights took 19.9073 GB ` |
|
This is sus: Does vllm work without textgrad? Like what happens if you call it directly |
|
I tried it with hugging face codes. It’s working fine too. |
|
@vinid I had this error with vLLM for Llama3.2 vision. Later I was able to run via vLLM with a different script. |
|
I don't have access to a vLLM instance, what I would suggest you do is to either:
You might be able to copy paste code from your script into this engine class and then use it to run vLLM.
Let me know if any of these options work! |
Thanks for adding VLM support to textgrad.
This doc describe how to use textgrad to do the autoprompt for
gpt-4o.I would like to try non gpt-4o models such as
Qwen2 VLM/ Llama 3.2 9B VLMand generate the prompt automatically from a base prompt. Found this script which you added recently as a part of VLM integratioin.Error:
The text was updated successfully, but these errors were encountered: