GRPO on InternVL3.5 does not support dynamic vision tokens

[sanchit97]

Description

The issue likely occurs because InternVL uses a dynamic amount of Vision tokens, which are replaced by <IMG_CONTEXT>.
Setting the size of each image to a fixed resolution fixes the error, but it is less than ideal.

Reproduction

training_args = GRPOConfig(
        output_dir=test
        bf16=True,
        remove_unused_columns = False,
        per_device_train_batch_size=4,
        num_train_epochs=4,
        logging_steps=50,
        max_prompt_length = 4096,
        eval_strategy="steps",
        eval_steps=500,
        max_completion_length = 512,
        num_generations = 4,
        learning_rate = 2e-7, 
        )

        trainer = GRPOTrainer(
            model=model, # internvl3.5 type
            args=training_args,
            reward_funcs=[..],
            train_dataset=grpo_train_dataset,
            eval_dataset=grpo_eval_dataset,  
            processing_class=processor,
        )
        
        trainer.train()

outputs:

Traceback (most recent call last):
  ...
[rank0]:   File "/mnt/home/../miniconda3/envs/../lib/python3.10/site-packages/transformers/models/internvl/modeling_internvl.py", line 654, in get_placeholder_mask
[rank0]:     raise ValueError(
[rank0]: ValueError: Image features and image tokens do not match: tokens: 1536, features 512

System Info

trl==0.22

Checklist

• I have checked that my issue isn't already filed (see open issues)
• I have included my system information
• Any code provided is minimal, complete, and reproducible (more on MREs)
• Any code provided is properly formatted in code blocks, (no screenshot, more on code blocks)
• Any traceback provided is complete

[ycma8]

Maybe try increasing max_prompt_length and see if that helps.

[sanchit97]

Maybe try increasing max_prompt_length and see if that helps.

I did try that, to no success. Have you had success in doing that?

[ckddls1321]

I wanted to share a small observation I made while working with the Hugging Face version of InternVL 3.5 (OpenGVLab/InternVL3_5-2B-HF). I found that when trying to prepare model inputs, calling the processor directly did not seem to format the chat prompt as expected, particularly concerning the placement of the image token.

However, by following the official documentation and using the tokenizer's apply_chat_template method, the input was parsed perfectly. This method correctly processes the conversation and inserts the token in the appropriate location.

On a related note, I'd also like to share a brief insight regarding deployment with vLLM. It appears that vLLM's official support is primarily optimized for the original, non-HF version of InternVL. While the HF version is certainly usable. For example, I encountered an issue with dummy input processing when handling video inputs that required a workaround.