feat: Add hf-mistral3 adapter for Ministral-3 models

[medhakimbedhief]

Summary

Adds support for evaluating Ministral-3 models (3B, 8B, 14B) which use Mistral3ForConditionalGeneration instead of AutoModelForCausalLM.

Closes #3483

Motivation

This bug was discovered during an internal evaluation project at Vago Solutions, a startup based in Germany. When attempting to benchmark Ministral-3 models using the standard hf backend, the evaluation fails with:

ValueError: Unrecognized configuration class <class 'transformers.models.mistral3.configuration_mistral3.Mistral3Config'> for this kind of AutoModel: AutoModelForCausalLM

This occurs because lm_eval hard-codes usage of AutoModelForCausalLM, while Ministral-3 models in Transformers 5.x are exposed as Mistral3ForConditionalGeneration (a VLM-style class).

Solution

Introduces a new model adapter Mistral3LM that:

• Inherits from HFLM- Dynamically imports Mistral3ForConditionalGeneration with graceful error handling
• Forces the causal backend (Mistral3 is decoder-only despite the class name)
• Overrides _model_call() to bypass the AutoModelForCausalLM assertion
• Handles text_config for proper max_length detection

Usage

lm_eval --model hf-mistral3
--model_args pretrained=mistralai/Ministral-3-3B-Instruct-2512-BF16,dtype=bfloat16
--tasks hellaswag
--device cuda:0
--batch_size 8

Files Changed

• lm_eval/models/mistral3.py - New adapter (created)- lm_eval/models/__init__.py
• Added hf-mistral3 to MODEL_MAPPING

Requirements

• Transformers >= 5.0.0 (or installed from main branch)

[CLAassistant]

All committers have signed the CLA.

[baberabb]

This is great! thanks for adding it!