How to use the finetuned model in whisper library

[keval-sakhiya01]

How to use the finetuned model in whisper library?

[Advait251206]

It depends on how the model was fine-tuned and which framework was used for training.

1. Fine-tuned with Hugging Face Transformers

This is the most common scenario.

Example:

from transformers import WhisperForConditionalGeneration
from transformers import WhisperProcessor

model = WhisperForConditionalGeneration.from_pretrained(
    "./my_finetuned_whisper"
)

processor = WhisperProcessor.from_pretrained(
    "./my_finetuned_whisper"
)

You can use the model directly through the Hugging Face pipeline:

from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="./my_finetuned_whisper"
)

result = pipe("audio.wav")
print(result["text"])

This is usually the easiest and safest approach.

---

2. Can I load it with OpenAI's Whisper library?

Not directly in most cases.

The OpenAI Whisper library expects checkpoints in the original Whisper format:

import whisper

model = whisper.load_model("large-v3")

A Hugging Face fine-tuned model contains:

pytorch_model.bin
model.safetensors
config.json
generation_config.json
...

while OpenAI Whisper expects:

model_state_dict
+
Whisper dimensions
+
OpenAI checkpoint structure

Therefore:

whisper.load_model("./my_finetuned_model")

will usually fail unless the weights have been converted.

---

3. If you trained using LoRA / PEFT

For PEFT models:

from peft import PeftModel
from transformers import WhisperForConditionalGeneration

base_model = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-large-v3"
)

model = PeftModel.from_pretrained(
    base_model,
    "./lora_checkpoint"
)

For inference you can either:

Option A

Keep the LoRA adapters attached:

model.generate(...)

Option B

Merge them into the base model:

merged_model = model.merge_and_unload()

merged_model.save_pretrained(
    "./merged_whisper"
)

After merging, you can use the resulting Hugging Face model normally.

---

4. If you want to use OpenAI Whisper APIs (model.transcribe())

You must convert the checkpoint into OpenAI Whisper's format.

This generally requires:

1. Loading the HF model.
2. Mapping parameter names.
3. Exporting a compatible state dict.
4. Loading into an OpenAI Whisper architecture.

Many community discussions (e.g. conversions between HF and OpenAI checkpoints) follow this pattern, but there is no officially supported one-command converter.

---

5. Recommended approach

For most users:

Fine-tuned with Transformers
        ↓
Use Transformers for inference

instead of:

Fine-tuned with Transformers
        ↓
Convert
        ↓
OpenAI Whisper library

The Hugging Face ecosystem already supports:

• Whisper
• LoRA / PEFT
• Quantization
• ONNX export
• Better batching
• Deployment tools

so there is usually little benefit in converting back unless you specifically need OpenAI Whisper's transcribe() implementation.

---

If you're unsure

Please share:

• Base model (tiny, base, small, large-v3, etc.)
• Training method (full fine-tuning, LoRA, QLoRA, DeepSpeed, etc.)
• Framework used (transformers, peft, whisper-finetune, etc.)
• Output files in your checkpoint directory

and it will be easier to determine the exact steps needed to run the model.