【Language Detection】 Wrong language detection and Decreasing detection quality when specifying language

[sayeka0119]

Background: I bought a DVD and got its video.
Question:
1 Although I have cut the long video into small ones, whisper cannot correctly detect its language. It always detects the language as English or Chinese, but actually the video is Japanese.
2 When I use --language , the transcription quality is much worse than the one without --language
My goal:
I want to make whisper correctly detect the video with normal quality and accuracy.

[Advait251206]

This behavior can happen with Whisper, especially when the audio comes from movies, DVDs, anime, or heavily edited media rather than clean speech recordings.

Why Whisper may detect Japanese audio as English or Chinese

Whisper's language detection is performed using only a relatively short portion of audio (typically the beginning of the recording).

If the start of the clip contains:

• Music
• Sound effects
• Background noise
• Very little speech
• Whispered speech
• Character voices
• Foreign words or names

the language detector may make the wrong choice.

For example:

Japanese speech + background music

can sometimes be misclassified as:

English
Chinese
Korean

especially if the speech content is short.

---

Why transcription sometimes becomes worse when using --language ja

This seems counterintuitive, but there are several possible explanations.

1. The audio isn't actually mostly Japanese

Many DVDs contain:

• Japanese dialogue
• English names
• Songs
• Foreign-language inserts

When you force:

--language ja

Whisper is no longer allowed to consider alternative language hypotheses.

As a result, foreign words may be transcribed incorrectly.

---

2. Poor audio quality

If the audio contains:

• Crowd noise
• Music
• Explosions
• Reverb
• Multiple speakers

Whisper's automatic language detection can sometimes act as a useful prior.

Forcing Japanese may remove that flexibility and actually reduce accuracy.

---

3. Very short clips

You mentioned cutting the video into smaller pieces.

If the clips are only:

1–3 seconds

or contain only a few words, language detection becomes much less reliable.

Example:

はい
うん
え？

There may not be enough information to confidently identify the language.

---

Things to try

Option 1: Use a larger model

If you're not already using one, try:

--model large-v3

or

--model large-v3-turbo

Language identification generally improves with larger models.

---

Option 2: Remove non-speech regions

Run VAD before transcription.

Examples:

• :contentReference[oaicite:0]{index=0}
• :contentReference[oaicite:1]{index=1}

Pipeline:

DVD audio
↓
VAD
↓
Speech-only segments
↓
Whisper

This often improves language detection dramatically.

---

Option 3: Test language detection separately

You can inspect Whisper's language probabilities:

import whisper

model = whisper.load_model("large-v3")

audio = whisper.load_audio("sample.wav")
audio = whisper.pad_or_trim(audio)

mel = whisper.log_mel_spectrogram(audio).to(model.device)

_, probs = model.detect_language(mel)

print(sorted(probs.items(), key=lambda x: x[1], reverse=True)[:10])

Example output:

ja: 0.82
zh: 0.09
en: 0.04
ko: 0.03

This helps determine whether the detector is genuinely confused.

---

Option 4: Use longer clips

Instead of:

2–5 second clips

try:

20–30 second clips

Language detection is usually much more stable when more speech is available.

---

Option 5: Verify the DVD audio track

Many DVDs contain multiple audio tracks:

Japanese
English Dub
Commentary Track

Double-check which audio stream you're extracting.

You can inspect streams using:

ffprobe input.mkv

or

ffmpeg -i input.mkv

---

Why "automatic" may look better than forced Japanese

Whisper's decoder was trained jointly on:

• language identification
• transcription
• translation

When language detection is correct, automatic mode often gives the best results because the model can leverage its internal language predictions.

If forcing:

--language ja

consistently reduces quality, that usually suggests one of:

1. The audio is difficult/noisy.
2. The clip is too short.
3. There is substantial non-Japanese content.
4. The detected language is not actually Japanese for much of the segment.

---

Recommendation

For DVD content, I would try:

1. Extract the original audio track.
2. Remove long non-speech sections using VAD.
3. Use large-v3 or large-v3-turbo.
4. Test on 20–30 second clips instead of very short cuts.
5. Inspect the language probabilities from detect_language().

In many cases, poor language detection on Japanese media is caused by music, sound effects, or clips that are too short rather than a problem with Whisper itself.