❓ Questions / Help / Support Unexpected WER values on CommonVoice Test

[kfmn]

Hi, I am trying Silero English STT model to recognize a random subset (100 files) from English Common Voice Test.
I use a code based on a snippet from your README file, namely

device = torch.device('cpu')  # gpu also works, but our models are fast enough for CPU

def transcribe_file(filename, model, decoder, utils):
    (read_batch, split_into_batches, read_audio, prepare_model_input) = utils  # see function signature for details
    test_files = glob(filename)
    batches = split_into_batches(test_files, batch_size=10)
    input = prepare_model_input(read_batch(batches[0]),
                            device=device)

    output = model(input)
    result=''
    for example in output:
        result += decoder(example.cpu()) + ' '
    return result


def transcribe_dir(wavdir, recfile, model, decoder, utils):
    with codecs.open(recfile, 'w', encoding='utf8') as f:
        for speech_file in os.listdir(wavdir):
            print(speech_file)
            result = transcribe_file(os.path.join(wavdir,speech_file), model, decoder, utils)
            fname=speech_file.split('/')[-1]
            uttid=fname[:-4]      # cut .wav
            if not (result=='<error>' or result=='<noresult>'):
                f.write(f'{uttid} {result}\n')



dirin = sys.argv[1]
lang = sys.argv[2]        # available values: 'en', 'es', 'dе'
result = sys.argv[3]


model, decoder, utils = torch.hub.load(repo_or_dir='snakers4/silero-models',
                                       model='silero_stt',
                                       language=lang,
                                       device=device)


transcribe_dir(dirin, result, model, decoder, utils)

I obtain WER of about 35% which differs substantially from that reported in you diagram within README file (less than 20%). This difference cannot be explained by selection bias or large confidence interval...
Please explain,

• what English model is used by default ?
• where can I found metrics for it ?
• how can I use more accurate model to obtain results presented in your diagram ?

[snakers4]

Which model are you using?
If you are using the community edition small models, looks roughly similar.

[snakers4]

The "Minimal example" from colab with large model works:

[kfmn]

Upd: I have found the downloaded model, but when trying to look into I get the following message:

End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.

It seems like model archive is broken...

[kfmn]

You are right, 'jit_xlarge' works properly, but simpy 'jit' causes trouble I described. Please check it.
And WER for jit_xlarge is about 27% which is much better than 35 but still much worse than you reported. Does it mean I am still using Community Edition model, not Enterprise Edition ? Is it possible to test EE too ?

[snakers4]

Yes
We reserve EE test for commercial clients

[kfmn]

Ok, got it