NEMO ASR on the audio recorded with phone

[kruthikakr]

Question :
Trying to run the Nemo for the audio recorded with the phone /zoom meeting , but the text output is very bad, the same audio which has been recorded with good microphone ,the text output is exact.

Can i know is there any reference for audio quality ? how to check the quality of the audio or how to use NeMo for real work applications.

[tiborkiss]

I did not played enough with that, but my first impression it was that with my studio microphone without dynamic compression and leveling, already was worst. Therefore at the beginning I will use with hardware dynamic compression. Once I am sure that this is the biggest issue, then I will make this compression in software also.

[kruthikakr]

ok thanks for replying. Now i need to use NeMo for audio recorded using phone /laptop microphone. Is there any way to improve the audio quality before giving it to the model. how to get the quality speech to text with phone recording /zoom audio ?

[tiborkiss]

Before going into development... first try to do some experiments. Take some audio processing tools, where you can do dynamic compression, limiting, even better if you can do proper noise reduction. Prepare manually a few variants of quality from the same file and compare the results. Then you will have the correct observation what makes sense to do and what doesn't make sense.
If you see the clear process what works, then the same thing can be implemented with librosa. (Of course some AI based noise reduction which you can make by iZotope RX you will not be able to reproduct by just using librosa.) Then you will go to search implementation examples from librosa or ISMIR, etc.
This requires a bit of work like this, a kind of R&D approach.

[jbalam-nv]

@kruthikakr Could you share some samples that you are seeing an issue with? Most of our training data consists of clean studio recordings but you should get reasonable transcripts for noisy audio say up to 20 dB SNR. Could you also share which model/models you experimented with?

Now i need to use NeMo for audio recorded using phone /laptop microphone. Is there any way to improve the audio quality before giving it to the model. how to get the quality speech to text with phone recording /zoom audio ?

You could finetune the model to with noise augmentation to improve noise robustness of the model, we have a noise robust QuartzNet model that you could give a try. We have a new release coming up (in a couple of weeks) with better models trained on more conversational data that may fit your application.

[kruthikakr]

**1. we tried with 1 zoom recorded audio which was m4a format and the the size was7.62MB and then converted in to .wav format (15 MB size) which is required by the Nemo using "ffmpeg". if there is an option to give in m4a , let us know the script .Sharing the both audio here (audio_data.zip).

2. Results using "QuartzNet15x5Base-En" model the transcript is pasted here.**

O.
Nexi, okay.
So I am just reading an article related to deepplenartic architectures for action recognition.
So, Oh, this is Thi the article related to.
Explore spacial temporal features.
Through various deep architectures, the role of optical flor is an import.
The impact ond real time capabilities and the compactness and indeve.
Retability of the learn features.
So before?
In that, I'll just start with Er.
Some paper or papers on the.
You see of fond, Zero, an dataset.
Oh.
So large scale video classification with convelsional red work.
And two stream networks.
Dearis to popularity of single stream under two Streem network in action recondition.
So the single frame network is essentially an image classubition atwork.
With no temporal features.
Late fusion uses too far apart frames and nfuses deep iancil features at the densely connected layers by flatktening.
Features process from the two fribs.
The strength of the single stream strategieis that.
We can use transfer learning from model.
Trained on last kilihis deterset.
We also have.
No one need to preporses the images for offical flow.
Edirectly used the Rgbma Tata in this architecture.
This makes single stream network a candate for real time processing.
In the fusion architectures.
Dhid, the author proposed.
The number of cerometer increases significantly from a deep toti sanente.
To mitigate this, the author proposed.
Using a multiresorlation streng.
One, the dambets say high resolution for Viar steam on Ah, center crop of the video combined with Et.
Low resolution contacts trim from the.
Hold vidio.
I seen Inda.
Architecture, Iao.
Oky, so I'll just to or change the ear.
Context.
Er one secend ham, just ster.
Let trile some.
So, generating optic, cof, floo, using, and videofloet.
To implementation.
The blog was originally published in Ta a blocg.
Bad slogan.
In a previous post, an introduction, tooptical flaws conducted, as well as an oervi of in its architecture, west of Pflonic.
This blot will focus in going deeper into optical floo.
Which will be done by generating optical Flo files?
Both from the standard.
Cintal Deta and the custom dance Vidio.
It will be conducted using a fork ofter.
Invidioflown into pytorch cord bass, which can be found in the.
Following link.
Okay, I think I know a ond two three minutes. I talk.
Thank you.
Yama'am thank you.
So?
Yeai'll start for a.
Some drenning stories.
Checking.
Like.
Don't even O bac bag, okay?
I'm selecting some.
I'm sorry.
A Ther deely, ie, no problem.
An ofa.
Your background dis clear now.
Nothing, I'll go into an none.
Ki'm starting to read one article.
My speech is made up of many frequencies at the same time that it is a sum.
Of all those frequencies.
To analyze the signal, we use the component frequencies as features.
For a transform is used to break the signal into.
These components.
We can use this splitting technique to convert the sound to a spectracram.
Ge frequency on the.
Mertical axis is splotted against.
The intensity of shady indicates the amplitude of the signal.
To create a spectrogram.
There are three lines sor.
Of the cord.
Divide the singer into.
Time frequencies.
Split each frim.
Frame singen into frequency components with an Ff. T.
Anthardies is weat.
Each timeframe is now represented with a vector of amplitude.
At each frequency.
Wel line up the vectors again.
Their time serius order, we have a.
Iual picture of the sound component.
Theepectogram.
Next you ill look at turfect.
Feature exsection techniques which werd.
Would reduce the noise and dimenslity of.
Our data.
So?
What his feature extruction with Ama Cisi.
Metal frequency.
Ceptum cofision analysis is the deduction of an audio signal.
To essential speech component.
Pita.
Using both mel, frequency analysis and separate anasis.
The range of frequencies are reduced and bint.
Into groups of frequencies that humans can distinguish.
The thing that is for ther.
Separated into sores and filter.
So that variations between speakers unrelated to.
Latin can be filtered away.
So that is from my side.
Hank you?
Ocalre, the article whit have shared in the.
Then I thought.
Your tue.
In the social ignitic action isjed he minor of omanging a language. Anyone who speaks our language does so. in our accent, The way of the native speaker. speaker of the language speaks, Fo, languer, is define the standard ponudication, and is generally considered to be standard for Tesmantano.
Your voice is not clar.
Not too.
Can use Um?
Ye fall nearer mouth.
The Mik never.
English has become an important language of O communication among the younger generations of the India, because of tic status as a language for the formal education, a large number of the young Indians in thebinokinnes.
T thatif, they speak one of the Twentie Twola Indian language as their first language alongside eglists.
An implication of this is that when speaking English, the intervetion from phonology of their first language, enample, malalum gives rise to the accent in the speech of the Indian speakers of English.
This accent is generally very distinct and is deadly.
Identiable, for example, as the Malarlum, English, acpenthe, sengunlix, acenabungala bungari in English action derecto.
This younger genetration of Windia, is also a large, an a growing group of a user of a speech pace technology, togh a Hn.
Andhelt device, and a voice assistant, This voice assistance has become very cool.
Entifyigi English is spoken in the native accent, Altourh, nonative accent. Speece continues to challenge for them for Sa. Rc. System of the Voyce assists have knowledge that the speaker is going to speak with a certain accent. The voice assistante may be trimeg, to listen to Sudten feature in the voice, which would lead to a great performance security.
But sheud de not.
O.
Yeah, ol kiv.
Oka.
Oh gesel, thank you.
Yen.

3. Results using "QuartzNet15x5NR-En" model the transcript is pasted here.
O Ol.
Nexsy O E.
So I am just reading an article related to Deplein Arctic architectures for action recognition.
So this is hi, the article related to.
Explore spacial temporal features.
Through various deep architectures, the role of optical flour is an import.
The impact owned real time capabilities and the compactness a
nd intevee.
Retability of the learn features.
So before?
In that, I'll just start with her.
Some paper or the papers on the.
You see a fone zero on at a set.
So lastscale video classification with convelsional red work.
And two stream networks.
Eis to popularity of single stream, anda two Strepm network in action, a condition.
So the single frame network is essentially an image classivition atwork.
With no temporal features.
Alate fusion uses two far apart frames and infuses deep irancill features at the densely connected layers by flatktning.
Features processed from the two fribs.
The strength of the single stream strategy is ther.
We can use transfer learning from model.
Trained on Lare killimi, etase.
We also have.
No, onnee to preposess the images for offical flow.
Directly used the Rgbmas Tata in this architectuce.
This makes single stream network a candate for real time processing.
In the fusion architectures.
Did the author proposed?
The number of carameter increases significantly from a deep todi sanente.
To mitigate this, the author proposed.
Using a multiresolution street.
N, the dambets a high resolution for vi steam on a center crop of the video combined with her.
Low resolution contects trim from the.
Hold ridio.
As seen in the.
Arhitectui.
Oyso, I'll just tar or change the or.
Context.
Erone, se cend am just stir.
Let's change some.
Sogenerating optic co flow, usinga.
To implementation.
The blog was originally published in De a block.
Bad slogan.
I aprevious post an introduction, Toptic Al Floos conducted, as well as an overview of Inix architecture, West offloic.
This blot will focus in going deeper into toptical flow.
Which will be done by generating optical flofiles?
Both from the standard.
Centrel Deta and the custom dance ridio.
It will be conducted using he fork ofter.
Invidio flow into pytoch hord bass, which can be found in the.
Following link.
Oki think I know around to three minutes I talke.
Thank you.
Yama, thank you.
So?
I'll start for E.
Some reading stories.
Icecking.
Een o bu. o.
I'm selecting some.
I'am sorry.
O.
Ano.
Background I ceno.
No, I'll go into an oe.
Ai'm starting to read one article.
My speech is made up of many frequencies at the same time thati it, isasum.
All those frequencies.
To analyze the signal, we use the component frequencies as features.
For a transform is used to break the signal into.
These components.
We can use this splitting technique to convert the sound to a spectrocram.
Get frequency on the.
Vertical axess is plotted against.
The intensity of shady indicates the amplitude of the signal.
To create a spectrogram.
There are three lines her.
Er.
Divide the singar into.
Time frequencies.
Split eas frim.
Frame singgan into frequency components with an Ff. T.
Ntarti iseet.
Each timeframe is now represented with a vector of amplitude.
At each frequency.
We line up the vectors again.
In theirtime series order, we have a.
Isual picture of the sound component.
The expectocam.
Next, you'll look at Teri.
Feature exsection techniques picturded.
Would reduce the noise and dimensility of.
Our data.
So?
What is feature extruction with Ama Piz?
Metal frequency.
Septem cofision analysis is the deduction of an audio signal.
Two essential speech component.
Et.
Using both mal, frequency analysis and seprate anlysis.
The range of frequencies are reduced and binn.
Into groups of frequencies that humans can distinguish.
The thingtat is for Ther.
Separated into sores and filter.
So that variations between speakers unelated to.
Articulation can be filtered away.
So that is from my side.
Hnyou.
Okay, I'll read the article which has shared in the.
Hen, I thought.
Ye do.
In the social liitic actiont is wethe, minor ofper managingal language. Anyone who speaks our language does so in our accent, The way of the nativist speaker. speaker of the language speaks, o languageis, define ter standard pronuniciation and is generally considered to be standard or deferent action.
Your voice is not cler.
Not to goo.
Can use em.
Ye fall near er mouth.
English has become an important language or for communication among the younger generation of the India, because of missus status as a language for the formal education, a large number of young Indians inarbinocene.
That if they speak one of the Twenti, two la Indian language as their first language alongside Iles.
An implication of this is that when speaking English, the intervesion from phonology of their first language, enample, Malalam gives srise to the accent in the speech of the Indian speakers of English.
This accent is generally very distinct and is deadly.
Identiable for the example, as the Madarlum English actenthor, sengunli, acentandbungala, bungai, in English action derectro.
This younger generation of Windia, is also a large and a growing group of a user of a steep pace technology through a han.
Andhel device, and a Voyd assistant, Thiss woyce, The assistance has become very good.
Ientiying this English, S spoken in the lative accent, Olwer, nonative accents, Speece continues to challenge for them, for S. A Rc. system of the voyce assists have knowledge that the speaker is going to speak with a certain accent. The voice assistants may be trying to listen to sudden feature in the voice, which would lead to a great performance security.
But should be not.
Ye.
Yeah, I'l ges.
Okin.
Oh, guesso, thank you.
Yen.

[kruthikakr]

Also provide some suitable or comparable tool /Script to record the audio using laptop /phone microphone, so that we get good transcript .It would be of great help, Thank you

[kruthikakr]

@jbalam-nv Can you please check on the sample audio and respective transcript and give us some inputs on how to improvise the result ?