Usage of MFCC function arm_mfcc_q31/f32

[hellgheast]

Dear Maintainer,

For our current development we are trying to use the function arm_mfcc_q31 in C (and in Python) and have some questions.

• How is this function supposed to be used for more than one FFT window (assuming you have a hop length of 160) ?
  We are currently using it this way, and are not sure if it's working as expected.

 
for (uint32_t i = 0,mfcccount = 0; (i + fftLen) <= MONO_AUDIO_NB_SAMPLES/2 && mfcccount < 100; i += HOP_LENGTH, mfcccount++) {
        ARMstatus = arm_mfcc_q31(
        &S,
        (q31_t*)&InputMonoResampledBuffer[i],
        (q31_t*)&OutputMFCC[mfcccount*nbDctOutputs],
        (q31_t*)TempMFCC
        );
    }

• Why in this file

  CMSIS-DSP/PythonWrapper/examples/testmfcc.py

  Line 101 in c28df48

  res=self.FFTSize*Q31toF32(resQ31)

  is the result of the Q31 MFCC conversion multiplied by the FFT length ?
• Is there any example in C or in python for outputting an array of MFCC sample using CMSIS-DSP ?
• We also saw this issue, Matching MFCC configuration with Librosa's #54 and when we tried the same audio sample (sine wave at 1 kHZ) we saw quite big difference between the output in float32 and q31, is it expected ?

window = sig.windows.hamming(FFTSize, sym=False)
      
filtLen,filtPos,packedFilters = mfcc.melFilterMatrix(F32,freq_min, freq_high, numOfMelFilters,sample_rate,FFTSize)

       
dctMatrixFilters = mfcc.dctMatrix(F32,numOfDctOutputs, numOfMelFilters)

mfccf32=dsp.arm_mfcc_instance_f32()

status=dsp.arm_mfcc_init_f32(mfccf32,FFTSize,numOfMelFilters,numOfDctOutputs,dctMatrixFilters,
    filtPos,filtLen,packedFilters,window)

tmp_nb = dsp.arm_mfcc_tmp_buffer_size(dt.F32,FFTSize,1)
tmp = np.zeros(tmp_nb,dtype=np.float32)
res=dsp.arm_mfcc_f32(mfccf32,pcm[0:FFTSize],tmp)

print("CMSIS-DSP")
print(res)

t = np.linspace(0,2,num_samples)
pcm = np.array(np.sin(2*math.pi * t * 1000),dtype=np.double)

mfccq31=dsp.arm_mfcc_instance_q31()

windowQ31 = dt.convert(sig.windows.hamming(FFTSize, sym=False),Q31)
filtLen,filtPos,packedFiltersQ31 = mfcc.melFilterMatrix(Q31,
                                                        freq_min, 
                                                        freq_high, 
                                                        numOfMelFilters,
                                                        sample_rate,
                                                        FFTSize)
dctMatrixFiltersQ31 = mfcc.dctMatrix(Q31,numOfDctOutputs, numOfMelFilters)


status=dsp.arm_mfcc_init_q31(mfccq31,FFTSize,numOfMelFilters,numOfDctOutputs,
    dctMatrixFiltersQ31,
    filtPos,filtLen,packedFiltersQ31,windowQ31)

#debugQ31 = toQ31(debug)

tmp_nb = dsp.arm_mfcc_tmp_buffer_size(dt.Q31,FFTSize,1)
tmp = np.zeros(tmp_nb,dtype=np.int32)

errorStatus,resQ31=dsp.arm_mfcc_q31(mfccq31,toQ31(pcm[0:FFTSize]),tmp)

res=FFTSize*Q31toF32(resQ31)

print("CMSIS-DSP-Q31")
print(resQ31)   
print(res)   

Best Regards,
Mohammed Ismail

[christophe0606]

@hellgheast Lots of good questions !

• How is this function supposed to be used for more than one FFT window

The tables used by the MFCC are generated by the script : Scripts\GenMFCCDataForCPP.py
There was a limitation (that has now been corrected). The window size was the fft length.
Now the window size can be smaller and the generated array will be padded with zero to match the fft length. (So window will not be centered on the fft length/2).
For instance, you could have a sliding window with an overlap of 320 samples. The MFCC input would be 640 samples (so the window length would be 640) and the FFT size would be 1024. With Python, you can do the same. In this, sig.windows.hamming(FFTSize, sym=False) FFTSize would be replaced by windowSize and the resulting array would have to be concatenated to an array of zeros so that the final array has FFTSize length.

• res=self.FFTSize*Q31toF32(resQ31)

This is wrong. Fixed point FFTs are scaled by their FFT length to avoid saturation but it is taken into account in the implementation. The documentation for the MFCC Q31 says the format is q8.23 so the float value should be 1.0 * resQ31 / (1<<23)
By doing this (and similar for the Q15 MFCC) we can see that both the Q31 and Q15 are returning close results. But, they also internally saturate easily so to check they both give the same result the input signal may have to be scaled down.

• Is there any example in C or in python for outputting an array of MFCC sample using CMSIS-DSP

You mean a real example not just a test ? I don't think there is a public one. I have a private one of a demo running on a board with TensorFlow Lite to implement keyword spotting. It uses the MFCC F32 and works. (I cannot make it public for now)

• we saw quite big difference between the output in float32 and q31, is it expected ?

It is not expected and I think it is going to take me a lot of time to find the issue.
I think it may be due to a wrong fixed point format before the log and it translates into a wrong offset after the log.

[christophe0606]

@hellgheast So it looks like the fixed point implementation cannot reproduce the float results because of the tradeoff saturation / accuracy. Either it saturates too often or many bits of accuracy must be lost. As consequence, the spectrum may be flattened and only retain the high amplitude components.
I have updated thd documentation to highlight this limitation. I would recommend to use the fixed point MFCCs only for AI tasks and if the network can be trained with the fixed point version and it demonstrates that it works.

They cannot just replace the float version without any retraining.

[hellgheast]

Dear Christophe, thanks a lot for all the clarifications !

Indeed, we were using this implementation for a keyword spotting application with MFCC computed using librosa as input data for the neural network and tought it would be nice to use to use the Q31 implementation instead of the F32.
We will retrain the neural net using the Python CMSIS wrapper, try with the floating implementation first and give feedback as soon as possible when we got it working.