Why do ctc.py and mmi.py use use_double_scores=True?

[galv]

I am referring to the following:

snowfall/snowfall/objectives/ctc.py

Line 38 in b7f76b6

use_double_scores=True

snowfall/snowfall/objectives/mmi.py

Line 85 in b7f76b6

log_semiring=True, use_double_scores=True)

This is a bit unusual to me. Was there a particular motivation? @csukuangfj it seems like you were the one who chose to use double instead of single precision.

[csukuangfj]

I believe double precision is more accurate for log_sum_exp.

Maybe @danpovey has more experience about this.

I have not compared the speed and accuracy between single and double precision.

[danpovey]

It was out of a concern that for long utterances, we might get roundoff errors being different in the forward vs backward computions, and posteriors that don't sum to 1, causing possible lack of cancellation between num and den.
However I don't recall whether this was an actual issue in practice. We should test the effect on speed and WER again.

[galv]

Okay. I definitely think it makes sense for long utterances to use double if we are in probability space. I will keep this in my mind as a potential knob to tune.

[danpovey]

They are log-probs not probs (overflow/underflow are not the issue).

…

On Thu, May 6, 2021 at 11:47 PM Daniel Galvez ***@***.***> wrote: Okay. I definitely think it makes sense for long utterances to use double if we are in probability space. I will keep this in my mind as a potential knob to tune. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#183 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLO3V3JDOGJP3WANNGVLTMK22NANCNFSM44GMSH4A> .

[galv]

Don't worry, I know that part (it would be concerning if I didn't!)