Training with custom CTC topology (with no blanks)

[desh2608]

I am trying to train an icefall model with phones as output units with a custom topology which resembles the "modified" CTC topology in k2, but without the blank symbol --> let's call this no-blank CTC. The idea is that instead of the "peaky" behavior that CTC shows, removing blank would force the phones to be better aligned with the acoustic frames.

I created the no-blank CTC topology, converted my texts into phone IDs, and then obtained the graph as follows:

transcript_fsa = k2.linear_fsa(token_ids, self.device)
transcript_fsa_with_self_loops = k2.arc_sort(
    k2.add_epsilon_self_loops(transcript_fsa)
)

res = k2.compose(
    self.ctc_topo,
    transcript_fsa_with_self_loops,
    treat_epsilons_specially=False,
)
res = k2.arc_sort(res)

Since I don't have a blank symbol, I created the nnet with only as many outputs as I have phone tokens. However, when I started training this with k2.ctc_loss(), I get infinity loss. This is not a training issue because I get this right at the start, i.e., when I compute the validation loss at the start. This suggests that the problem is with the arc scores in the composition most likely. On looking around in k2, I found the following:

k2/k2/python/k2/autograd.py

Line 796 in 42e92fd

# The first column of b_fsas.scores is -inf,

Why is the first column of the dense FSA always negative infinity? Also, if I want to train with such a topology, are there other changes which may be needed?

[pkufool]

Why is the first column of the dense FSA always negative infinity?

The first column is designed for final arc (label = -1) in k2 fsa, see

k2/k2/python/k2/dense_fsa_vec.py

Lines 24 to 41 in 42e92fd

	class DenseFsaVec(object):
	# Note: you can access members self.scores and self.dense_fsa_vec.
	# self.scores is a torch.Tensor containing the scores; it will
	# contain rows of the `log_probs` arg given to __init__ interspersed
	# with rows representing final-arcs. The structure is something like:
	#
	# [ [ -inf x x x x x x ]
	# [ -inf x x x x x x ]
	# [ -inf x x x x x x ]
	# [ 0 -inf -inf -inf.. ]
	# [ -inf x x x x x x ]
	# ...
	# ]
	# where the x's come from the `log_probs` arg, and the 0's and
	# -inf's are added by this class (those special rows with no x's
	# correspond to the final-arcs in the FSAs; the 0 corresponds to
	# symbol -1.)

[GCQiu]

hi， i am working this problem now, do you now how to train a ctc model without blank?

[danpovey]

You could maybe fake it by setting the blank logprob to -inf before the log_softmax, it may have the same effect.

…

On Mon, Jul 29, 2024, 8:23 PM GCQiu ***@***.***> wrote: hi， i am working this problem now, do you now how to train a ctc model without blank? — Reply to this email directly, view it on GitHub <#1222 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLOYYUGRIGDNACDVHELTZO4BMVAVCNFSM6AAAAABLVO6DK2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJXGM4DGMRRGU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>