convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py does not work due to weight norm key

[gau-nernst]

System Info

NA

Who can help?

@sanchit-gandhi

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Fairseq uses nn.utils.weight_norm(), which produces weight_g and weight_v keys

https://github.com/facebookresearch/fairseq/blob/d9a627082fd03ec72a27a31a4e56289bfcb2e4e4/fairseq/models/wav2vec/wav2vec2.py#L939

The weight conversion script assumes this old-style weight norm

transformers/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py

Lines 118 to 121 in 2daf2c3

	elif weight_type == "weight_g":
	hf_pointer.weight_g.data = value
	elif weight_type == "weight_v":
	hf_pointer.weight_v.data = value

Thus, the weight conversion script will fail since transformers has switched to nn.utils.parametrizations.weight_norm for quite some time, which produces parametrizations.weight.original0 and parametrizations.weight.original1 keys instead.

transformers/src/transformers/models/wav2vec2/modeling_wav2vec2.py

Lines 392 to 394 in 1218e43

	weight_norm = nn.utils.weight_norm
	if hasattr(nn.utils.parametrizations, "weight_norm"):
	weight_norm = nn.utils.parametrizations.weight_norm

Thus the weight conversion script should be updated accordingly.

Expected behavior

Weight conversion script works.

[amyeroberts]

cc @sanchit-gandhi

[gau-nernst]

I had something working for me. f you don't have time to look into it, I can open a PR. I can provide a snippet to test it with some original fairseq checkpoints too. I

[amyeroberts]

@gau-nernst Yes please, that would be great cc @kamilakesbi too :)