Implement Pytorch Metrics

[bonham79]

TorchMetrics support is pretty reliable nowadays and makes distributed training less annoying (no more World sizes, yay!). It also syncs well with Wandb logging and allows monitoring of training batch performance. Any complaints about me migrating validation logging to this?

[kylebgorman]

I don't know anything about it but it sounds like a positive.

[bonham79]

It's just a suite of metrics that torch sets up to manage multi-gpus under the hood. You just pass it your tensors during the training loop and it will store submetrics. Then when you need the actual metric you call and it does the calculation and memory collection under the hood. It saves you from desync issues if you have more than GPU for training. Also PTL supports it so it helps reduce boilerplate for other metrics.

[Adamits]

I think I tried to do this last year, and had some issues getting the features I actually wanted from it to track, so I gave up, but I think it was in beta or something then. I'd be happy if you got it working :)

[bonham79]

I just got it running for BLEU scores on another project, so CER/WER should be stable by now.

[kylebgorman]

I piloted this a bit. We can do our form of accuracy using torchmetrics.classification.MulticlassExactMatch, but they don't have an equivalent of our SER, just edit distance limited to strings. (So we'd have to convert back to strings, add all the necessary bookkeeping of letting the models see the indexes, and we wouldn't really be doing "SER" anymore either.) I filed a feature request but probably the best solution is to just use the same API they do to implement SER rather than submitting it to their library.

[bonham79]

Yeah, they have a bias towards strings as final output, it's a bit annoying since you need to do metric calculations on CPU at terminus. It's somewhat painless to implement new metrics so long as the distributed training is managed properly. I support implementing it on our side and then once it's robust enough we can push to library if they ever allow more robust metric balancing.