Bias path is not working correctly

[umarbutler]

Hi,
I have observed that including bias seems to cause the loss produced by my models to diverge significantly from the losses produced without cut cross-entropy. They tend to be much, much lower, often in the negative ranges. Omitting bias seems to improve things dramatically, such that losses are within a range of +/-0.05 of their originals.

[erikwijmans]

What implementation is this with? Can you try running with impl="cce_kahan_full_c" or impl="cce_kahan_full_c_full_e"? Maybe the bias is negatively impacting gradient filtering.

[umarbutler]

I was using cce_kahan_full_c. FYI, I switched to Liger Kernel's fused cross entropy loss and did not have the same issue.

[erikwijmans]

I found a bug in the bias code. It was unfortunately something the tests weren't catching. I pushed a fix for that bug in #53