Loss Function Used

[witnik]

Hi Shion,

I was trying to reproduce the model and realized that you used NLL loss in github yet it was mentioned in the paper to use Cross Entropy Loss. Would you mind suggesting which one of these loss function I should go further on my own?

Thanks

[shionhonda]

Cross entropy loss is equivalent to NLL loss following the log-softmax operation (see here for more detail).

My Transformer model includes log-softmax in the forward function, so it is correct to use NLL loss.

smiles-transformer/smiles_transformer/pretrain_trfm.py

Line 61 in 7ffa26e

out = F.log_softmax(out, dim=2) # (T,B,V)

To answer your question, it's up to your implementation. If your model outputs log of probability, use NLL loss. If it outputs logits, use cross entropy loss.