At this point I'm using MSE or Huber loss to train the net, but suppose two examples:
q=.9, prediction q^=.7q=.1, prediction q^=-.1
In both cases any kind of widely used errors would give the same loss. However, it is clear that case two is a huge problem, while the first case is quite OK - the engine is probably objectively winning in both cases.
This needs a careful consideration and SPRT testing setup, though, so deferring until the engine + eval is mature enough.
At this point I'm using MSE or Huber loss to train the net, but suppose two examples:
q=.9, predictionq^=.7q=.1, predictionq^=-.1In both cases any kind of widely used errors would give the same loss. However, it is clear that case two is a huge problem, while the first case is quite OK - the engine is probably objectively winning in both cases.
This needs a careful consideration and SPRT testing setup, though, so deferring until the engine + eval is mature enough.