Check weights and gradients for anomalies

[pasky]

We are maybe a little careless in the way we train the more complex (CNN, RNN etc.) models, in that we should carefully check the gradients and also the actual weight matrices; for CNN, some papers renormalize weights if their norm is too large, for RNN something similar might be necessary. I suspect that since we are getting reasonable-looking results, it's probably not a crucial issue, nevertheless we might get some improvements from deeply understanding the practical progression of training in our models.

[pasky]

http://www.lab41.org/taking-keras-to-the-zoo/ suggests that high dropout might contribute to vanishing gradient. This could explain why on Ubuntu Dialogue with very long sentences (spad=160), dropout=0 is much worse than high dropout that is beneficial for AnsSentSel (spad=80). Should be easy to verify experimentally.