Can we efficiently use neural tangent kernel?

[breuderink]

Can we use neural tangent kernel [1] to get the benefits of new neural net architectures while using the budgeted kernel machines? See also the paper on the path kernel [2].

The kernel needs to compute the dot product of the gradient of a neural network fed with two different inputs. Perhaps we can efficiently compute this dot product?

[1] Jacot, Arthur, Franck Gabriel, and Clément Hongler. "Neural tangent kernel: Convergence and generalization in neural networks." arXiv preprint arXiv:1806.07572 (2018).
[2] Domingos, Pedro M.. “Every Model Learned by Gradient Descent Is Approximately a Kernel Machine.” ArXiv abs/2012.00152 (2020): n. pag.

[breuderink]

To use the neural tangent kernel, we need to compute the dot product of the gradients of a neural network with on different inputs. Perhaps we can use k(a, b) = Tr(AB) = Tr(BA), where A is the product of the Jacobians for the neural net with input a, and B for input b. That way, we might be able to compute the kernel layer by layer without storing all gradients.