Sparse differentiation for matrices with a few dense rows and columns

[mjohnson541]

For Jacobians that are almost sparse except for a couple rows and columns that are entirely dense it'd be handy to be able to exploit the sparsity by calculating the dense rows with reverse diff and the columns independently of the dense rows. Continuing discussion from ReactionMechanismGenerator/ReactionMechanismSimulator.jl#28.

[ChrisRackauckas]

The general mixing problem is a bit different ("a bit" is an understatement), but I have been seeing a general flow where there are N dense rows mixed with a sparse matrix. In this case, you could calculate those N rows with N reverse-diff calls, and then color the rest of the matrix. This would be a nice specialized function to have in the library for things like spectral discretizations.

[mjohnson541]

So we've been looking at a simple system for our application where we would feed in the indices of the dense rows, modify the sparsity pattern to clear those rows, generate the color vector assuming those rows are zeros, compute the jacobian using sparse forward diff with the sparse color vector and then replace the (incorrect) dense rows of that jacobian with rows computed using reverse mode auto diff. @hwpang

Would any of that belong here?

[ChrisRackauckas]

All of that most definitely belongs here! I think, as a first step, reverse-mode sparse differentiation should be the first goal. Setting it up with ReverseDiff and Zygote would be a good idea. Same interface, i.e. with a sparsity and colorvec, except requiring that the colorvec is generated on the row space via the adjoint of the Jacobian sparsity pattern.

I think once that's done, a smart routine that does mixed mode should get worked out. I think we want to handle things like "the first row + tridiagonal", in which case we'd have a full colorvec for forward mode (for the tridiagonal) and some way to specify to just do e1, and then piece that together. Exactly what that API looks like is unclear to me right now, so I'd just make sure the reverse mode version is done first.