Contribution: section on implicit functions

[charlesm93]

Restating my post on https://discourse.mc-stan.org/t/help-write-an-autodiff-handbook/12640/13.

I propose grouping the chapters on ODEs, algebraic equations, and integrals, into one section on implicit functions. This is in part because for all these methods, we can apply the adjoint method to construct differentiation algorithms.

The content would be as follows:

Automatic differentiation for implicit functions

1. Algebraic equations: implicit function theorem, adjoint method, practical considerations
2. Optimization problems: extending results from the algebraic equations, equality constraint, inequality constraints, conditions for existence of derivatives and diagnostics
3. Ordinary differential equations: full Jacobian, adjoint method, tricks for users
4. Action of matrix exponential: adjoint method based on ODE case
5. Differential algebraic equations: adjoint method
6. Discrete sequences: adjoint method

We've already written a lot of this down, it's just matter of putting it together.

[charlesm93]

@bob-carpenter would it be possible to have rights to create a branch? Thank you!

[bob-carpenter]

OK. I invited you as a collaborator for this repo. You can always fork.

This all looks great. My only concern is that we keep the math to a level I can understand. I haven't been able to understand your adjoints paper yet.

[charlesm93]

Thank you.

I drafted the chapter on algebraic equations. Let me know if the tone and amount of details agrees with your goals for the book.

[bob-carpenter]

Cool. Thanks, I'll take a look.

[bob-carpenter]

This looks great. That's a presentation of the implicit function theorem that I can understand! So overall, the level's just what I was hoping for.

There are a few steps I could use help with around the Largrangians and a few things I do understand that I think can be defined more explicitly.

And we need to reach some consensus on notation or every chapter's going to look different.

I'd like to see the final results summarized in rules for adjoints (reverse mode) and tangents (forward-mode). Something where it'll be easy to look up the result without having unpack a bunch of formulas in narratives.

It's (partial f / partial u) that needs to be invertible in the statement.

Is j a conventional function name for adjoint methods? I'd rather stick to more usual function names.

[charlesm93]

I'd like to see the final results summarized in rules for adjoints (reverse mode) and tangents (forward-mode). Something where it'll be easy to look up the result without having unpack a bunch of formulas in narratives.

I can summarize the results in a table or a summary section at the end. The adjoint method is designed with reverse mode in mind, but I can work out the forward rules.

And we need to reach some consensus on notation or every chapter's going to look different.

Ok, I'll check what you've already written and adjust.

Is j a conventional function name for adjoint methods? I'd rather stick to more usual function names.

Sounds good.