AD through warp

[SomTambe]

This issue is for tracking the progress of enabling AD through ImageTransformations.warp.
author: @SomTambe
mentors: @DhairyaLGandhi @johnnychen94

Background

ImageTransformations.jl implements a backward mode warp, which performs the exact operations as elaborated in #11 . The target would be to get the gradient with respect to the input warp matrix W, and as a long term goal, also the input image X.

Roadmap

• Implement adjoints for _getindex
• Implement adjoints for BSplines and Extrapolations.
• Implement adjoints for SMatrix and SVector.
• Write rrule for homography struct.
• Write tests to check individual gradients in pipeline using FiniteDifferences.
• Write tests to check gradient of whole pipeline using FiniteDifferences.
• Make sure the index-wise gradients are correct.
• Make sure the individual gradients are correct in the pipeline.
• Make sure accumulated gradients for the whole image are correct.
• Train the parameters of a homography matrix using the local warp function.
• Extend the current adjoints to write a complete adjoint for the ImageTransformations.warp!
• Train the parameters of a homography matrix using the ImageTransformations.warp function.

[DhairyaLGandhi]

Make sure the index-wise gradients are correct.
Make sure the individual gradients are correct in the pipeline.
Make sure accumulated gradients for the whole image are correct.

Can we get tests in for these

[SomTambe]

yeah sure, I'm in the process of making the new branch and creating a PR

[johnnychen94]

Ad though warp task can be separated into at least four steps:

1. AD through the backward coordinate map ϕ. Homography matrix and other existing types from CoordinateTransformations are good candidates for this.
2. AD though the value estimator τ. Since we use the interpolation/extrapolation family, we need to interact with Interpolations.jl properly and find out potential missing items.
3. verify the warp operation on single-pixel works
4. composite the high-order warp on the full image scale

Each step should correspond to at least one PR.

You don't need to support everything in one PR, for example, SVector support can be delayed when all these are done, SVector is mainly for performance consideration.

---

Again, and again, do keep in mind, every non-trivial PR should have its associated test cases. If the PR is submitted without test cases, it requires the reviews to understand every detail of the code, which isn't always the case. Approval from reviewers doesn't mean the code is written correctly, it only means 1) the codes and implementation don't look suspicious and 2) the codes are written in a good style. Correctness is guaranteed by test cases and CI. PRs without high test coverage usually don't get high-quality review feedback because reviewers don't want to spend their time investigating something that can't be verified very quickly.

There might be some misunderstanding here on our mentorship here, because everyone here contributes with their free time from ❤️, one of the golden rules (I said) in open source development is to save others' time. Submitting a good-to-review PR, opening a well-explained and reproducible issue, reducing unnecessary multi-rounds communications, are some practices of this rule.