Make the trust region subsolver exchangeable. (#294)

* trust regions subsolver is now exchangeable.

Changelog.md

@@ -5,6 +5,16 @@ All notable Changes to the Julia package `Manopt.jl` will be documented in this
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.41] - 02/11/2023
+
+### Changed
+
+– `trust_regions` is now more flexible and the sub solver (Steinhaug-Toint tCG by default)
+  can now be exchanged.
+- `adaptive_regularization_with_cubics` is now more flexible as well, where it previously was a bit too
+  much tightened to the Lanczos solver as well.
+- Unified documentation notation and bumped dependencies to use DocumenterCitations 1.3
+
 ## [0.4.40] – 24/10/2023
 
 ### Added

Project.toml

@@ -1,7 +1,7 @@
 name = "Manopt"
 uuid = "0fc0a36d-df90-57f3-8f93-d78a9fc72bb5"
 authors = ["Ronny Bergmann <[email protected]>"]
-version = "0.4.40"
+version = "0.4.41"
 
 [deps]
 ColorSchemes = "35d6a980-a343-548e-a6ea-1d62b119f2f4"
@@ -38,13 +38,16 @@ ColorSchemes = "3.5.0"
 ColorTypes = "0.9.1, 0.10, 0.11"
 Colors = "0.11.2, 0.12"
 DataStructures = "0.17, 0.18"
+LinearAlgebra = "1.6"
 LRUCache = "1.4"
-ManifoldDiff = "0.2, 0.3.3"
+ManifoldDiff = "0.3.8"
 Manifolds = "0.9"
 ManifoldsBase = "0.15"
+Markdown = "1.6"
 PolynomialRoots = "1"
+Random = "1.6"
 Requires = "0.5, 1"
-Statistics = "1"
+Statistics = "1.6"
 julia = "1.6"
 
 [extras]

Readme.md

@@ -74,7 +74,24 @@ To refer to a certain version or the source code in general we recommend to cite
 ```
 
 for the most recent version or a corresponding version specific DOI, see [the list of all versions](https://zenodo.org/search?page=1&size=20&q=conceptrecid:%224290905%22&sort=-version&all_versions=True).
-Note that both citations are in [BibLaTeX](https://ctan.org/pkg/biblatex) format.
+
+If you are also using [`Manifolds.jl`](https://juliamanifolds.github.io/Manifolds.jl/stable/) please consider to cite
+
+```
+@article{AxenBaranBergmannRzecki:2023,
+    AUTHOR     = {Seth D. Axen and Mateusz Baran and Ronny Bergmann and Krzysztof Rzecki},
+    DOI        = {10.1145/3618296},
+    EPRINT     = {2021.08777},
+    EPRINTTYPE = {arXiv},
+    JOURNAL    = {AMS Transactions on Mathematical Software},
+    NOTE       = {accepted for publication},
+    TITLE      = {Manifolds.jl: An Extensible {J}ulia Framework for Data Analysis on Manifolds},
+    YEAR       = {2023}
+}
+```
+
+as well.
+Note that all citations are in [BibLaTeX](https://ctan.org/pkg/biblatex) format.
 
 ## Further and Similar Packages & Links
 

docs/make.jl

@@ -108,7 +108,7 @@ tutorials_menu =
 bib = CitationBibliography(joinpath(@__DIR__, "src", "references.bib"); style=:alpha)
 makedocs(;
     format=Documenter.HTML(;
-        mathengine=MathJax3(), prettyurls=get(ENV, "CI", nothing) == "true"
+        prettyurls=false, assets=["assets/favicon.ico", "assets/citations.css"]
     ),
     modules=[
         Manopt,

docs/src/assets/citations.css

@@ -0,0 +1,19 @@
+/* Taken from https://juliadocs.org/DocumenterCitations.jl/v1.2/styling/ */
+
+.citation dl {
+  display: grid;
+  grid-template-columns: max-content auto; }
+.citation dt {
+  grid-column-start: 1; }
+.citation dd {
+  grid-column-start: 2;
+  margin-bottom: 0.75em; }
+.citation ul {
+ padding: 0 0 2.25em 0;
+ margin: 0;
+ list-style: none;}
+.citation ul li {
+ text-indent: -2.25em;
+ margin: 0.33em 0.5em 0.5em 2.25em;}
+.citation ol li {
+ padding-left:0.75em;}

docs/src/functions/adjoint_differentials.md

@@ -8,6 +8,6 @@ Pages   = ["adjoint_differentials.jl"]
 # Literature
 
 ```@bibliography
-Pages = ["functions/adjoint_differentials.md"]
+Pages = ["adjoint_differentials.md"]
 Canonical=false
 ```

docs/src/functions/bezier.md

@@ -8,6 +8,6 @@ Pages   = ["bezier_curves.jl"]
 # Literature
 
 ```@bibliography
-Pages = ["functions/bezier.md"]
+Pages = ["bezier.md"]
 Canonical=false
 ```

docs/src/functions/gradients.md

@@ -1,13 +1,12 @@
 # [Gradients](@id GradientFunctions)
 
-For a function $f:\mathcal M→ℝ$
-the Riemannian gradient $\operatorname{grad}f(x)$ at $x∈\mathcal M$
+For a function ``f:\mathcal M→ℝ``
+the Riemannian gradient ``\operatorname{grad}f(x)`` at ``x∈\mathcal M``
 is given by the unique tangent vector fulfilling
-
-$\langle \operatorname{grad}f(x), ξ\rangle_x = D_xf[ξ],\quad
-\forall ξ ∈ T_x\mathcal M,$
-where $D_xf[ξ]$ denotes the differential of $f$ at $x$ with respect to
-the tangent direction (vector) $ξ$ or in other words the directional
+``\langle \operatorname{grad}f(x), ξ\rangle_x = D_xf[ξ],\quad
+\forall ξ ∈ T_x\mathcal M,``
+where ``D_xf[ξ]`` denotes the differential of ``f`` at ``x`` with respect to
+the tangent direction (vector) ``ξ`` or in other words the directional
 derivative.
 
 This page collects the available gradients.
@@ -20,6 +19,6 @@ Pages   = ["gradients.jl"]
 # Literature
 
 ```@bibliography
-Pages = ["functions/gradients.md"]
+Pages = ["gradients.md"]
 Canonical=false
 ```

docs/src/functions/proximal_maps.md

@@ -29,6 +29,6 @@ Pages   = ["proximal_maps.jl"]
 # Literature
 
 ```@bibliography
-Pages = ["functions/proximal_maps.md"]
+Pages = ["proximal_maps.md"]
 Canonical=false
 ```

docs/src/helpers/checks.md

@@ -10,6 +10,6 @@ Pages   = ["checks.jl"]
 ## Literature
 
 ```@bibliography
-Pages = ["helpers/checks.md"]
+Pages = ["checks.md"]
 Canonical=false
 ```

docs/src/helpers/data.md

@@ -13,6 +13,6 @@ Pages   = ["artificialDataFunctions.jl"]
 # Literature
 
 ```@bibliography
-Pages = ["helpers/data.md"]
+Pages = ["data.md"]
 Canonical=false
 ```

docs/src/plans/index.md

@@ -17,5 +17,30 @@ Still there might be the need to set certain parameters within any of these stru
 
 ```@docs
 set_manopt_parameter!
+get_manopt_parameter
 Manopt.status_summary
-```
+```
+
+Where the following Symbols are used
+
+The following symbols are used.
+The column “generic” refers to a short hand that might be used – for readability if clear from context.
+
+| Symbol       | Used in | Description                                                | generic |
+| :----------- | :------: | ;-------------------------------------------------------- | :------ |
+| `:active` | [`DebugWhenActive`](@ref) | activity of the debug action stored within | |
+| `:Basepoint` | [`TangentSpace`]() | the point the tangent space is at           | `:p` |
+| `:Cost` | generic |the cost function (e.g. within an objective, as pass down) | |
+| `:Debug` | [`DebugSolverState`](@ref) | the stored `debugDictionary` | |
+| `:Gradient` | generic |the gradient function (e.g. within an objective, as pass down) | |
+| `:Iterate` | generic | the (current) iterate – similar to [`set_iterate!`](@ref) – within a state | |
+| `:Manifold` | generic |the manifold (e.g. within a problem, as pass down) | |
+| `:Objective` | generic | the objective (e.g. within a problem, as pass down) | |
+| `:SubProblem` | generic | the sub problem (e.g. within a state, as pass down) | |
+| `:SubState` | generic | the sub state (e.g. within a state, as pass down) | |
+| `:λ` | [`ProximalDCCost`](@ref), [`ProximalDCGrad`](@ref) | set the proximal parameter within the proximal sub objective elements | |
+| `:p`         | generic | a certain point         | |
+| `:X`         | generic | a certain tangent vector | |
+| `:TrustRegionRadius` | [`TrustRegionsState`](@ref) | the trust region radius | `:σ` |
+| `:ρ`, `:u` | [`ExactPenaltyCost`](@ref), [`ExactPenaltyGrad`](@ref) | Parameters within the exact penalty objetive | |
+| `:ρ`, `:μ`, `:λ` | [`AugmentedLagrangianCost`](@ref) and [`AugmentedLagrangianGrad`](@ref) | Parameters of the Lagrangian function | |

docs/src/plans/objective.md

@@ -160,10 +160,10 @@ ManifoldProximalMapObjective
 get_proximal_map
 ```
 
-
 ### Hessian Objective
 
 ```@docs
+AbstractManifoldHessianObjective
 ManifoldHessianObjective
 ```
 
@@ -234,3 +234,21 @@ get_grad_inequality_constraint!
 get_grad_inequality_constraints
 get_grad_inequality_constraints!
 ```
+
+### Subproblem Objective
+
+This objective can be use when the objective of a sub problem
+solver still needs access to the (outer/main) objective.
+
+```@docs
+AbstractManifoldSubObjective
+```
+
+#### Access functions
+
+```@docs
+Manopt.get_objective_cost
+Manopt.get_objective_gradient
+Manopt.get_objective_hessian
+Manopt.get_objective_preconditioner
+```

docs/src/plans/stepsize.md

@@ -29,6 +29,6 @@ Filter = t -> t != Stepsize
 ## Literature
 
 ```@bibliography
-Pages = ["plans/stepsize.md"]
+Pages = ["stepsize.md"]
 Canonical=false
 ```

docs/src/references.bib

@@ -203,7 +203,7 @@ @book{Boumal:2023
     EDITION   = {First},
     PUBLISHER = {Cambridge University Press},
     DOI       = {10.1017/9781009166164},
-    URLDATE   = {2023-07-13},
+    NOTE      = {Homepage to the book: [nicolasboumal.net/book/index.html](https://www.nicolasboumal.net/book/index.html).},
     ABSTRACT  = {Optimization on Riemannian manifolds-the result of smooth geometry and optimization merging into one elegant modern framework-spans many areas of science and engineering, including machine learning, computer vision, signal processing, dynamical systems and scientific computing. This text introduces the differential geometry and Riemannian geometry concepts that will help students and researchers in applied mathematics, computer science and engineering gain a firm mathematical grounding to use these tools confidently in their research. Its charts-last approach will prove more intuitive from an optimizer's viewpoint, and all definitions and theorems are motivated to build time-tested optimization algorithms. Starting from first principles, the text goes on to cover current research on topics including worst-case complexity and geodesic convexity. Readers will appreciate the tricks of the trade for conducting research and for numerical implementations sprinkled throughout the book.},
     ISBN      = {978-1-00-916616-4}
 }
@@ -215,7 +215,7 @@ @book{doCarmo:1992
     TITLE     = {Riemannian Geometry},
     AUTHOR    = {do Carmo, Manfredo Perdigão},
     YEAR      = {1992},
-    SERIES    = {Mathematics: Theory & Applications},
+    SERIES    = {Mathematics: Theory \& Applications},
     PAGES     = {xiv+300},
     PUBLISHER = {Birkhäuser Boston, Inc., Boston, MA},
     DOI       = {10.1007/978-1-4757-2201-7},
@@ -266,7 +266,7 @@ @article{DaiYuan:1999
     MONTH     = jan,
     NUMBER    = {1},
     PAGES     = {177--182},
-    PUBLISHER = {Society for Industrial & Applied Mathematics (SIAM)},
+    PUBLISHER = {Society for Industrial \& Applied Mathematics (SIAM)},
     TITLE     = {A Nonlinear Conjugate Gradient Method with a Strong Global Convergence Property},
     VOLUME    = {10},
     YEAR      = {1999}
@@ -279,7 +279,7 @@ @article{DiepeveenLellmann:2021
     NOTE      = {arXiv: [2102.10309](https://arxiv.org/abs/2102.10309)},
     NUMBER    = {4},
     PAGES     = {1565--1600},
-    PUBLISHER = {Society for Industrial & Applied Mathematics ({SIAM})},
+    PUBLISHER = {Society for Industrial \& Applied Mathematics ({SIAM})},
     TITLE     = {An Inexact Semismooth Newton Method on Riemannian Manifolds with Application to Duality-Based Total Variation Denoising},
     VOLUME    = {14},
     YEAR      = {2021},
@@ -307,7 +307,7 @@ @book{Fletcher:1987
     AUTHOR    = {Fletcher, R.},
     EDITION   = {2},
     LOCATION  = {Chichester},
-    PUBLISHER = {John Wiley & Sons Ltd.},
+    PUBLISHER = {John Wiley \& Sons Ltd.},
     SERIES    = {A Wiley-Interscience Publication},
     TITLE     = {Practical Methods of Optimization},
     YEAR      = {1987}
@@ -358,7 +358,7 @@ @article{HagerZhang:2005
     MONTH     = jan,
     NUMBER    = {1},
     PAGES     = {170--192},
-    PUBLISHER = {Society for Industrial & Applied Mathematics (SIAM)},
+    PUBLISHER = {Society for Industrial \& Applied Mathematics (SIAM)},
     TITLE     = {A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search},
     VOLUME    = {16},
     YEAR      = {2005}
@@ -425,7 +425,7 @@ @article{IannazzoPorcelli:2017
     NUMBER    = {1},
     PAGES     = {495--517},
     PUBLISHER = {Oxford University Press ({OUP})},
-    TITLE     = {The Riemannian Barzilai{\textendash}Borwein method with nonmonotone line search and the matrix geometric mean computation},
+    TITLE     = {The Riemannian Barzilai–Borwein method with nonmonotone line search and the matrix geometric mean computation},
     VOLUME    = {38},
     YEAR      = {2017},
 }
@@ -467,7 +467,7 @@ @article{LiuBoumal:2019
     YEAR      = {2019},
     MONTH     = mar,
     DOI       = {10.1007/s00245-019-09564-3},
-    JOURNAL   = {Applied Mathematics & Optimization},
+    JOURNAL   = {Applied Mathematics \& Optimization},
     TITLE     = {Simple algorithms for optimization on Riemannian manifolds with constraints},
     URL       = {https://arxiv.org/abs/1901.10000}
 }

docs/src/solvers/ChambollePock.md

@@ -3,31 +3,34 @@
 The Riemannian Chambolle–Pock is a generalization of the Chambolle–Pock algorithm [ChambollePock:2011](@citet*)
 It is also known as primal-dual hybrid gradient (PDHG) or primal-dual proximal splitting (PDPS) algorithm.
 
-In order to minimize over $p∈\mathcal M$ the cost function consisting of
+In order to minimize over ``p∈\mathcal M`` the cost function consisting of
+In order to minimize a cost function consisting of
 
 ```math
 F(p) + G(Λ(p)),
 ```
 
-where $F:\mathcal M → \overline{ℝ}$, $G:\mathcal N → \overline{ℝ}$, and
-$Λ:\mathcal M →\mathcal N$.
-If the manifolds $\mathcal M$ or $\mathcal N$ are not Hadamard, it has to be considered locally,
-i.e. on geodesically convex sets $\mathcal C \subset \mathcal M$ and $\mathcal D \subset\mathcal N$
-such that $Λ(\mathcal C) \subset \mathcal D$.
+ over ``p∈\mathcal M``
+
+where ``F:\mathcal M → \overline{ℝ}``, ``G:\mathcal N → \overline{ℝ}``, and
+``Λ:\mathcal M →\mathcal N``.
+If the manifolds ``\mathcal M`` or ``\mathcal N`` are not Hadamard, it has to be considered locally,
+i.e. on geodesically convex sets ``\mathcal C \subset \mathcal M`` and ``\mathcal D \subset\mathcal N``
+such that ``Λ(\mathcal C) \subset \mathcal D``.
 
 The algorithm is available in four variants: exact versus linearized (see `variant`)
 as well as with primal versus dual relaxation (see `relax`). For more details, see
 [BergmannHerzogSilvaLouzeiroTenbrinckVidalNunez:2021](@citet*).
 In the following we note the case of the exact, primal relaxed Riemannian Chambolle–Pock algorithm.
 
-Given base points $m∈\mathcal C$, $n=Λ(m)∈\mathcal D$,
-initial primal and dual values $p^{(0)} ∈\mathcal C$, $ξ_n^{(0)} ∈T_n^*\mathcal N$,
-and primal and dual step sizes $\sigma_0$, $\tau_0$, relaxation $\theta_0$,
-as well as acceleration $\gamma$.
+Given base points ``m∈\mathcal C``, ``n=Λ(m)∈\mathcal D``,
+initial primal and dual values ``p^{(0)} ∈\mathcal C``, ``ξ_n^{(0)} ∈T_n^*\mathcal N``,
+and primal and dual step sizes ``\sigma_0``, ``\tau_0``, relaxation ``\theta_0``,
+as well as acceleration ``\gamma``.
 
-As an initialization, perform $\bar p^{(0)} \gets p^{(0)}$.
+As an initialization, perform ``\bar p^{(0)} \gets p^{(0)}``.
 
-The algorithms performs the steps $k=1,…,$ (until a [`StoppingCriterion`](@ref) is fulfilled with)
+The algorithms performs the steps ``k=1,…,`` (until a [`StoppingCriterion`](@ref) is fulfilled with)
 
 1. ```math
    ξ^{(k+1)}_n = \operatorname{prox}_{\tau_k G_n^*}\Bigl(ξ_n^{(k)} + \tau_k \bigl(\log_n Λ (\bar p^{(k)})\bigr)^\flat\Bigr)
@@ -46,9 +49,9 @@ The algorithms performs the steps $k=1,…,$ (until a [`StoppingCriterion`](@ref
 Furthermore you can exchange the exponential map, the logarithmic map, and the parallel transport
 by a retraction, an inverse retraction, and a vector transport.
 
-Finally you can also update the base points $m$ and $n$ during the iterations.
+Finally you can also update the base points ``m`` and ``n`` during the iterations.
 This introduces a few additional vector transports. The same holds for the case
-$Λ(m^{(k)})\neq n^{(k)}$ at some point. All these cases are covered in the algorithm.
+``Λ(m^{(k)})\neq n^{(k)}`` at some point. All these cases are covered in the algorithm.
 
 ```@meta
 CurrentModule = Manopt
@@ -110,6 +113,6 @@ Manopt.update_prox_parameters!
 ## Literature
 
 ```@bibliography
-Pages = ["solvers/ChambollePock.md"]
+Pages = ["ChambollePock.md"]
 Canonical=false
-```
+```

docs/src/solvers/DouglasRachford.md

@@ -65,5 +65,5 @@ For specific [`DebugAction`](@ref)s and [`RecordAction`](@ref)s see also
 ## Literature
 
 ```@bibliography
-Pages = ["solvers/DouglasRachford.md"]
+Pages = ["DouglasRachford.md"]
 ```