Skip to content

Commit e1950d5

Browse files
authored
Guard the power-mean relaxation factor estimate (#88)
The asymmetric power-mean sweeps overrelax by a factor `ω` estimated from plain sweeps and raised from the rates of later windows. Two defects in that rule made some solves far slower than plain sweeps. An `ω` set on a plateau was never lowered. Far from the fixed point the imbalance can stay nearly constant for several sweeps before it contracts quickly. An `ω` estimated there is near its cap, and above the optimum the relaxed rate is `ω - 1` whatever the plain rate is, so the windows read it as steady progress. The plain estimate now waits while the contraction is still improving from one sweep to the next, and sets `ω` no higher than 1.5, where a relaxed sweep contracts by at least half even above the optimum. Windows raise it from there. A window was judged against the smallest imbalance of all earlier windows. The imbalance is the size of the plain step, which a relaxed iterate overshoots by `ω - 1` times, so its level rises with `ω`, and windows at a raised `ω` could show no progress while the iteration converged. Those windows tripped the stall rule, which halves `ω - 1` when it should rise. Each window is now judged against the window before it. The first window after `ω` changes is still judged against one at the old `ω` and reads as slow, so one window shrinks `2 - ω` by at most a factor of 8. The handoff to Newton after the plain estimate is predicted at the rate of the factor that estimate implies, as before, not at the capped factor the first relaxed sweeps use. On 1200 random matrices of 2 to 8 rows and columns (lognormal entries with σ of 1, 3 or 6, up to half zeroed), with `p` of 1 and 2 and the default and flat starts, 4800 runs in all. With `newton = false`, on the 4663 runs that plain sweeps finish in 50,000: | | before | after | |---|---|---| | sweeps | 552,513 | 254,336 | | runs more than 2 sweeps slower than plain | 213 | 9 | | worst ratio to plain | 85 | 1.73 | | runs where the stall rule fires (of 4800) | 55 | 3 | | runs plain does not finish that the rule does | 87 of 137 | 101 of 137 | The 3 remaining stalls are runs that neither rule finishes in 10^5 sweeps. With `newton = true`: | | before | after | |---|---|---| | runs handed to Newton | 949 | 974 | | passes over the support | 317,998 | 302,284 | | factorizations | 5130 | 5227 | Sweeps with `newton = false` on larger problems, plain / before / after: | problem | plain | before | after | |---|---|---|---| | bidiagonal n=100, flat start, p=1 | 97,194 | 6114 | 6466 | | bidiagonal n=100, flat start, p=2 | >10^5 | 7610 | 7651 | | cyclic bidiagonal n=50, p=1 | 19,630 | 2063 | 739 | | cyclic bidiagonal n=200, p=2 | >10^5 | 27,464 | 27,459 | | cyclic tridiagonal n=100, p=2 | 82,699 | 940 | 940 | | integral equation n=100, p=1 | >10^5 | 16,220 | 2217 | | sparse lognormal 200x200 3%, flat start, p=2 | 3557 | 223 | 241 | Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent cb4bc5e commit e1950d5

2 files changed

Lines changed: 89 additions & 27 deletions

File tree

‎src/powermean.jl‎

Lines changed: 52 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -17,15 +17,21 @@ const POWERMEAN_MAXITER = 10_000
1717
const POWERMEAN_TOL_EPS = 4096
1818

1919
# Overrelaxation: the number of plain sweeps whose imbalance contraction gives the
20-
# first estimate of the rate; the window (in sweeps) over which a relaxed rate is
21-
# measured, which lengthens as `ω → 2` because the rate per sweep is at best
20+
# first estimate of the rate; the factor by which the log of that contraction may
21+
# grow from one sweep's estimate to the next and still count as steady; the
22+
# largest `ω` a plain estimate may set; the window (in sweeps) over which a relaxed
23+
# rate is measured, which lengthens as `ω → 2` because the rate per sweep is at best
2224
# `ω - 1`, so one factor of `e` takes about `1/(2 - ω)` sweeps, and the oscillating
23-
# imbalance shows no net decrease over shorter windows; the number of consecutive
24-
# windows without progress that halve `ω - 1`; the number of halvings a single
25-
# group's relaxed step may try; and the largest relaxation factor used.
25+
# imbalance shows no net decrease over shorter windows; the largest factor by
26+
# which one window may shrink `2 - ω`; the number of consecutive windows without
27+
# progress that halve `ω - 1`; the number of halvings a single group's relaxed
28+
# step may try; and the largest relaxation factor used.
2629
const POWERMEAN_NESTIMATE = 4
30+
const POWERMEAN_STEADY = 1.25
31+
const POWERMEAN_OMEGA_START = 1.5
2732
const POWERMEAN_WINDOW = 16
2833
const POWERMEAN_WINDOW_SCALE = 4.0
34+
const POWERMEAN_RAISE = 8
2935
const POWERMEAN_NSTALL = 4
3036
const POWERMEAN_NBACKTRACK = 4
3137
const POWERMEAN_OMEGA_MAX = 1.995
@@ -339,18 +345,35 @@ end
339345
# plain rate per sweep is `σ²` (the second singular value squared of the
340346
# normalized weight matrix) and whose optimal overrelaxation is
341347
# `ω* = 2/(1 + sqrt(1 - σ²))`, with rate `ω* - 1`. Plain sweeps give a first
342-
# estimate of `σ²` from the contraction of the imbalance. Each relaxed window
343-
# re-estimates it from the observed rate `λ` through the SOR eigenvalue relation
344-
# `sqrt(λ) = (ωμ + sqrt(ω²μ² - 4(ω - 1)))/2`, which raises `ω` when it is below
345-
# `ω*`. The imbalance of a relaxed iteration is not monotone, so progress is
346-
# judged by the smallest imbalance within each window; after several consecutive
347-
# windows without progress `ω - 1` is halved, down to plain sweeps and a fresh
348-
# estimate.
348+
# estimate of `σ²` from the contraction of the imbalance.
349+
#
350+
# Above `ω*` the relaxed rate is `ω - 1` whatever `σ²` is, so the windows below
351+
# read an `ω` that is too large as steady progress and never lower it. The plain
352+
# estimate is the one that can be too large: far from the fixed point the
353+
# imbalance can stay nearly constant for several sweeps before it contracts at a
354+
# much faster rate. Two conditions on the estimate guard against that. It waits
355+
# while the contraction is still improving from one sweep to the next, and it
356+
# sets `ω` no higher than `POWERMEAN_OMEGA_START`, where a relaxed sweep contracts
357+
# by at least half even above `ω*`. Only a window's measured rate takes `ω` higher.
358+
#
359+
# Each relaxed window re-estimates `σ²` from its observed rate `λ` through the SOR
360+
# eigenvalue relation `sqrt(λ) = (ωμ + sqrt(ω²μ² - 4(ω - 1)))/2`, which raises `ω`
361+
# when it is below `ω*`. The imbalance of a relaxed iteration is not monotone, so
362+
# a window is represented by its smallest imbalance, and its rate is measured
363+
# against the window before it. The imbalance is the size of the plain step, which
364+
# a relaxed iterate overshoots by `ω - 1` times, so its level rises with `ω`:
365+
# against the smallest imbalance of all earlier windows, windows at a raised `ω`
366+
# could show no progress while the iteration converges. The first window after
367+
# `ω` changes is still measured against one at the old `ω`, whose lower level
368+
# makes it read as slow, so one window shrinks `2 - ω` by at most a factor of
369+
# `POWERMEAN_RAISE`. After several consecutive windows without progress `ω - 1` is
370+
# halved, down to plain sweeps and a fresh estimate.
349371
#
350372
# With `budget`, the iteration also returns early, unconverged, once the rate
351373
# estimates predict that more than `budget` further sweeps are needed: from the
352-
# plain estimate, at the rate of the relaxed iteration it implies, and at the end
353-
# of each relaxed window, at the observed rate.
374+
# plain estimate, at the rate of the relaxed iteration with the factor that
375+
# estimate implies (which the windows raise `ω` toward), and at the end of each
376+
# relaxed window, from that window's smallest imbalance at the observed rate.
354377
_powermean_sinkhorn!(α, β, R::GroupedSupport{T}, C::GroupedSupport{T}, p::T,
355378
maxiter::Integer, tol::T) where {T} =
356379
_powermean_sinkhorn!(α, β, R, C, p, maxiter, tol, typemax(Int))
@@ -360,8 +383,8 @@ function _powermean_sinkhorn!(α, β, R::GroupedSupport{T}, C::GroupedSupport{T}
360383
ωmax = T(POWERMEAN_OMEGA_MAX)
361384
ω = one(T)
362385
nplain = 0 # consecutive plain sweeps in the current estimate
363-
r1 = r2 = T(Inf) # imbalance one and two plain sweeps back
364-
rbest = T(Inf) # smallest imbalance of the completed relaxed windows
386+
r1 = r2 = r3 = T(Inf) # imbalance one, two and three plain sweeps back
387+
rprev = T(Inf) # smallest imbalance of the previous relaxed window
365388
rwin = T(Inf) # smallest imbalance of the current window
366389
nwin = 0 # sweeps in the current window
367390
nstall = 0 # consecutive windows without progress
@@ -376,39 +399,41 @@ function _powermean_sinkhorn!(α, β, R::GroupedSupport{T}, C::GroupedSupport{T}
376399
if ω == 1
377400
nplain += 1
378401
if nplain >= POWERMEAN_NESTIMATE
379-
ρ = sqrt(r / r2)
380-
if zero(T) < ρ < one(T)
381-
ω = min(ωmax, 2 / (1 + sqrt(1 - ρ)))
382-
rbest, rwin, nwin, nstall = r, T(Inf), 0, 0
383-
_powermean_nsweeps(r, tol, _sor_rate(ρ, ω)) > budget && break
402+
ρ, ρprev = sqrt(r / r2), sqrt(r1 / r3)
403+
if zero(T) < ρ < one(T) && zero(T) < ρprev < one(T) &&
404+
log(ρ) >= POWERMEAN_STEADY * log(ρprev)
405+
ωopt = min(ωmax, 2 / (1 + sqrt(1 - ρ)))
406+
ω = min(ωopt, T(POWERMEAN_OMEGA_START))
407+
rprev, rwin, nwin, nstall = r, T(Inf), 0, 0
408+
_powermean_nsweeps(r, tol, _sor_rate(ρ, ωopt)) > budget && break
384409
end
385410
end
386-
r2, r1 = r1, r
411+
r3, r2, r1 = r2, r1, r
387412
else
388413
nwin += 1
389414
rwin = min(rwin, r)
390415
wlen = _powermean_window(ω)
391416
nwin < wlen && continue
392-
λ = (rwin / rbest)^(one(T) / wlen)
417+
λ = (rwin / rprev)^(one(T) / wlen)
393418
if !(λ < 1)
394419
nstall += 1
395420
if nstall >= POWERMEAN_NSTALL
396421
nstall = 0
397422
ω = 1 + (ω - 1) / 2
398423
if ω - 1 < (ωmax - 1) / 16
399-
ω, nplain, r1, r2 = one(T), 0, T(Inf), T(Inf)
424+
ω, nplain, r1, r2, r3 = one(T), 0, T(Inf), T(Inf), T(Inf)
400425
end
401426
end
402427
else
403428
nstall = 0
404429
if λ > ω - 1
405430
μ2 = (λ + ω - 1)^2 / (ω^2 * λ)
406-
μ2 < 1 && (ω = max(ω, min(ωmax, 2 / (1 + sqrt(1 - μ2)))))
431+
μ2 < 1 && (ω = max(ω, min(ωmax, 2 - (2 - ω) / POWERMEAN_RAISE, 2 / (1 + sqrt(1 - μ2)))))
407432
end
408433
end
409-
rbest = min(rbest, rwin)
434+
rprev = rwin
410435
rwin, nwin = T(Inf), 0
411-
λ < 1 && _powermean_nsweeps(rbest, tol, λ) > budget && break
436+
λ < 1 && _powermean_nsweeps(rprev, tol, λ) > budget && break
412437
end
413438
end
414439
imb = max(_powermean_imbalance(α, β, R, p), _powermean_imbalance(β, α, C, p))

‎test/powermean.jl‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,13 @@ end
176176
MatrixCovers._log_support(MatrixCovers._col_support(B, Float64)),
177177
2.0, 10^5, 4096 * eps())[2]
178178
@test nsweeps < 10_000
179+
# With the factor at its cap, roundoff holds the imbalance of this problem
180+
# near 1e-12. A tighter tolerance is reached only once the windows without
181+
# progress lower the factor.
182+
a, b, st = @test_logs soft_cover!(PowerMean{2}(), ones(100), ones(100), B;
183+
tol=1e-14, maxiter=20_000, newton=false)
184+
@test st.converged
185+
@test pm_imbalance(2, a, b, B) < 1e-13
179186
# Wide dynamic range with sparse support.
180187
Ssp = sprand(rng, 200, 200, 0.03) + I
181188
Ssp.nzval .= exp.(4.6 .* randn(rng, nnz(Ssp)))
@@ -187,6 +194,36 @@ end
187194
@test pm_imbalance(2, a, a, T) < 1e-11
188195
end
189196

197+
@testset "overrelaxation: a plateau does not set the factor" begin
198+
# From a flat start the imbalance stays nearly constant over the first sweeps
199+
# and then contracts quickly: plain sweeps take 55 (`p = 1`) and 70 (`p = 2`).
200+
# A factor read from the first sweeps' rate is near its cap, where the
201+
# relaxed iteration takes thousands.
202+
A = [3.5 0.0; 0.0 1.1; 0.38 4.3e6]
203+
for (p, nplain) in ((1, 55), (2, 70))
204+
a, b, st = @test_logs soft_cover!(PowerMean{p}(), ones(3), ones(2), A; newton=false)
205+
@test st.converged && st.nsweeps <= nplain
206+
@test pm_imbalance(p, a, b, A) < 1e-11
207+
end
208+
# The first relaxed sweeps run below the factor the plain estimate implies.
209+
# The handoff to Newton is predicted at the implied factor's rate; at the
210+
# rate of the factor in use, this solve (73 plain sweeps) would hand over.
211+
A = [49000.0 0.00067 9.2; 0.072 46.0 0.83]
212+
_, _, st = @test_logs soft_cover!(PowerMean{1}(), ones(2), ones(3), A)
213+
@test st.converged && !st.newton && st.nsweeps <= 73
214+
end
215+
216+
@testset "overrelaxation: a window is judged against the window before it" begin
217+
# Plain sweeps take over 10^5 here. The imbalance level rises each time the
218+
# factor does, so windows judged against the smallest imbalance of all earlier
219+
# ones show no progress and the factor is halved instead of raised, which
220+
# takes the iteration past the default sweep limit.
221+
A = [44.0 0.013 0.0 0.15 36.0; 0.086 6.3 0.02 150.0 0.087; 0.0 0.0021 0.39 0.0 1.7]
222+
a, b, st = @test_logs soft_cover(PowerMean{2}(), A; newton=false)
223+
@test st.converged
224+
@test pm_imbalance(2, a, b, A) < 1e-11
225+
end
226+
190227
@testset "start independence and the bipartite gauge" begin
191228
A = [1.0 2.0 0.5; 0.25 3.0 1.0]
192229
ref = soft_cover(A)

0 commit comments

Comments
 (0)