Skip to content

Add opt-in least-squares knot-curve calibration (v7.3.0) - #361

Open
alecloudenback wants to merge 30 commits into
perf/exact-spline-fitfrom
feat/least-squares-fit
Open

alecloudenback wants to merge 30 commits into
perf/exact-spline-fitfrom
feat/least-squares-fit

Conversation

@alecloudenback

@alecloudenback alecloudenback commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Add opt-in Fit.LeastSquares(; weights = nothing, maxiters = 200) for knot curves fitted to overlapping quotes, repeated maturities or more quotes than knots.

curve = fit(ZeroRateCurve(rates, knots, Spline.Cubic()), bond_quotes, Fit.LeastSquares(weights = w))

Defaults remain Fit.Loss(). Automatic Exact/LeastSquares selection remains in the separate v8 PR #362; #359 retains the Optim migration.

Implementation and supported scope

  • Require fixed cashflows (Cashflow, Bond.Fixed and compositions), covering every built-in quote family. Other contracts use Fit.Loss. Remove the primal-only repricing fallback and its paired adapter methods, cached scale state and unused accessors.
  • Price through a sparse cashflow matrix. Gross-size aggregation takes absolute amounts before combining payments at the same date, matching the shared projected-cashflow definition.
  • Separate LM iteration, contracting Gauss–Newton polishing and full Newton certification. Exact and LS retain separate solvers but share the signed trial-step helper.
  • Use one AD residual-curvature calculation and Newton-step acceptance criterion for polishing and implicit sensitivities. Newton corrections must resolve the same smooth interpolation branch as their Hessian; differentiation also retains additional conditioning and kink checks. This includes residual curvature rather than a Gauss–Newton sensitivity approximation.
  • Remove redundant time-grid and duplicate-tenor checks; the canonical grid validates maturities. Update examples/error messages that incorrectly called Exact or LeastSquares the default. The loss-refit documentation now states ArgumentError for a non-square differentiated fit.

The minimum is local from standard seed rates, with no roughness penalty. Monotone Convex, PCHIP and Akima can fail at shape switches, including square systems that Exact solves; the method's contract and NEWS now state this explicitly.

Validation and performance

Analytic derivatives for an independently solved inconsistent two-quote/one-knot problem cover prices and weights, including the low-curvature case that distinguishes the full Hessian from Gauss–Newton. All 195 targeted LS assertions pass on Julia 1.13 and 1.10, including the random-index sweep, weighted/zero-weight fits, kink checks, unsupported contracts and FX.

On the 1,000-bond/nine-cubic-knot probe, warm fits stay around 1.7–1.8 ms. Shared AD curvature increases first-call latency from about 1.7 s to 8 s in a fresh process with dependencies loaded. This is the cost of consistent curvature/certification; it does not slow the warm solve. SparseArrays is a direct stdlib dependency and is skipped by the Downgrade job's package downgrading.

Release target: v7.3.0, in stack #369: #364 → #357 → #361 → #350 → #363.

Combined-stack validation: 16,595 passed, one existing @test_broken on Julia 1.13.1, including Aqua and extensions. Fourteen existing default/Loss calibration cases remain bit-identical to v7.2. Strict executed documentation, Runic and diff checks pass. The follow-up passes 551 targeted assertions on Julia 1.10.12: 465 calibration, 64 quote-shift and 22 independent financial checks. Runic 1.13.0 in file mode passes all 391 tracked Julia files across the five branch heads; #357 also passes its standalone 466-assertion suite..

alecloudenback and others added 2 commits October 24, 2026 22:39
A knot curve can now be fitted to more quotes than knots, such as a curve
of ten knots fitted to an index of a thousand bonds with repeated
maturities: `fit(ZeroRateCurve(rates, knots, spline), quotes)` minimizes
the (optionally weighted, `Fit.LeastSquares(weights = w)`) squared pricing
residuals by Levenberg-Marquardt on the residual Jacobian. Quotes with
fixed cashflows are priced through their cashflow matrix, once per
distinct date. `Fit.LeastSquares()` is the default for knot curves; with
knots at the quote maturities it is the exact fit.

On 1,000 bonds and 10 cubic knots the refit takes 1.7 s first / 3.1 ms
warm (was 11.0 s / 28 ms with Optim's Newton on the loss), with the same
knot rates to 6e-12. Fits are differentiable by the implicit function
theorem on the optimality condition, including the residual curvature
term, with respect to prices, cashflow amounts and weights.

Quotes that do not determine the knots (fewer quotes than knots, a knot
moving no price, an ill-conditioned problem) throw an ArgumentError.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
Options (a roughness penalty, a robust loss) can then be added as fields
without changing existing calls.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
alecloudenback and others added 2 commits October 25, 2026 10:46
… scale) into feat/least-squares-fit

Conflicts in src/fit.jl, resolved with #357's `Fit.Exact` in place of the internal `__ExactFit`:
a refit without a method is `Fit.Exact()` when its knots are the quote maturities and
`Fit.LeastSquares()` otherwise, and `optimizer`/`solve_kwargs` need an explicit `Fit.Loss`, as in
#357. Drop the copy of `__fixed_cashflows` that #357 moved to src/fit_implicit.jl.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
…ares stops registering

Address the review of #361:

- Fit.LeastSquares(; weights, maxiters = 200) always minimizes, also with knots at the quote
  maturities: it reaches the exact fit's curve when one reprices the quotes, and the closest curve
  when none does, where Fit.Exact() throws. It no longer validates the weights and then runs the
  exact solve, so all-zero weights were ignored there.
- It needs at least as many quotes with a positive weight as knots, so all-zero weights throw.
- Weights are checked for dual numbers by element: a Vector{Real} of dual weights gave a zero
  derivative.
- Convergence. Levenberg-Marquardt cannot take the gradient cosine below about sqrt(eps), where the
  sum of squares stops registering a decrease, and the fit accepted only a cosine of 1e-8 there, so
  fits stalled on that floor depending on rounding (MonotoneConvex on all three CI platforms). On a
  stall it now takes Gauss-Newton steps while they contract and then, when those contract slowly,
  Newton steps on the gradient with a Hessian by central differences of the analytic gradient
  (no second-order duals to compile). A fit is accepted when its last step is at most 1e-10 in
  knot rates. The exact-repricing stop uses #357's gross-value scale.
- Monotone convex and PCHIP curves switch formulas, and a least-squares minimum can lie on a
  switch, where no smooth method converges. A fit that stops there throws a FitConvergenceError
  that says so (`Yield.__near_kink` on its remaining step). In a sweep of 20 random 80-bond indices,
  every linear, quadratic and cubic fit converges (75 of 75), and 13 and 12 of 25 monotone convex
  and PCHIP fits do; every other one throws the kink error, on Julia 1.13.1 and 1.14.0-alpha1.
- The Jacobian configuration is built once per fit, with the Jacobian's type asserted, and the
  vector arithmetic is out of line, as in #357's exact solve. The 1,000-bond refit keeps its knot
  rates bit for bit, its first call, and its warm time.
- The exact fit's errors point to Fit.LeastSquares() as the closest-curve alternative.
- CI: SparseArrays joins the stdlibs in the Downgrade job's skip list, and the Fit.LeastSquares
  docstring is attached to its struct again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
@alecloudenback

Copy link
Copy Markdown
Member Author

Thanks for the review. d65eac3 merges #357's review changes, and 15e46fa addresses this one.

1. Zero weights ignored with knots at the maturities. Fit.LeastSquares() no longer runs the exact solve with aligned knots: it always minimizes. It now needs at least as many quotes with a positive weight as knots, so your all-zero cases throw an ArgumentError. Before, the refit and the spline fit both returned a curve. With aligned knots, least squares reaches the exact fit's curve (to 1e-12 in the tests) when one exists. When none does, it returns the closest curve, where Fit.Exact() throws: on a quadratic spline through alternating ZCB and par yields at 15%, it misprices one quote by 9% at a stationary point.

2. Weight sensitivities lost for abstractly typed vectors. Dual numbers are now found element by element. Your Vector{Real} case gives −6.0725e-6, the same as the typed vector; it gave 0.0 before.

3. Platform-sensitive convergence. There were two causes.

  • A smooth stall on the cosine floor. Levenberg–Marquardt accepts a step only if the sum of squares falls, and at cosine c the largest relative decrease is about c². So it cannot get below c ≈ √eps ≈ 1.5e-8, and the old acceptance threshold of 1e-8 sat on that floor; which side a fit landed on depended on rounding. The CI MonotoneConvex failures (cosines 1.0–3.5e-8) were this case. On a stall the fit now:
    • takes Gauss–Newton steps while they contract;
    • when those contract slowly (the residual curvature nearly cancels JᵀJ in some direction; one Linear index contracted at 0.79 per step), takes Newton steps on the gradient, with a Hessian by central differences of the analytic gradient. That needs only the Jacobian's code;
    • is accepted when its last step is at most 1e-10 in knot rates.
  • Kinks. MonotoneConvex and PCHIP switch formulas between regions of knot rates, so a least-squares minimum can lie on a switch. There the Jacobian jumps (by 30–45% under an infinitesimal step, in the cases I looked at), and no smooth method converges. Such a fit now throws a FitConvergenceError that says so, using Yield.__near_kink on its remaining step, and suggests a smooth interpolation, knots at the quote maturities, or Fit.Loss() with NelderMead().

On a sweep of 20 random 80-bond indices, with counts the same on Julia 1.13.1 and 1.14.0-alpha1 (and now in test/least_squares.jl):

  • every Linear, Quadratic and Cubic fit converges (75 of 75, including quotes priced one at a time);
  • 13 of 25 MonotoneConvex and 12 of 25 PCHIP fits converge, and every other one throws the kink error.

The previous head failed 22 of 100 fits on a similar sweep, so the kink limitation predates this round. master's route for these refits (Fit.Loss(), LBFGS) also fails on most of the kinked cases. How to treat kinked interpolations is decision 1 in the updated description.

The CI failures beyond those: SparseArrays joins the stdlibs in the Downgrade job's skip list, and the Fit.LeastSquares docstring is attached to its struct again.

On the 1,000-bond index the knot rates are bit-identical to the previous head, and first calls and warm times are unchanged.

🤖 Generated with Claude Code

alecloudenback and others added 3 commits October 25, 2026 15:26
…feat/least-squares-fit

Without a method, a knot-curve fit given an optimizer or solve_kwargs is a
Fit.Loss() fit again, as in v7.1; without them, a refit is Fit.Exact() with
knots at the quote maturities and Fit.LeastSquares() otherwise.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
…-fit

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
A second review found a fit accepted 1.1e-6 from its minimum, with price
derivatives 179 times too large: one knot and two zero-coupon prices whose
residual curvature cancels JᵀJ but for 1e-5. The fit accepted a Gauss-Newton
step of 5.6e-11, which did not contract; the Newton step with the full
Hessian was 9.9e-5. A Gauss-Newton step can understate the distance to the
minimum without bound, and in several dimensions contraction ratios cannot
see a nearly flat direction.

- A stalled fit is accepted only where the Hessian JᵀJ + S is positive
  definite and its Newton step is at most 1e-10; the returned rates include
  that step. S, the residuals' curvature, comes from central differences of
  its gradient with the residuals held fixed (steps of cbrt(eps)/t at the
  last cashflow time), within 2.4e-10 of the exact Hessian, with no
  second-order dual numbers. The step is solved through J's pivoted QR, as a
  Cholesky of I + R⁻ᵀSR⁻¹.
- Otherwise Newton steps, halved until they keep the sum of squares and
  decrease the gradient; a Hessian that is not positive definite throws.
- Levenberg-Marquardt stops once a step's predicted decrease is within
  rounding of the sum of squares, rather than growing its damping through
  futile trials. On the 1,000-bond index the fit is 12% (cubic) and 30%
  (linear) faster warm, with knot rates within 6e-16.
- A difference stencil that crosses an interpolation's formula switch throws
  the kink error (Yield.__crosses_kink).
- The implicit-function derivatives factor the exact Hessian by Cholesky and
  require its Newton step at the fitted rates to be at most 1e-10.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
@alecloudenback

Copy link
Copy Markdown
Member Author

Thanks, confirmed and fixed in d6281df.

Low-curvature acceptance. I reproduced it on 15e46fa:

  • the fit returned 0.049999999944 after accepting a Gauss–Newton step of 5.6e-11 that did not contract;
  • the Newton step with the full Hessian was 9.9e-5;
  • the first price's risk came out at −94,261, against −528 at the minimum.

A contraction test on Gauss–Newton steps can't fix this in general: in several dimensions, a nearly flat direction barely shows in the step. So a fit is now accepted only with the full Hessian.

  • The Hessian. H = JᵀJ + S, where S is the residuals' curvature. S comes from central differences of its gradient, with the residuals held fixed, in steps of cbrt(eps)/t at the last cashflow time t.
    • On the 1,000-bond index it is within 2.4e-10 of the exact nested-AD Hessian. The previous difference Hessian, of the full gradient, was 5e-8 off.
    • It reuses the Jacobian's code, so nothing new compiles. Nested AD here would add 7.4 s to the first call.
  • Acceptance. The fit is accepted iff H is positive definite and the Newton step H⁻¹g is at most 1e-10, and the returned rates include that step. It is solved through J's pivoted QR, as a Cholesky of I + R⁻ᵀSR⁻¹, so it works where JᵀJ itself is too ill-conditioned to factor.
  • Otherwise: Newton steps, each halved until it keeps the sum of squares and decreases ‖g‖. If the Hessian is not positive definite, the fit throws, since the point is not a minimum.
  • Derivatives. The implicit-function derivatives factor their exact Hessian by Cholesky, so a saddle throws. They also require the exact Newton step at the fitted rates to be at most 1e-10 before returning derivatives.

Your case now gives:

  • Rate: 0.049998896500438, 1.6e-13 from the bisected minimum, which is at the gradient's rounding.
  • Risk: [−527.73803, −2277.1435], against [−527.73811, −2277.1438] at the minimum (ratio 0.99999985). Central differences of the fit give −527.73.

It is now a test in test/least_squares.jl.

To check the other fits, I differentiated every converged fit of the 20-index sweep (60 smooth, 11 monotone convex): all 71 pass the exact-Hessian checks.

Cost. Levenberg–Marquardt now stops once a step's predicted decrease is within rounding of the sum of squares, instead of growing its damping through 7–10 futile trials. That saves more than the Hessian costs (about 0.28 ms). On the 1,000-bond index, interleaved against 15e46fa:

15e46fa d6281df
cubic, warm 3.12–3.19 ms 2.70–2.87 ms
linear, warm 3.35–3.44 ms 2.38–2.41 ms
allocations 11.8 MiB 10.7 MiB
first call 1.71–1.73 s 1.73–1.75 s

Knot rates stay within 6e-16.

Kinks. The difference stencil must not straddle a formula switch, or its curvature would belong to no formula; when it does, the fit throws the kink error. The new Yield.__crosses_kink, beside __near_kink, checks that. __near_kink is too conservative here: it flagged a PCHIP minimum whose 10y and 15y knots differ by 4.9e-7, where the stencil crosses nothing.

The sweep counts are unchanged and the same on Julia 1.13.1 and 1.14.0-alpha1:

  • 75 of 75 linear, quadratic and cubic fits converge;
  • 13 of 25 monotone convex;
  • 12 of 25 PCHIP, with knot rates within 4e-14 of the previous head.

Keeping the recommendation for smooth interpolation prominent stays the plan; how to treat kinked interpolations is still decision 1 in the description.

Keyword rule (from #357). Solver settings without a method now select Fit.Loss(), as in v7.1, so nothing breaks in 7.2.0; details on #357. Here, a refit without settings is Fit.Exact() with knots at the quote maturities and Fit.LeastSquares() otherwise.

🤖 Generated with Claude Code

alecloudenback and others added 9 commits October 25, 2026 16:28
The reviewer's one-knot case now also checks, alongside the fitted rate,
that the fit is a minimum whose Newton step by the analytic Hessian is at
rounding, and that its price derivatives match central differences of the
fit. The difference step is 3e-12, because a price lower by 3e-10 (the 50y
quote) removes this local minimum.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
…quares-fit

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
…sest curve

The fit minimizes locally from FinanceModels' fixed starting rates, so its
result need not be the best fit of all curves; "the closest curve" claimed
more than the solver guarantees. A curve that reprices the quotes is a global
minimum, which the fit returns when it reaches it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
…into feat/least-squares-fit

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
…ifferentiated

From a review of the documentation:
- The sensitivity guide no longer says only exact fits are differentiated:
  exact fits and bootstrap differentiate the repricing equations, least
  squares its optimality condition JᵀWr = 0 at a local minimum (with the
  residual-curvature term), and loss fits must approximately reprice. It
  states what least-squares differentiation supports.
- The Fit.LeastSquares docstring separates purpose, initialization (FM's
  standard starting rates, a local minimum), acceptance and the comparison
  with Fit.Exact; maxiters limits Levenberg-Marquardt, and polishing may take
  more steps.
- The Newton correction is a local estimate of the remaining error in the
  knot rates, not the distance to the minimum; JᵀJ + S is the Hessian of
  half the sum of squares; the gradient cosine is zero at a stationary point;
  a Hessian that is not positive definite fails the fit's check.
- The weighting example names its duration (modified, at the bond's yield)
  and why it makes residuals yield errors; the kink note is conditional.
- The solver's overview comment is cut to its rationale, with thresholds
  beside their constants; NEWS keeps only the user-facing rule.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
@alecloudenback alecloudenback changed the title Fit knot curves to many overlapping quotes by least squares (Fit.LeastSquares) Add opt-in least-squares knot-curve calibration (v7.2.0) Oct 26, 2026
@alecloudenback alecloudenback changed the title Add opt-in least-squares knot-curve calibration (v7.2.0) Add opt-in least-squares knot-curve calibration (v7.3.0) Nov 9, 2026
@alecloudenback
alecloudenback added this pull request to stack #369 November 9, 2026 04:12

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant