Repository navigation
Add opt-in least-squares knot-curve calibration (v7.3.0) - #361
alecloudenback wants to merge 30 commits into
Conversation
A knot curve can now be fitted to more quotes than knots, such as a curve of ten knots fitted to an index of a thousand bonds with repeated maturities: `fit(ZeroRateCurve(rates, knots, spline), quotes)` minimizes the (optionally weighted, `Fit.LeastSquares(weights = w)`) squared pricing residuals by Levenberg-Marquardt on the residual Jacobian. Quotes with fixed cashflows are priced through their cashflow matrix, once per distinct date. `Fit.LeastSquares()` is the default for knot curves; with knots at the quote maturities it is the exact fit. On 1,000 bonds and 10 cubic knots the refit takes 1.7 s first / 3.1 ms warm (was 11.0 s / 28 ms with Optim's Newton on the loss), with the same knot rates to 6e-12. Fits are differentiable by the implicit function theorem on the optimality condition, including the residual curvature term, with respect to prices, cashflow amounts and weights. Quotes that do not determine the knots (fewer quotes than knots, a knot moving no price, an ill-conditioned problem) throw an ArgumentError. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
Options (a roughness penalty, a robust loss) can then be added as fields without changing existing calls. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
… scale) into feat/least-squares-fit Conflicts in src/fit.jl, resolved with #357's `Fit.Exact` in place of the internal `__ExactFit`: a refit without a method is `Fit.Exact()` when its knots are the quote maturities and `Fit.LeastSquares()` otherwise, and `optimizer`/`solve_kwargs` need an explicit `Fit.Loss`, as in #357. Drop the copy of `__fixed_cashflows` that #357 moved to src/fit_implicit.jl. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
…ares stops registering Address the review of #361: - Fit.LeastSquares(; weights, maxiters = 200) always minimizes, also with knots at the quote maturities: it reaches the exact fit's curve when one reprices the quotes, and the closest curve when none does, where Fit.Exact() throws. It no longer validates the weights and then runs the exact solve, so all-zero weights were ignored there. - It needs at least as many quotes with a positive weight as knots, so all-zero weights throw. - Weights are checked for dual numbers by element: a Vector{Real} of dual weights gave a zero derivative. - Convergence. Levenberg-Marquardt cannot take the gradient cosine below about sqrt(eps), where the sum of squares stops registering a decrease, and the fit accepted only a cosine of 1e-8 there, so fits stalled on that floor depending on rounding (MonotoneConvex on all three CI platforms). On a stall it now takes Gauss-Newton steps while they contract and then, when those contract slowly, Newton steps on the gradient with a Hessian by central differences of the analytic gradient (no second-order duals to compile). A fit is accepted when its last step is at most 1e-10 in knot rates. The exact-repricing stop uses #357's gross-value scale. - Monotone convex and PCHIP curves switch formulas, and a least-squares minimum can lie on a switch, where no smooth method converges. A fit that stops there throws a FitConvergenceError that says so (`Yield.__near_kink` on its remaining step). In a sweep of 20 random 80-bond indices, every linear, quadratic and cubic fit converges (75 of 75), and 13 and 12 of 25 monotone convex and PCHIP fits do; every other one throws the kink error, on Julia 1.13.1 and 1.14.0-alpha1. - The Jacobian configuration is built once per fit, with the Jacobian's type asserted, and the vector arithmetic is out of line, as in #357's exact solve. The 1,000-bond refit keeps its knot rates bit for bit, its first call, and its warm time. - The exact fit's errors point to Fit.LeastSquares() as the closest-curve alternative. - CI: SparseArrays joins the stdlibs in the Downgrade job's skip list, and the Fit.LeastSquares docstring is attached to its struct again. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
|
Thanks for the review. d65eac3 merges #357's review changes, and 15e46fa addresses this one. 1. Zero weights ignored with knots at the maturities. 2. Weight sensitivities lost for abstractly typed vectors. Dual numbers are now found element by element. Your 3. Platform-sensitive convergence. There were two causes.
On a sweep of 20 random 80-bond indices, with counts the same on Julia 1.13.1 and 1.14.0-alpha1 (and now in
The previous head failed 22 of 100 fits on a similar sweep, so the kink limitation predates this round. master's route for these refits ( The CI failures beyond those: SparseArrays joins the stdlibs in the Downgrade job's On the 1,000-bond index the knot rates are bit-identical to the previous head, and first calls and warm times are unchanged. 🤖 Generated with Claude Code |
…feat/least-squares-fit Without a method, a knot-curve fit given an optimizer or solve_kwargs is a Fit.Loss() fit again, as in v7.1; without them, a refit is Fit.Exact() with knots at the quote maturities and Fit.LeastSquares() otherwise. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
…-fit Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
A second review found a fit accepted 1.1e-6 from its minimum, with price derivatives 179 times too large: one knot and two zero-coupon prices whose residual curvature cancels JᵀJ but for 1e-5. The fit accepted a Gauss-Newton step of 5.6e-11, which did not contract; the Newton step with the full Hessian was 9.9e-5. A Gauss-Newton step can understate the distance to the minimum without bound, and in several dimensions contraction ratios cannot see a nearly flat direction. - A stalled fit is accepted only where the Hessian JᵀJ + S is positive definite and its Newton step is at most 1e-10; the returned rates include that step. S, the residuals' curvature, comes from central differences of its gradient with the residuals held fixed (steps of cbrt(eps)/t at the last cashflow time), within 2.4e-10 of the exact Hessian, with no second-order dual numbers. The step is solved through J's pivoted QR, as a Cholesky of I + R⁻ᵀSR⁻¹. - Otherwise Newton steps, halved until they keep the sum of squares and decrease the gradient; a Hessian that is not positive definite throws. - Levenberg-Marquardt stops once a step's predicted decrease is within rounding of the sum of squares, rather than growing its damping through futile trials. On the 1,000-bond index the fit is 12% (cubic) and 30% (linear) faster warm, with knot rates within 6e-16. - A difference stencil that crosses an interpolation's formula switch throws the kink error (Yield.__crosses_kink). - The implicit-function derivatives factor the exact Hessian by Cholesky and require its Newton step at the fitted rates to be at most 1e-10. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
|
Thanks, confirmed and fixed in d6281df. Low-curvature acceptance. I reproduced it on 15e46fa:
A contraction test on Gauss–Newton steps can't fix this in general: in several dimensions, a nearly flat direction barely shows in the step. So a fit is now accepted only with the full Hessian.
Your case now gives:
It is now a test in To check the other fits, I differentiated every converged fit of the 20-index sweep (60 smooth, 11 monotone convex): all 71 pass the exact-Hessian checks. Cost. Levenberg–Marquardt now stops once a step's predicted decrease is within rounding of the sum of squares, instead of growing its damping through 7–10 futile trials. That saves more than the Hessian costs (about 0.28 ms). On the 1,000-bond index, interleaved against 15e46fa:
Knot rates stay within 6e-16. Kinks. The difference stencil must not straddle a formula switch, or its curvature would belong to no formula; when it does, the fit throws the kink error. The new The sweep counts are unchanged and the same on Julia 1.13.1 and 1.14.0-alpha1:
Keeping the recommendation for smooth interpolation prominent stays the plan; how to treat kinked interpolations is still decision 1 in the description. Keyword rule (from #357). Solver settings without a method now select 🤖 Generated with Claude Code |
The reviewer's one-knot case now also checks, alongside the fitted rate, that the fit is a minimum whose Newton step by the analytic Hessian is at rounding, and that its price derivatives match central differences of the fit. The difference step is 3e-12, because a price lower by 3e-10 (the 50y quote) removes this local minimum. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
…quares-fit Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
…sest curve The fit minimizes locally from FinanceModels' fixed starting rates, so its result need not be the best fit of all curves; "the closest curve" claimed more than the solver guarantees. A curve that reprices the quotes is a global minimum, which the fit returns when it reaches it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
…into feat/least-squares-fit Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
…ifferentiated From a review of the documentation: - The sensitivity guide no longer says only exact fits are differentiated: exact fits and bootstrap differentiate the repricing equations, least squares its optimality condition JᵀWr = 0 at a local minimum (with the residual-curvature term), and loss fits must approximately reprice. It states what least-squares differentiation supports. - The Fit.LeastSquares docstring separates purpose, initialization (FM's standard starting rates, a local minimum), acceptance and the comparison with Fit.Exact; maxiters limits Levenberg-Marquardt, and polishing may take more steps. - The Newton correction is a local estimate of the remaining error in the knot rates, not the distance to the minimum; JᵀJ + S is the Hessian of half the sum of squares; the gradient cosine is zero at a stationary point; a Hessian that is not positive definite fails the fit's check. - The weighting example names its duration (modified, at the bond's yield) and why it makes residuals yield errors; the kink note is conditional. - The solver's overview comment is cut to its rationale, with thresholds beside their constants; NEWS keeps only the user-facing rule. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nw2FiivhYrC16532c1mJQN
Add opt-in
Fit.LeastSquares(; weights = nothing, maxiters = 200)for knot curves fitted to overlapping quotes, repeated maturities or more quotes than knots.Defaults remain
Fit.Loss(). Automatic Exact/LeastSquares selection remains in the separate v8 PR #362; #359 retains the Optim migration.Implementation and supported scope
Cashflow,Bond.Fixedand compositions), covering every built-in quote family. Other contracts use Fit.Loss. Remove the primal-only repricing fallback and its paired adapter methods, cached scale state and unused accessors.The minimum is local from standard seed rates, with no roughness penalty. Monotone Convex, PCHIP and Akima can fail at shape switches, including square systems that Exact solves; the method's contract and NEWS now state this explicitly.
Validation and performance
Analytic derivatives for an independently solved inconsistent two-quote/one-knot problem cover prices and weights, including the low-curvature case that distinguishes the full Hessian from Gauss–Newton. All 195 targeted LS assertions pass on Julia 1.13 and 1.10, including the random-index sweep, weighted/zero-weight fits, kink checks, unsupported contracts and FX.
On the 1,000-bond/nine-cubic-knot probe, warm fits stay around 1.7–1.8 ms. Shared AD curvature increases first-call latency from about 1.7 s to 8 s in a fresh process with dependencies loaded. This is the cost of consistent curvature/certification; it does not slow the warm solve. SparseArrays is a direct stdlib dependency and is skipped by the Downgrade job's package downgrading.
Release target: v7.3.0, in stack #369: #364 → #357 → #361 → #350 → #363.
Combined-stack validation: 16,595 passed, one existing
@test_brokenon Julia 1.13.1, including Aqua and extensions. Fourteen existing default/Loss calibration cases remain bit-identical to v7.2. Strict executed documentation, Runic and diff checks pass. The follow-up passes 551 targeted assertions on Julia 1.10.12: 465 calibration, 64 quote-shift and 22 independent financial checks. Runic 1.13.0 in file mode passes all 391 tracked Julia files across the five branch heads; #357 also passes its standalone 466-assertion suite..