Behavior of BFGS

[NightWinkle]

Hi,

I'm not sure if this is the desired behavior, but BFGS directly stops if f(y0) == 0. As an example :

from optimistix import minimise, BFGS, rms_norm
import equinox as eqx
import jax.numpy as jnp


@eqx.filter_jit
def f(y, args):
    return jnp.sum(jnp.square(y))


N = 2


def f2(y, args):
    return f(y, args) - N


y0 = jnp.ones((N,))

res = minimise(
    f,
    BFGS(rtol=1e-13, atol=1e-13, norm=rms_norm),
    y0=y0,
    max_steps=1024,
)

print(res.stats["num_steps"])
assert jnp.allclose(res.value, jnp.zeros((N,)))

res = minimise(
    f2,
    BFGS(rtol=1e-13, atol=1e-13, norm=rms_norm),
    y0=y0,
    max_steps=1024,
)

print(res.stats["num_steps"])
assert jnp.allclose(res.value, jnp.zeros((N,)))

This is because then the termination condition is already validated at first step.
It's not really a big problem since you just need to offset the cost by a constant but it took me a while to figure out where the problem was coming from.
Maybe the initial value for f_info should be set to infinity, or maybe this should be mentioned in the doc ?

[NightWinkle]

Yes, but notice I initialize at $y = 1$. The mínima is unique and at $0$. Le jeu. 29 févr. 2024, 20:37, Jason Rader ***@***.***> a écrit :

…

Hmm, does this persist for functions whose derivative is not $0$ at $0$? $y^Ty - N$ has derivative $0$ at $y = 0$. It's unsurprising if BFGS does not move when initialised at a critical point. — Reply to this email directly, view it on GitHub <#47 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2T2XDAUO6QVOPLTUQHGLTYV6BOFAVCNFSM6AAAAABEAAKIICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZRHAZTAMBZHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[patrick-kidger]

Ah, I think I agree. In fact I think we might be making the same mistake in other solvers as well, e.g.

optimistix/optimistix/_solver/gradient_methods.py

Line 142 in 57be72f

f_info = FunctionInfo.EvalGrad(jnp.zeros(f_struct.shape, f_struct.dtype), y)

I'd be happy to take a pull request (+ some tests) adjusting this!