Check termination befor the first iteration

[danielhorn]

At the moment: Termination is checked for the first time after iteration one.
What if the initial design itself fullfills the termination criterion? In this case, we make one iteration. But we don't have to. We should also check the termination before iteration one - and end with a warning(?) in this case.

(I know - if the initial design is enough to terminate, something has to be wrongly specified. That's exactly what I've done a minute ago. But we don't have to do this one iteration here, so we should not.)

[berndbischl]

I guess you are right.
Actually we should probably refactor the termination a bit anyhow.

Would you do a PR? And before talk a bit here how we do that?

[jakob-r]

Certainly true. It's basically just this lines

if (getOptStateTermination(opt.state) > 0L) break

in mboTemplate.OptProblem.
Please self-assign or assign me if you want to.

[jakobbossek]

At the moment we have some hard-coded stopping conditions in shouldTerminate.
It would be way better to have a kind of plugin mechanism for stopping conditions. I think I developed good solutions for that in cmaesr and/or ecr.

[jakobbossek]

What I mean is let the user write his own stopping conditions which may depend on the opt.path entries and in particualar the extras stuff in the opt.path.
What do you think?

[berndbischl]

@jakobbossek
Would you be willing to refactor this the way you described? With a generic solution and the 2 special cases: a) budget reached, b) y-value reached.

Also the generic solution should depend on the currently fitted model(s).

[jakobbossek]

I would pass the optState to the stopping conditions. The fitted model is in there I think, isn't it?

[berndbischl]

yup

[danielhorn]

I also do like Jakobs idea. But as Bernd mentioned, I think we should talk about it first. In the best case Jakob will explain his interface and we simply agree with him.
ANd we should be careful here - it feels like we could make some easy things ("stop after 100 f.evals") complicated for the user. Some "shortcuts" like discussed in #138 would be really handy here.

[berndbischl]

Agreed, lets write down the API here. And simple common things will always be settable in a simple manner.

[berndbischl]

We could do:

setMBOControlTermination(budget = 100, y = 0.2) # stop after 100 fevals or when y is <= 0.2

and

setMBOControlTermination(fun = function(OptState))

where the user can programm his own stopping function

for bugdet, y, etc
we basically already provide such an internal function.

That seems goof and general?

[jakobbossek]

Bascially there would be a kind of generator for each stopping condition, e.g., makeMBOBudgedTermination. The generator creates objects of type mboStoppingCondition
with attributes name, message and objective.type (or a similar one). The latter might be useful
in catching faulty control settings, since some stopping conditions may be limited to single-objective optimization only (e.g., y-value reached).

The user can pass an arbitrary number of stopping conditions, e.g.,

setMBOControlTermination(
  makeMBOBudgetTermination(budget = 100),
  makeMBOIterTermination(max.iter = 10)
)

We could also offer a shortcut helper of the following form for the most common stopping conditions:

setMBOControlTermination(budget = 100, y = 0.2, ...)

[danielhorn]

My first idea was to use a separate termination object (makeMBOTerminator) and to detach the termination stuff from the control-object.The termination-object itself is simply a function(opt.state) that returns TRUE or FALSE and a termination message.

The terminator has some simple parameters like budget, time.budget, ... and the general fun-argument, as you proposed.

The advantage would be the better differentiation between termination-parameters and control-parameters. The disadvantage would be an additional parameter for the main mbo.

[jakobbossek]

@danielhorn: This is in general exactly what I described.

[jakobbossek]

Ok, it is not exactly what I described, but similar. I discribed multiple terminator objects, i.e., one terminator object for each stopping condition and a convenient helper which generates the terminator objects internally for the user for the most important stopping conditions.

[jakobbossek]

I thought about the interface once again. The best way would be to allow custom termination objects. Hence, we should provide a function makeMBOTermination(fun, stop.message) which expects a function that takes an opt.state as its only argument and a mandatory stop message.

Moreover, there should be a setMBOControlTermination, which expects all the predefined termination stuff like iters and time.budget and an optional list of custom MBOTerminator objects.

fn = makeSphereFunction(2L)
...
makeMyTerminator = function(target.value, eps = 1e-4) {
  force(target.value)
  myTerminator = function(opt.state) {
     # do stuff on opt.path or opt.state
  }
  makeMBOTerminator(myTerminator, stop.message = "Found global opt")
}
...
ctrl = makeMBOControl(...)
ctrl = setMBOControlTermination(iters = 100, makeMyTerminator(0))
res = mbo(fn, ctrl)

There would be MBOTerminator objects for max iters and time budget as well, which are generated automatically by setMBOControlTermination. However, we could go even without setMBOControlTermination and just add an additional argument more.stop.conds to makeMBOControl.

What do you think?

[jakobbossek]

Ok. We talked about that. The current interface is:

fn = makeSphereFunction(2L)
myTerminator = function(opt.state) {
   # do stuff on opt.path or opt.state
  return(list(
    stop.message = "Found global optimum",
    cond = TRUE/FALSE
  ))
}
ctrl = makeMBOControl(...)
ctrl = setMBOControlTermination(iters = 100, myTerminator)
res = mbo(fn, ctrl)

[jakobbossek]

We decided, that custom stopping conditions should be given by a simple function with a list return value. What if the stopping condition depends on a user defined value, e.g., target.fun.value.
How to make this available in the custom function without using global variables? With my proposed approach we would implement:

myTerminator = function(target.fun.value) {
   assertNumber(target.fun.value, na.ok = FALSE)
   force(target.fun.value)
   makeMBOTerminator(
      fn = function(opt.state) {  getOptStateBestValue(opt.state) <= target.fun.value) },
      message = sprintf("Target fun value %f reached.", target.fun.value)
   )
}

[jakobbossek]

ping

[berndbischl]

I do need to write down the answer.....

I dont get why you dont see that your constructor / force still works?

makeMyTerminator = function(myconst) {
  force(myconst)
  testterm = function(optstate) {
    if (optstate < myconst)
      return(list(term = TRUE, msg = sprintf("reached it")))
    else
      return(list(term = FALSE, msg = NA_character_))
  }
}

f = makeMyTerminator(10)

f(20)
f(3)

[jakobbossek]

Ok. Thats clear. But without the constructor it is not possible?

[berndbischl]

just called you, real talking maybe easier?

[jakobbossek]

Done.