Accessing model predictions/posterior or surrogate model with DesirabilityObjective

[sskaye]

Is there a way to access the model predictions/posterior when using a DesirabilityObjective (or any objective with multiple targets)? I need this in order to examine how well the model is fitting the data.

For a SingleTargetObjective, I use the campaign.get_surrogate() method, fit the surrogate, and then use the posterior method of the surrogate to compute the predictions.

Unfortunately, I can't do the same with a DesirabilityObjective as surrogate model access via campaign.get_surrogate() is currently only supported for a single, untransformed target (Source).

Is there another way to get the model predictions for a set of provided measurements? If not, I'm also willing to try implementing accessing the surrogate model for DesirabilityObjective. Does anyone know what the challenge is there? or could point me in the right direction?

Thanks!

[AVHopp]

Thank for your interest in BayBE :)
BayBE offers the possibility to add custom hooks to basically any function that you want to modify. In our examples, this functionality is e.g. used to calculate the probability of improvement scores during a campaign, see here for the corresponding example and here for some more details on the general functionality. Using such hooks, you should hopefully be able to get access to the desired information. If not (or if we can assist further) feel free to ask :)
In addition, we would be more than happy to support you with a direct implementation of this feature. Regarding the potential challenges, I'll hand it over to @AdrianSosic who implemented the general functionality (he is currently out of office, so might take a while until he can comment here).

[AdrianSosic]

Hi @sskaye, you are not the first one asking for the feature, indeed. See here, and my answer to it:
#355 (comment)

Getting the model behind the desirability objective should be a rather minor extension (which is also already on our TODO list) but note that the current implementation of that objective is inherently scalar, i.e. there is no way at the moment to get predictive for the individual targets, only the desirability itself is accessible.

This will fundamentally change within the next few weeks, where we'll do another core refactor of the backend such that it finally supports multiple targets (i.e. in a Pareto sense), which will then also allow us to expose their individual surrogate models. I'm afraid that this is a bit of a bigger architecture change, though, so it's not done in a single day. My plan is to start working on in next week or the week after, depending on other workload. Nonetheless, proposals are always welcome 😉 But the final design choices will require quite a bit of alignment among the developers.

Does this answer your question?

[sskaye]

Thanks for the quick response.

@AVHopp, the custom hooks plus exposure of the model behind the desirability objective looks like it'll get me most of what I need. I'll try it out tomorrow.

@AdrianSosic, yes that all answers my question. Looking forward to the addition of multi-target support. I'm getting pretty good results using the DesirabilityObjective for multi-target optimizations, but some of the algorithms I'm comparing to directly work with multiple targets, so it will be nice to get a cleaner comparison.

[Scienfitz]

@sskaye

I've just opened PR #386, which removes some of the restrictions. In particular you now can get the surrogate model for

• a single target which had a transformation applied
• a desirability objective

After the PR is merged your problem should be solved as long as youre happy with getting the posterior for the desirability and not the raw targets. Let me know if thats the case or not. For other reasons we plan to publish another version 0.11.1 asap, including this change.

@AdrianSosic
I think the restriction was somethat prohibitite. As long as users are aware that the y of the model they extract is the transformed target / objective, why would you not allow them to have it? A note explaining that has been added.

I think he way forward is the mentioned implementation of #373 . Once thats here, I think get_surrogate should get an option indicating whether the model(s) for untransformed targets / objective are desired or not. Until then I see no reason to prohibit the model extraction for scalarized models.

[sskaye]

Thanks! Yes, getting access to the posterior for the desirability will work, as that lets me score how well the model is fitting the data.

[sskaye]

Thanks, implemented scoring the model fits using the new access to the posterior for DesirabilityObjectives. All tests show as working and it's already unearthed some useful info. Thanks for your quick help!

[Scienfitz]

great to hear, many thanks for confirming

[Scienfitz]

closed via #386