Accessing predictions

[PatWalters]

Thanks for releasing an extremely useful package.

If I run a campaign and call campaign.recommend I can get the next set of n recommended experiments. Is it possible to access the mean and variance for the predicted values based on the parameters generated by campaign.recommend? For instance, If my target is yield and campaign.recommend proposes 5 experiments, can I get the predicted mean and variance on the yield for each of the 5 sets of conditions?

It looks like I can do something like this. Is this correct?

recommend_df = campaign.recommend(batch_size=5)
posterior = campaign.posterior(recommend_df)
pred_mean =  posterior.mean
pred_var = posterior.variance

[Scienfitz]

Hi @PatWalters!

That seems about right. Does this work for you?

I just tried this with the basic example in examples/Basics/campaign.py and notice some potential bug. The default recommender will do a random recommendation in the first iteration, posterior wont be available during that. Theoretically we intend that as soon as the first datapoint is added the posterior should be available (because a model-based recommender is used then), however I noticed that this didn't work as intended and even though data was added .posterior causes RuntimeError: The current recommender is of type 'RandomRecommender'. Surrogate models are only available for recommender subclasses of 'BayesianRecommender'.

A simple fix here is to call campaign.recommend(any_number) again. The results can be discarded. This call just makes sure that the actual model (not the random recommender) is fitted. Then the posterior is available.

Also, if you want the posterior over not just recommendations but all candidates you can do campaign.posterior(searchspace.discrete.exp_rep).

You can also do quite a bit more advanced "injection" of hooks for custom analysis with register_hooks . Here you can see how that can be used to print the value of any acquisition function (not just the one used for the optimization loop) or for auto-stopping a campaign. The downside here is that you'd need more in-depth understanding of baybe objects and their signatures.

PS: I am def a bit startstruck right now, big fan of th blog :)

[PatWalters]

Hi @Scienfitz,

Thank you for the quick answer! I'm primarily a drug discovery guy, and I've just started working on modeling for process chemistry. It's comforting that many of the techniques you use are similar to what we use for active learning. It looks like when BayBE optimizes for multiple objectives; it optimizes an average over the objectives. Do you have plans to add EHVI?

[Scienfitz]

Yes, we consider active learning a special case of Bayesian optimization. You can do actice learning with baybe.

EHVI and Pareto optimization in general is coming as part of our next big construction site that will provide multi-output modeling and integrates it with all of the existing machinery

[AdrianSosic]

Hi @PatWalters, thanks for the kind words. If you enjoy the package, we'd appreciate if you'd leave us a ⭐. They really help us develop this package further 👍🏼

Regarding the "bug" that @Scienfitz mentioned: this is actually intended behavior since the posterior method attempts to extract the posterior of the "current" recommender, i.e. the one of the active round of recommendations. As @Scienfitz explained correctly, this fails when called on the first batch of recommendations when using the default configuration of a campaign, since it uses a non-predictive recommender as long as there are no data points available. (Adding measurements increases the number of datapoints to > 0, but without requesting another batch, we are still in the "first round of recommendations", hence the error.)

That said, the feature is rather new and we could still improve the behavior if necessary. Plus, we definitely need to add documentation. May I ask what behavior you would expect / prefer, now that you know the underlying mechanics? Perhaps we can find a more user-friendly default. But as you'd probably agree, there is no "obvious" choice what to return when meta-recommenders (i.e. combinations of multiple recommenders) are involved

[MuriloBarranco]

[AdrianSosic]

Hi @MuriloBarranco, was this post by mistake?