Accessing predictions with multiple objectives

[PatWalters]

I've been figuring out how to optimize for multiple objectives using a DesirabilityObjective. If I do something like this.

objective = DesirabilityObjective(
    targets=targets,
    weights=[50, 50],
    scalarizer="MEAN",
)
searchspace = SearchSpace.from_product(parameters)
campaign = Campaign(searchspace, objective)
# df contains the experiments we've performed so far
campaign.add_measurements(df)
rec = campaign.recommend(3)

Then I call

p = campaign.posterior(rec)
p.mean

I get

tensor([[0.8652],
        [0.8790],
        [0.9023]])

I'm assuming p.mean is giving me the desirability scores. Is there a way to get the predictions for the individual components that comprise the targets in the DesirabilityObject?

Thanks!

[AdrianSosic]

Hi @PatWalters, at the moment, the answer is "yes and no" 🙃 but will be "yes" only soon. Let me explain:

Current limitation

Getting predictions for the individual target directly from an existing DesirabilityObjective is not possible because the surrogate model is currently being fitted directly on the desirability values, which means we'd have to invert the mapping from target values to desirability value – which is impossible since it's non-injective.

Upcoming solutions

We plan to ship two features soon, both of which solve the problem in their own way:

• We'll add a true MultiTargetObjective for Pareto optimization, which inherently comes with separate surrogate models for each target, meaning that extracting their predictions will become trivial.
• There will also be a second version of the desirability objective, where the modeling is done at the target level and the desirability values are computed from taht.

Current workaround

That said, there is also a workaround that you can do already today. That is, simply construct the corresponding surrogate models manually. Here is an example of how you could do it:

import pandas as pd

from baybe.campaign import Campaign
from baybe.objectives.desirability import DesirabilityObjective
from baybe.parameters.numerical import NumericalDiscreteParameter
from baybe.searchspace.core import SearchSpace
from baybe.surrogates.gaussian_process.core import GaussianProcessSurrogate
from baybe.targets.numerical import NumericalTarget

searchspace = SearchSpace.from_parameter(NumericalDiscreteParameter("x", range(10)))
targets = [
    NumericalTarget("t0", "MAX", bounds=(0, 1)),
    NumericalTarget("t1", "MAX", bounds=(0, 1)),
]
objective = DesirabilityObjective(targets)
measurements = pd.DataFrame({"x": [2, 5, 8], "t0": [0.2, 1, 0.3], "t1": [0.9, 0.2, 0.4]})

campaign = Campaign(searchspace, objective)
campaign.add_measurements(measurements)
recommendations = campaign.recommend(3)

surrogate = GaussianProcessSurrogate()
surrogate.fit(searchspace, targets[0].to_objective(), measurements)
p0= surrogate.posterior(recommendations)

surrogate = GaussianProcessSurrogate()
surrogate.fit(searchspace, targets[1].to_objective(), measurements)
p1 = surrogate.posterior(recommendations)

print(p0.mean)
print(p1.mean)

Note, however, that these models were not actually used in the recommendation process. So they simply show you what the same GP architecture would give you if applied to the targets individually.

[PatWalters]

@AdrianSosic There's something here that I don't understand. If I take your code above and simply change t0 to range between 0 and 100.

import pandas as pd

from baybe.campaign import Campaign
from baybe.objectives.desirability import DesirabilityObjective
from baybe.parameters.numerical import NumericalDiscreteParameter
from baybe.searchspace.core import SearchSpace
from baybe.surrogates.gaussian_process.core import GaussianProcessSurrogate
from baybe.targets.numerical import NumericalTarget

searchspace = SearchSpace.from_parameter(NumericalDiscreteParameter("x", range(10)))
targets = [
    NumericalTarget("t0", "MAX", bounds=(0, 100)),
    NumericalTarget("t1", "MAX", bounds=(0, 1)),
]
objective = DesirabilityObjective(targets)
measurements = pd.DataFrame({"x": [2, 5, 8], "t0": [20, 100, 30], "t1": [0.9, 0.2, 0.4]})

campaign = Campaign(searchspace, objective)
campaign.add_measurements(measurements)
recommendations = campaign.recommend(3)

surrogate = GaussianProcessSurrogate()
surrogate.fit(searchspace, targets[0].to_objective(), measurements)
p0= surrogate.posterior(recommendations)

surrogate = GaussianProcessSurrogate()
surrogate.fit(searchspace, targets[1].to_objective(), measurements)
p1 = surrogate.posterior(recommendations)

print(p0.mean)
print(p1.mean)

I get this result.

tensor([[0.8135],
        [0.1563],
        [0.9875]], grad_fn=<UnsqueezeBackward0>)
tensor([[0.3939],
        [1.0654],
        [0.2165]], grad_fn=<UnsqueezeBackward0>)

I don't understand why the posterior for t0 still seems to range between 0 and 1. Btw, thanks for continuing to answer my naive questions.

[AdrianSosic]

Hi @PatWalters, apologies for the late response, I'm on part time these days.

I understand that this behavior can appear very surprising and we clearly need to improve our API to avoid confusion on the user side. Also, there isn't yet a fully fledged user guide available for the surrogates and we should work on that, too. But the explanation of the effect is mentioned here, somewhat hidden in the little info box.

In essence: it is due to the same limitation we've already talked about previously, namely that the surrogate is currently constructed on the transformed target values (see discussion on desirability above). Here, the mechanism is that your surrogate describes the scaled target values, where the scaling is with respect to the target bounds you've defined in your code. Remove the bounds and there will be no transformation involved, so that you will get the surrogate predictions on the original scale (for that to work in combination with your desirability objective, you'd need to create unscaled copies of your targets just for the sake of prediction).

As you have probably noticed by now, the entire surrogate API is not yet fully fleshed out from the user's perspective. In fact, we had an internal discussion (ping @Scienfitz) whether or not to provide access to the surrogate in the presence of target transformations already at the current state, exactly because of this potential confusion. There will definitely be a refactoring soon that makes the hidden details explicit to the user. For example, I can images something like a transformed: bool argument with default value False that let's the user control the type of surrogate access.

Does all of that make any sense to you?

[PatWalters]

@AdrianSosic, thanks; that makes sense.

[AdrianSosic]

Hey @PatWalters, seems I'm a bit too late since your message has been already edited 🙃 Still, wanted to give you the answer:
The backtransformation should be

x_unscaled = x_posterior * (x_upper_bound - x_lower_bound) + x_lower_bound

Tthat is, the last term was missing in your equation. However, I wouldn't even bother doing manual backtransformations but rather simply create a copy of the corresponding target without the bounds. That way, the correct calculation is done internally.

Hope that we can ship an improved interface soon ✌🏼

[PatWalters]

Thanks, @AdrianSosic! I edited my response because I figured out that it was easier to create a new target without bounds. Here's the function I wrote. I'm eager to try the new version!

```python
def fit_surrogate_model(searchspace_in, tgt_name, tgt_mode, df_in, rec_in):
    """
    Fits a Gaussian Process surrogate model to the given data and generates predictions for the recommendations.

    Parameters:
    searchspace_in (SearchSpace): The search space containing the parameters.
    tgt_name (str): The name of the target variable.
    tgt_mode (str): The mode of the target variable, either "MAX" or "MIN".
    df_in (pd.DataFrame): The input dataframe containing the experimental data.
    rec_in (pd.DataFrame): The dataframe containing the recommendations for which predictions are to be made.

    Returns:
    pd.DataFrame: A dataframe containing the mean and standard deviation of the predictions for the target variable.
    """
    surrogate = GaussianProcessSurrogate()
    tgt = NumericalTarget(name=tgt_name, mode=tgt_mode)
    surrogate.fit(searchspace_in, tgt.to_objective(), df_in)
    post = surrogate.posterior(rec_in)
    return pd.DataFrame({tgt.name + "(mean)" : post.mean.detach().numpy().flatten(),
                         tgt.name + "(std)" : np.sqrt(post.variance.detach().numpy().flatten())})