Basic transfer learning example

[sgbaird]

I took the basic transfer learning example and brought it into a Colab notebook: https://colab.research.google.com/drive/1YOVW7hxdBlRrmnrYirubU5Yj2Z7GEqup?usp=sharing. It seemed to run OK (thanks for providing a SMOKE_TEST by the way).

While we could optimize the function using NumericalContinuousParameters, we use discrete parameters here because it lets us interpret the percentages shown in the final plot directly as the proportion of candidates for which there were target values revealed by the training function.

I think I get the desire to show proportion of candidates. For example, the candidate pool has 125 points, we start off with 20% = 25 candidates evaluated on the training function, then we see how the optimization progresses as we evaluate any of the 125 candidates on the test function. So, you're able to report that 20% of the candidates were evaluated on a related task. Some thoughts:

1. POINTS_PER_DIM=5 seems a bit low in the example. DIMENSION = 3 and POINTS_PER_DIM = 5, so 125 points in the candidate pool. It would probably be best to mention this explicitly in the text, and perhaps even in the figure itself.

2. I think a quasi-random method, or at least a grid search with some jitter added to the training datapoints for each of the repeat campaigns would make sense. See 1.0-traditional-doe-vs-bayesian.ipynb and search for "jitter". In this case, jitter is added to function parameter space itself (e.g., add 0.24 to all $x_1$ values and subtract 0.68 from all $x_2$ values for campaign 1).

(aside: you can see the plot outputs directly in the HTML version - excuse the overly verbose outputs).

3. As the number of candidates tends towards infinity, it becomes continuous search space, and the meaning of the percentage loses some of its meaning (e.g., 1% of 1000 is 10 datapoints, but 1% of 1e6 points is 10k datapoints). Is the main appeal of taking this approach for an analytical function to set up the framework to align with benchmarks with experimental datasets that are discrete?

[sgbaird]

Some related literature based on Meta AE workshop 2024:

(1) Fan, Z.; Han, X.; Wang, Z. HyperBO+: Pre-Training a Universal Hierarchical Gaussian Process Prior for Bayesian Optimization.
(2) Fan, Z.; Han, X.; Wang, Z. Transfer Learning for Bayesian Optimization on Heterogeneous Search Spaces. arXiv February 13, 2024. https://doi.org/10.48550/arXiv.2309.16597.
(3) Astudillo, R.; Frazier, P. I. Bayesian Optimization of Function Networks. arXiv December 31, 2021. https://doi.org/10.48550/arXiv.2112.15311.
(4) Astudillo, R.; Frazier, P. Bayesian Optimization of Composite Functions. In Proceedings of the 36th International Conference on Machine Learning; PMLR, 2019; pp 354–363.

Additionally:

• https://github.com/pytorch/botorch/blob/main/tutorials/composite_bo_with_hogp.ipynb
• https://botorch.org/tutorials/composite_mtbo
• https://botorch.org/tutorials/meta_learning_with_rgpe

[AVHopp]

Let me share some of my thoughts on our examples (@Scienfitz @AdrianSosic please feel free to add to this, in particular if you interpret things differently).

The examples that we have on the webpage are meant for users to demonstrate how to use BayBE as easily as possible. That is, the examples are intended to be very minimal and should not contain a lot of explanations, involved computations or similar. For more explanations, we have the user guides, and we do not aim to have extensive examples showing everything you can do on our webpage.

In particular, our examples are not meant to necessarily show the best possible performance that you can get with BayBE. They should show "Hey, this is something that you can do with BayBE, and look how easy it is!", and should provide users with a first "template" to implement their own use cases. The numbers/dimensions in the examples were mainly chosen in a way to have a good compromise between "reasonable computation time" and "results look sufficiently promising".

I think the points that you make here (which are good!) are important, but they are not what we aim to do with our examples. But let's hear @AdrianSosic and @Scienfitz thoughts on this :)

[AVHopp]

To add to this: I think what you are proposing here is going more in the direction of creating an additional section for "Benchmarks" next to the existing examples and user guides. From a content perspective, this would definitely be something very nice to have but I think maintaining this might be rather difficult

[sgbaird]

Thanks for clarifying! This makes sense. As @AVHopp mentioned, this was coming primarily from a benchmarking perspective. I will keep this in mind.

[AVHopp]

@sgbaird can we then close this? We can directly open a new discussion on the topic of "Benchmarking BayBE" or something similar and basically just copying your first comment, but I guess this would make the discussion clearer

[sgbaird]

Added #300