General feedback - documentation, functionality, and software dev

[sgbaird]

I had a nice time reviewing this repository! Overall I think it's a really comprehensive, clean, and well-documented project. Thank you for open-sourcing it! Find below some questions and suggestions:

Docs improvements

General

• It has a nice, clean look, both on the GitHub README and on the documentation site. This is really important I think.

README

• A Colab notebook or similar would be really good I think. See e.g., https://colab.research.google.com/drive/1VEHXBLVkn5NZ7N-Oj6-dc_hkIfwFcUE-?usp=sharing. I needed to %pip install 'baybe[chem,simulation]' numpy==1.24.4 on Colab, otherwise it seems to work OK (see BUG: AttributeError: module 'numpy.linalg._umath_linalg' has no attribute '_ilp64' numpy/numpy#25150 (comment)). Consider moving "Quick Start" into a tutorial notebook and provide a Colab link. It looks like you already have it set up to convert Jupyter notebooks into html pages (e.g., https://emdgroup.github.io/baybe/examples/Constraints_Discrete/mixture_constraints.html)

• It appears that the README example is doing only a single iteration. I would have expected to see an optimization loop and some information about best parameters, though I get that this is geared more towards wetlab scientists.

• Maybe clarify in the README that people can choose from different scaling methods with a link to the docs? I eventually happened upon that part of the docs.

• I think the detailed installation information could go into an "Advanced Installation" section (either a separate README that gets incorporated into the docs, or near the end of the README). Within the main README in a "quick installation" section, then include a link to the advanced installation instructions. See Reorder README sections and print dataframe #95

• The README example doesn't have much by way of outputs (e.g., print statements and expected output). See Reorder README sections and print dataframe #95

• Same for visual representation, such as an optimization trace using BayBE on a task. Are there any built-in visualization methods? If not, consider including at least some examples of visualizing performance

Webpage

• It would be nice to have an "Edit on GitHub" link on your documentation pages -- it makes it a lot easier for others to contribute I think. See enable edit on github button #94

• It would be nice if the user guide linked to a corresponding tutorial or section of tutorials. For example, linking https://emdgroup.github.io/baybe/userguide/strategy.html to https://emdgroup.github.io/baybe/examples/Basics/strategies.html#

• At a glance, this was difficult to parse: Similar to the SequentialStrategy, the StreamingSequentialStrategy enables the utilization of arbitrary iterables to select recommender. Note that this strategy is however not serializable. (https://emdgroup.github.io/baybe/userguide/strategy.html#the-streamingsequentialstrategy). I think I kind of get it, but not necessarily when or how I would want to use it.

• I think there is too much granularity on some of your docs pages on pages like https://emdgroup.github.io/baybe/examples/Constraints_Discrete/Constraints_Discrete.html (i.e., lots of repeat, not a whole lot of valuable information gained from the bottom-most headings). No worries if this would be difficult to change.

• It would be nice to get some more details about each of the "Examples" sections rather than needing to click into each one to better understand what it's about. I.e., https://emdgroup.github.io/baybe/examples/examples.html could have some text at the top.

• As I'm going into more of the tutorials, I'm seeing that it's really comprehensive. For example, a demonstration of adding existing data https://emdgroup.github.io/baybe/examples/Backtesting/full_initial_data.html. I think there needs to be a better way to highlight/organize/point people to the tutorials they care about most. Happy to discuss more.

Terminology

Backtesting

I don't think "Backtesting" is common terminology for chem/materials informatics communities, at least in North America. It seems to be more common in finance, for example: https://en.wikipedia.org/wiki/Backtesting. When I wandered into https://emdgroup.github.io/baybe/_autosummary/baybe.simulation.html#module-baybe.simulation, I finally realized that what you refer to as simulation and backtesting is what I would typically refer to as benchmarking. I was thinking that maybe you implemented multi-task BO, where you could leverage physics-based simulations to help inform wetlab/experimental search campaigns. It took a while before this became clear to me.

Transfer learning

Right now, "Transfer learning: Mix data from multiple campaigns and accelerate optimization" is mentioned on https://emdgroup.github.io/baybe/misc/readme_link.html#, but it doesn't seem like this is really implemented yet, other than https://emdgroup.github.io/baybe/_autosummary/baybe.simulation.simulate_transfer_learning.html#baybe.simulation.simulate_transfer_learning. However, it doesn't appear to me that transfer learning is being used here. Even going through the function (https://emdgroup.github.io/baybe/_modules/baybe/simulation.html#simulate_transfer_learning), it was a bit tough to realize what was happening until I looked up TaskParameter. Suddenly, it made sense to me that what you're referring to as a task parameter is what I refer to as a contextual variable. This is also really good for me to see that contextual variable optimization is supported. However, I don't really consider this as transfer learning. In my mind, transfer learning means using one model to inform another. In contextual Bayesian optimization, certain variables are being fixed at each prediction. Perhaps I misunderstood something though. I imagine this will become clearer once https://emdgroup.github.io/baybe/userguide/transfer_learning.html has been developed.

Functionality

Multi-objective

It seems that Expected Hypervolume Improvement (EHVI) isn't one of the supported options for multi-objective optimization. Could you comment on this? With the DESIRABILITY mode, are each of the targets modeled independently prior to scalarization? If not, I tend to have a hard time referring to something like this as multi-objective optimization. In my mind, it's single-objective optimization of a fixed scalarization of several objectives. As alluded to in https://emdgroup.github.io/baybe/userguide/objective.html#desirability, it's good that a clarification is made about the scales being combined.

Batch conditioning

Do you perform conditioning on your batches (i.e., compute a joint acquisition function value)? For example, using fantasy point modeling. This is one of the easiest "gotchas" of batch optimization. See facebook/Ax#778 (comment) and https://youtu.be/JzgkSR6FFyM?si=dzv3RVvjKrZlkjlH

Comparison to other packages

What needs does BayBE fulfill that other packages don't? I think the README should clarify what makes BayBE stand apart from others and reference these other packages, too. For example, there's Ax (https://ax.dev), Gauche (https://github.com/leojklarner/gauche), Atlas (https://github.com/aspuru-guzik-group/atlas), Olympus, and https://github.com/experimental-design/bofire.

For example:

• Ax is a general-purpose tool, also built on BoTorch, but has to be retrofitted in many cases to support wetlab experiment setups
• Atlas is a nice framework, also based on BoTorch, but is not as well-maintained (single developer, non-recent commits - Riley is pretty busy)
• Olympus is a nice benchmarking framework for Bayesian optimization for chemistry and materials science and supports comparison of many algorithms across many datasets; however, it has the same issues as with Atlas. Also, it isn't as straightforward to apply this to custom datasets, and it's limited to single-objective optimization without categorical parameters, if I'm not mistaken.
• BoFire is also developed by chemistry/materials-oriented folks. It has been evolving, and is in a decently polished, though minimal state now. I don't think it natively supports chemical encodings, but it supports a number of other things, especially in terms of constraints (e.g., NChooseK constraints: https://experimental-design.github.io/bofire/nchoosek_constraint/)
• Gauche is the most similar to BayBE in my mind. This is one where I suggest looking closely and considering similarities and differences. One of these differences could be in the vision/roadmap you have for BayBE, which may not be the same roadmap Gauche is intending.

I keep what is probably an overly inclusive list of GitHub repos at https://github.com/stars/sgbaird/lists/optimization-and-tuning and a shortlist at https://github.com/AccelerationConsortium/awesome-self-driving-labs/blob/main/readme.md#optimization. I added BayBE to these lists recently.

I'm also interested to see an optimization comparison/benchmark of using the Mordred encoding with the solvent vs. treating it as a purely categorical variable.

Software development

• I can appreciate that BayBE seems well-maintained from a software developer perspective! This is welcome in the fields of chemistry and materials science, which understandably often lacks this.

• There are a lot of dependencies. I'm glad you split them up into groups!

• I notice you have a lot of >= dependencies in https://github.com/emdgroup/baybe/blob/main/pyproject.toml. Is this overly restrictive? It's OK if you don't think so.

• The docstrings look really nice, and it's nice to have the function cross-linking across the API docs.

• I look forward to seeing how you use hypothesis testing here!

Feel free to convert to a discussion if desired, and happy to refactor into multiple items if that would be better.

[Scienfitz]

Hey @sgbaird, first off many thanks for taking such a thorough look, we really appreciate it!
Had a chat with the other core devs (@AVHopp @AdrianSosic ) and will provide below our collective brief answers to your suggestion and comments.

Upfront, one general comment to explain the context: For our open-source release, we
followed the mantra "If you are considering releasing code, you probably should have done so earlier.", i.e. especially the docs are still very rudimentary to date and
we are working hard on improving them. So many of the existing pages are still
"placeholders" and many of the points you mentioned are already on our TODO list.
Nonetheless, thanks a lot for raising them!

A Colab notebook or similar would be really good I think. See e.g...

In principle we have all these examples which are already ready to use and very similar, but I guess it would
not be much work to publish them on Colab. We have no experience with it though, do you think this can be
automated as part of our docs pipeline? A nice thing would be a general open in collab button on each
example, but we would need to autopublish them with each deployment.

It appears that the README example is doing only a single iteration....

We want to keep the README lightweight re example code. We don't think it's much of a jump to understand how
all this would work in iterations when you've seen one iteration + we can link to the basic campaign
example which elaborates more and has a full loop. We can prob do a better job at the latter, currently
there's no link indeed.

Maybe clarify in the README that people can choose from different scaling methods with a link to the docs? I eventually happened upon that part of the docs.

We also think we can do a better job here, thanks for flagging. Again the README should be lightweight,
but there's a lot of complexity in designing the strategy part of the Campaign object, you can choose
all kinds of surrogate models, optimizers, recommenders etc. For all of this there are (and will be more)
examples and user guides so we should simply link them in the README without writing
too much about it

I think the detailed installation information could go into an "Advanced Installation" section...

Sure, lets discuss the details in #95

The README example doesn't have much by way of outputs (e.g., print statements and expected output). See #95

Makes sense to us, also part of #95

Same for visual representation, such as an optimization trace using BayBE on a task. Are there any built-in visualization methods? If not, consider including at least some examples of visualizing performance

Good suggestion, we indeed have one nice learning curve comparison on a public data set comparing chemical
encodings (something you requested below). It would not relate to the code in the README though as
again that should be minimal. It could prob still be mentioned as an example for advanced
usage (with link to example), what do you think?

Re visualization we don't plan to provide any builtins, we might consider granting access too much for
interesting info (acqf function values and so on) but plotting it is not intended functionality of this package

It would be nice to have an "Edit on GitHub" link on your documentation pages -- it makes it a lot easier for others to contribute I think. See #94

We have all agreed that this is a good suggestion, but the suggested implementation does not work for
most pages because most of the docs content is built on demand, the built-in edit button links to these
temporary files causing a 404. So unless autodoc and examples can be excluded or configured
separately we don't see the change merged.

It would be nice if the user guide linked to a corresponding tutorial or section of tutorials. For example, linking https://emdgroup.github.io/baybe/userguide/strategy.html to https://emdgroup.github.io/baybe/examples/Basics/strategies.html#

Absolutely agree. You have encountered our doc in a not fully completed state, there are
three versions of files in the userguide i) basically empty ii) minimal content and iii)
detailed content (eg parameters or constraints). We have tried to do what you suggest
in iii) and will over time convert all other pages to the state of iii) too, so your
suggestion will automatically be included

At a glance, this was difficult to parse: Similar to the SequentialStrategy, the StreamingSequentialStrategy enables the...

We will provide more context and a usage idea on the more detailed userguide upcoming

I think there is too much granularity on some of your docs pages on pages like ... (i.e., lots of repeat, not a whole lot of valuable information gained from the bottom-most headings). No worries if this would be difficult to change.
It would be nice to get some more details about each of the "Examples" sections rather than needing to click into each one to better understand what it's about. I.e., https://emdgroup.github.io/baybe/examples/examples.html could have some text at the top.

Agreed, we will remove the subsections from it and have a small summary sentence on top of each section.

As I'm going into more of the tutorials, I'm seeing that it's really comprehensive. For example, a demonstration of adding existing data https://emdgroup.github.io/baybe/examples/Backtesting/full_initial_data.html. I think there needs to be a better way to highlight/organize/point people to the tutorials they care about most. Happy to discuss more.

Let's put this in a separate discussion, as we are also very interested in your thoughts of how to make said connection better, it is pretty crucial indeed.

Backtesting
I don't think "Backtesting" is common terminology for ...

We've had similar thoughts, but overall the issue is that you can not only use our simulation utilities for
backtesting (a la finance) or benchmarking (a la backtesting while comparing settings or codes), but also
use an oracle for lookup, which might not be historical data but simulation-based ground truths. For
instance reactors can be simulated comprehensively with differential equations approximating reality well,
we would use baybe simply to find the optimal rector settings using the same utilities. This is not well
described as benchmark or backtest. But we lack a term describing both scenarios. Personally I
prefer backtesting to benchmarking because i) its very similar use in finance is actually a good
analogy and ii) We can also use it to test settings with not referencing any other works or algorithms or
existing metrics etc, ie not really benchmarking

Right now, "Transfer learning: Mix data from multiple campaigns and accelerate optimization" is mentioned on ..., but it doesn't seem like this is really implemented yet, other than ....

Two things. Indeed here we currently have the problem that we have no example and user guide for the
transfer learning, hence giving the impression that it might not be implemented. But it is fully
functioning and simple to use via TaskParameter (only takes effect when using a GP surrogate though).
This scenario would combine data from the contexts/campaigns while recommending values only for
the third (currently ongoing campaign):

TaskParameter(name='Scenario', values=['Oct2023', 'Dec2023', 'Jan2023'], active_values=['Jan2023'])

This can be abstracted easily to other scenarios such as changing substrates
(while screening same reaction conditions) or formulating mixtures for different cell lines.

The terminology here is tricky. The act of combining the data of several contexts can be
done via multiple models, shared architectures, special kernels in a single model and whatnot.
We do not necessarily want to limit to any of these methods (while currently we only have one).
Feel free to open a separate discussion thread if you have more suggestions and or comments here

Multi-objective
It seems that Expected Hypervolume Improvement (EHVI) ...

Indeed this is already on our roadmap as we fully agree with the downsides and limitations of scalarizing.
The main issue at the moment is to integrate multiple targets into the entire framework,
not choosing the right botorch settings. Will come at some point.

Batch conditioning
Do you perform conditioning on your batches (i.e., compute a joint acquisition function value)? For example, using fantasy point modeling. This is one of the easiest "gotchas" of batch optimization. See ...

Indeed, we are using joint batch optimization in the main and default optimizer SequentialGreedy.
Currently, this is explained in a box in the
campaign userguide.
Probably, it should also be mentioned somewhere in the strategy guide. Do have any
suggestion where you would ideally like to see info about it?
As a side note: the method is not based on fantasizing but uses the "pending points"
approach explained in the botorch paper, which is implemented in botorch's standard
optimization routines.

Comparison to other packages
What needs does BayBE fulfill that other packages don't? I think the README should clarify what makes BayBE stand apart from others and reference these other packages, too....

It's very tricky to maintain such a list, imagine we do all the work of working out exactly all
differences and write it into the README only to find its outdated 3 weeks later... Hence
instead we have chosen to highlight the main features of BayBE at the very top of the README,
the user can choose whether they think that's available in other packages and needed for them.
Thanks also for adding our code to your list, we think this is a much better place to have a
more central place where codes are compared as opposed to having an outdated comparison in
each code README itself

I'm also interested to see an optimization comparison/benchmark of using the Mordred encoding with the solvent vs. treating it as a purely categorical variable.

As described above we have a plot showing nicely the tremendous impact these can have, it is probably a nice plot to show already on the README

I notice you have a lot of >= dependencies in https://github.com/emdgroup/baybe/blob/main/pyproject.toml. Is this overly restrictive? It's OK if you don't think so.

I'm not sure why you consider them over restrictive? I guess they are even less restrictive than ~=,
but overall we try to keep the allowed versions low (except for main ones like botorch where we
want latest features of course). So far we haven't heard from anyone expressing problems due to restrictive
dependencies in practice but we will keep looking out