Error when frameworks installation script fails

[PGijsbers]

If we detect an error during setup, we want to stop the benchmark process immediately, instead of continuing.
Currently, the failed process goes unnoticed and often results in unexpected errors like missing dependencies during the execution of the exec.py script.

[eddiebergman]

It seems it would be easier to detect this at the level of the subprocess call, rather than injecting a -e, this in effect offloads to the responsibility to the creator of the script to exit with the correct status code.

This would require a template that includes and documents a -e call in there, so people are aware of it if not super familiar with writing bash scripts. People writing a script for Windows, of which I imagine there are not too many, would have to do so in the way expected of a .bat file or whatever it is.

[PGijsbers]

Thanks for your thoughts!

detect this at the level of the subprocess call

Sorry, I am not 100% sure I understand. Are you suggesting that the scripts themselves invoke set -e or similar? Meaning effectively no change, other than perhaps documenting and encouraging the use of set -e for setup scripts? E.g., include it in a template such as https://github.com/PGijsbers/amlb-template/blob/main/interpretml/setup.sh.
Since the integration scripts are user code (from the benchmark perspective), I'd generally prefer a more defensive approach. Though I could live with simply embedding it in current scripts + a template, if setting it through the automl benchmark is a bad idea.

Windows isn't officially supported, doesn't work with any current setup scripts, and we currently have no plan to support it (outside of docker mode), so I would prefer to find the best solution for unix-based machines.

[eddiebergman]

It's not too much of an issue in the end, just need to ensure that using set -e twice isn't an issue.

I'm usually for offloading these issues to the user, namely just because it's hidden behaviour and might end up with their own manual running being different than what the benchmark does. For example, the git clone <src> <dest> will error if <dest> exists, but this is not always an issue for the setup, as you might still want to checkout a certain version.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.90%. Comparing base (b719142) to head (f361b3c).
Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #685      +/-   ##
==========================================
+ Coverage   68.15%   68.90%   +0.74%     
==========================================
  Files          54       55       +1     
  Lines        6730     6757      +27     
==========================================
+ Hits         4587     4656      +69     
+ Misses       2143     2101      -42     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[PGijsbers]

Related to #681