Finetuning fails in Colab

[nischa564]

When we run the demo 04_Finetuning.ipynb, Colab throws an error after the following steps:

1. We install the necessary packages
2. We restart the runtime and run the imports, the hyperopt throws the error:

`ValueError Traceback (most recent call last)
in ()
1 # loss in this case refers to -f1
2 # search strategy should be in ['parzen', 'random']
----> 3 model = finetune_hyperopt(X, y, strategy='random', models=models, cv=5, max_evals=32, mtype='C')

9 frames
/content/DataFactory/datafactory/finetuning/finetuning_hyperopt.py in finetune_hyperopt(X, y, strategy, models, params, max_evals, cv, mtype)
73 algo=algo,
74 max_evals=max_evals,
---> 75 trials=trials)
76
77 best_model = trials.results[np.argmin([r['loss'] for r in trials.results])]['model']

/usr/local/lib/python3.7/dist-packages/hyperopt/fmin.py in fmin(fn, space, algo, max_evals, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin, points_to_evaluate, max_queue_len, show_progressbar)
386 catch_eval_exceptions=catch_eval_exceptions,
387 return_argmin=return_argmin,
--> 388 show_progressbar=show_progressbar,
389 )
390

/usr/local/lib/python3.7/dist-packages/hyperopt/base.py in fmin(self, fn, space, algo, max_evals, rstate, verbose, pass_expr_memo_ctrl, catch_eval_exceptions, return_argmin, show_progressbar)
637 catch_eval_exceptions=catch_eval_exceptions,
638 return_argmin=return_argmin,
--> 639 show_progressbar=show_progressbar)
640
641

/usr/local/lib/python3.7/dist-packages/hyperopt/fmin.py in fmin(fn, space, algo, max_evals, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin, points_to_evaluate, max_queue_len, show_progressbar)
405 show_progressbar=show_progressbar)
406 rval.catch_eval_exceptions = catch_eval_exceptions
--> 407 rval.exhaust()
408 if return_argmin:
409 return trials.argmin

/usr/local/lib/python3.7/dist-packages/hyperopt/fmin.py in exhaust(self)
260 def exhaust(self):
261 n_done = len(self.trials)
--> 262 self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
263 self.trials.refresh()
264 return self

/usr/local/lib/python3.7/dist-packages/hyperopt/fmin.py in run(self, N, block_until_done)
225 else:
226 # -- loop over trials and do the jobs directly
--> 227 self.serial_evaluate()
228
229 try:

/usr/local/lib/python3.7/dist-packages/hyperopt/fmin.py in serial_evaluate(self, N)
139 ctrl = base.Ctrl(self.trials, current_trial=trial)
140 try:
--> 141 result = self.domain.evaluate(spec, ctrl)
142 except Exception as e:
143 logger.info('job exception: %s' % str(e))

/usr/local/lib/python3.7/dist-packages/hyperopt/base.py in evaluate(self, config, ctrl, attach_attachments)
831 def evaluate(self, config, ctrl, attach_attachments=True):
832 memo = self.memo_from_config(config)
--> 833 use_obj_for_literal_in_memo(self.expr, ctrl, Ctrl, memo)
834 if self.pass_expr_memo_ctrl:
835 rval = self.fn(expr=self.expr, memo=memo, ctrl=ctrl)

/usr/local/lib/python3.7/dist-packages/hyperopt/utils.py in use_obj_for_literal_in_memo(expr, obj, lit, memo)
167 for node in pyll.dfs(expr):
168 try:
--> 169 if node.obj == lit:
170 memo[node] = obj
171 except AttributeError:

/usr/local/lib/python3.7/dist-packages/pandas/core/generic.py in nonzero(self)
1328 def nonzero(self):
1329 raise ValueError(
-> 1330 f"The truth value of a {type(self).name} is ambiguous. "
1331 "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
1332 )

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().`

[HuangYiran]

1. incomp..
   install the package with:
   !pip install cloudpickle==1.6.0
   !pip install imgaug==0.2.6
   !pip install scipy==1.7.3

remove the installation:
!pip install scipy==1.5.0

2. ambigious error
   remove the X and y from the search space and put it in the objective function may solve the problem

[riedel]

I would encourage using the %pip cell magic and make it one line btw:

%pip install cloudpickle==1.6.0 imgaug==0.2.6 scipy==1.7.3

or better put all the requirements just in a file

[riedel]

the whole code seems a bit messed up anyways:

• why do you only need dependencies on colab and nowhere else
• why does the auto-sklearn package install its dependencies?
• and why do you do this curl madness instead of pip install -r https://raw.githubusercontent.com/automl/auto-sklearn/master/requirements.txt
• if you are cloning your own repo anyways why not actually put all this stuff into a local requirements.txt that you can simply install via pip install -r