🚀 Feature Request

Delegate experiments scheduling to launcher and add feedback loop from launcher to sweeper/experiment generator to allow for stream processing with ProcessPoolExecutors and similar. Mainly to increase utilization of resources and avoid waiting for all batch processes to finish.

Motivation

Is your feature request related to a problem? Please describe.
My team uses optuna for hp sweeps of ML model with different training time, so if for example we use 8 gpu server and train 8 models in parallel with eg joblib launcher we can waste up to 1/4 walltime waiting for all batch jobs to finish.

Pitch

Describe the solution you'd like
I would like to propose creating experiment generator class (so it's customizable). Sole porpoise of this class is returning new configuration and receiving experiment results to update study.
I have my implementation based on generator's send method and it's working pretty good with my own loky launcher(aka concurrent.features.ProcessPoolExecutor) which I can also contribute.

Describe alternatives you've considered

it's the only way we can achieve high gpu utilization.
Alternatively we can adopt optuna plugin to accept futures and manage them, but I think job launching and result awaiting is a role of launcher
Are you willing to open a pull request? (See CONTRIBUTING)
Yes I can open pull request with my implementation if this feature is interesting

Additional context

Add any other context or screenshots about the feature request here.