Hyperparameter tuning best practices.

[drisspg]

I have recently switched over an existing pipeline from spacy 2 to spacy 3. Previously I was training the components in the pipeline with a custom script based off the old documentation and not the spacy train cli option. Since switching I have migrated the training to use the cli. I am now taking a looking at hyper-parameter tuning the multi-label text classifier and the NER components of the pipeline. The most documentation I could find on a procedure for doing so was: https://github.com/explosion/projects/tree/v3/integrations/wandb. While this should work it doesn't seem to take advantage of some of the stated integrations: ray and wandb, both of which have powerful tuning frameworks. I was wondering if anyone has tried to integrate spacy hyperparamter tuning with either ray[tune] https://docs.ray.io/en/master/tune/index.html or W&B Sweeps: https://docs.wandb.ai/guides/sweeps? If so did you end up basically reimplementing spacy.training.loop.train() but with some code injection to register and monitor the appropriate metrics needed for either of these two frameworks.

[psimm]

What do you mean when you say that the guide at https://github.com/explosion/projects/tree/v3/integrations/wandb doesn't make use of sweeps? The train-search command runs a wandb sweep. Or do you mean combining wandb and ray to do the sweep runs in parallel?

I have used the wandb integration and set up hyperparameter tuning using wandb sweeps. I used the sweeps_using_config.py way and created a spacy project command that uses python -m wandb sweep sweep.yml.

The only thing that I couldn't get to work was overrides. Does anyone know how to use wandb sweep and set overrides such as --gpu-id ${vars.gpu}?

[svlandeg]

What do you mean when you say that the guide at https://github.com/explosion/projects/tree/v3/integrations/wandb doesn't make use of sweeps?

This was true when the original post was written, as this was changed in August: explosion/projects#74

[psimm]

Thanks for clarifying that, I forgot to check the date of the original post.