Automatic step sizes for SVRG #207
Conversation
|
This needs to be more exhaustively tested to make sure it works on real datasets. |
|
The regularization strength might have to be added to the |
|
This PR provides the infrastructure for computing an optimal stepsize and batch_size for SVRG based on the GLM configurations. The optimal hyperparameters depends on the loss function L-smoothness. This means that for each model configuration (observation noise, link function, regularization), one may need to to compute a different estimate of the smoothness parameters. Here, I implemented a look-up table that should be easy to extend whenever new estimates becomes available (for example if we derive the L-smoothness for Gamma + softplus observations). |
|
With my edits, I added:
|
| else: | ||
| assert opt_state.stepsize > 0 | ||
| assert isinstance(opt_state.stepsize, float) | ||
| model.fit(X, y) |
There was a problem hiding this comment.
I would consider removing model.fit() calls in this test (as well as in test_glm_optimal_config_set_initial_state_pytree, and both functions with the same names in the TestPopulationGLM class) since its not being consistently called across all cases, and since no checks are happening after the model is fit
| else: | ||
| assert ( | ||
| "stepsize" in result and result["stepsize"] > 0 | ||
| ), "Stepsize should be computed since it was not provided." |
There was a problem hiding this comment.
I don't know if I missed it in one of the previous tests, but I didn't notice any test that explicitly tests the result of a computed stepsize. They only check that stepsize > 0. Is it worth it to have a test check the result explicitly, similar to test_calculate_optimal_batch_size_svrg_all_config at the end?
Attempt to automatically determine the batch- and step sizes for SVRG when fitting a GLM with Poisson observations and a softplus inverse link function.
Based on this paper.