Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When fitting a lasso, is normalization performed even if fit_intercept=False? #488

Open
mathurinm opened this issue Nov 23, 2021 · 0 comments

Comments

@mathurinm
Copy link

I'm trying to understand what's the stopping criterion on the (sub)gradient norm, in order to scale gradient_tol correctly.

If I understood correctly, for a quadratic datatif/family="gaussian", and vanishing coefs, the grad argument of _norm_min_subgrad should be X.T @ y / len(y)

When I debug, I see this being true only when I have previously centered and normalized X. otherwise, even if in my GeneralizedLinearRegressor I have set fit_intercept=False, I see:

  • a P1 array which is not constant
  • in update_quadratics(), gradient_rows is equal to y / len(y) as expected, however grad = gradient_rows @ data.X does not yield a grad equal to X.T @ y / len(y).
    It seems this is due to X being a
    Mat: <class 'tabmat.dense_matrix.DenseMatrix'> of shape (10, 5). Shift: [0. 0. 0. 0. 0.] Mult: [1.03404146 1.22010398 0.81522932 1.0686763 1.27198591] so scaling happens.

Why is this the case eventhough clf.scale_predictors, and clf._center_predictors are False ?

To reproduce, print gradient_rows and grad in update_quadratics, and run:

from glum import GeneralizedLinearRegressor
import numpy as np

np.random.seed(0)
X = np.random.randn(10, 5)
y = np.random.randn(10)
X -= X.mean(axis=0)
alpha = 0.001


clf = GeneralizedLinearRegressor(
    alpha=alpha, gradient_tol=1000, fit_intercept=False, family="gaussian",
    l1_ratio=1, verbose=10).fit(X, y)

print("grad rows should be", y / len(y))  # it is
print("grad should be", X.T @ y / len(y))   # it is not!

If you center and scale X before, then the problem disappears

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant