using mlxtend with statsmodels

[FlorinAndrei]

I want to do in Python things that are trivial to do in R. For example, search for the best model, perhaps using ExhaustiveFeatureSelector(), but using AIC or BIC as the search criteria.

statsmodels calculate AIC and BIC by default for their models, and expose them as attributes. That would simplify things a lot.

[rasbt]

I think the challenge here is that ExhaustiveFeatureSelector() is meant to be somewhat "general purpose" with respect to classification and regression models, but the AIC & BIC cannot be easily calculated for all models. E.g., considering a decision tree, random forest, or k-nearest neighbor classifier that are not fit via maximum likelihood, for example, I wouldn't know how to compute the AIC (I don't think it makes sense for those).

I haven't thought about this too deeply, but it could be possible to generate a custom scorer function that supports that, but this would be very specific to the model and take some work (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer)

There do seem to be some specialized implementations of certain models that support AIC and BIC, and the exhaustive feature selection could be applied as an outer loop to those https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLarsIC.html#sklearn.linear_model.LassoLarsIC
But yeah, it's a bit of a hack.

[FlorinAndrei]

I think - and please correct me if I'm wrong - if mlxtend was able to use models from the Statsmodels library as well, instead of just Scikit-learn, then using AIC, BIC, etc would be trivial, since Statsmodels tends to compute those values for free, it's just an attribute in the fitted model.

Combining that with the Formula models in statsmodels.formula.api would be very useful - the ease of manual model exploration from R, but written in Python, along with all the production-ready tools from Python.

The main thing here is how and when you pass data into the model - Statsmodels and Scikit-learn differ quite a bit, the former wants data from the very moment you define the model, while the latter only requires data when you call .fit(). If it wasn't for this difference, writing a wrapper for Statsmodels would be trivial.

[rasbt]

Yes, it would be awesome if mlxtend would be compatible with statsmodels in addition to scikit-learn. I am not sure if it is trivial, because ideally it shouldn't negatively affect the current scikit-learn support. Like you suggest, if mlxtend supported statsmodels, we could have an optional class constructor argument for using AIC, BIC for statsmodels models.

To be honest, I'd love the idea to support statsmodels models in mlxtend, but I am not sure how much work that would involve and whether I would have the capacity to work on sth like this in the near future.