Overlap with sklearn

[amueller]

Hey folks!
This looks like a really cool library that I hadn't seen before.
It looks like there's a couple of things we now have upstream in sklearn and I wanted to note that so you could maybe add that to the docs.

Also, I'd be really curious to see which parts could be upstreamed. Contributing to sklearn can be a bit of a lengthy process, but if some of your tools are generally useful it might be nice to put them into sklearn.

Btw, TrainOnlyTransformerMixin is quite interesting. I was wondering how you do that because the scikit-learn API doesn't allow it (which I dislike), but it seems you're hashing and checking for equality? That's certainly an interesting approach. Ideally I'd like to fix the API so this isn't as awkward.

I think EstimatorTransformer is the same as the sklearn VotingClassifier, right? Though maybe that's not super clear from the name?

And you're probably aware that ColumnSelector + FeatureUnion has finally after a long wait been implemented in ColumnTransformer. So this example is actually significantly shorter in sklearn now:
https://scikit-lego.readthedocs.io/en/latest/preprocessing.html#Example-2

Most cases of IdentityTransformer are now handled with the special 'passthrough' string.

I thought RandomRegressor is the same as DummyRegressor but apparently we don't have a sampling strategy for DummyRegressor. It think that would be a nice addition if you're interested.

[amueller]

Having a callback for logging in pipelines might also be nice to upstream...

[koaning]

Hi @amueller happy to hear you like some of our experiments. I'll note on your comments one by one here but I imagine @MBrouns might also have some thoughts to share.

The idea behind the EstimatorTransformer is that sometimes, the output of a model is something you'd like to add as a feature. Say, you'd like to add the output of Kmeans to a RandomForestRegressor. Then you can wrap EstimatorTransformer(Kmeans()) and this effectively turns Kmeans.predict() into Kmeans.transform() while keeping everything compatible for a pipeline/gridsearch. This means that clustering and regression can also be featurized. Does the voting classifier also handle that? The name doesn't imply it.

On the ColumnSelector/FeatureUnion we were indeed aware of some work there, but I recall at the time that we implemented them that it was still in "experimental" status. I personally enjoy the implementation we have because at all times everything remains a lego brick that you can always click in a pipeline or a feature-union but this is highly personal.

I wasn't aware of the passthrough tho so I'll have a look.

[MBrouns]

Hey @amueller! Thanks for reaching out. I think it's definitely worth it to contribute some changes upstream.

The TrainOnlyTransformerMixin was definitely a bit awkward to implement this way, and it also doesn't pass check_estimator fully because the NotAnArray that's used for testing doesn't really lend itself to being hashed in this way. Changing the API upstream to allow for this sounds like quite a drastic change, but it might be worth it. What would the process for discussing something like this be? Open a SLEP?

In a similar vein, I'd also love the option to change the number of observations inside a pipeline, to make for example outlier removal an explicit part of the fitting pipeline. I get that that's not really desirable on the test data so having different operations on train and test might open the door for something like this as well.

The EstimatorTransformer does indeed seem to do what the VotingClassifier also does, but I wouldn't have ever figured that out without you pointing it out. There might be room for having it as a separate named entity to facilitate serendipity. What's your opinion on that?

I think the randoms strategy and the log pipeline are some things that could be upstreamed relatively easily. What would that process look like? Open up an issue with the suggestion and see what comes out of that?

[koaning]

To extend @MBrouns's point. We're a little bit obsessed with outliers and other reasons to assign doubt to an algorithm. In particular, we'd love for a pipeline to be able to say "wont predict" in the case that we detect, say, an outlier in production. The point is made clear in the example in this 5 min pydata segment. We think it'd be great to have systems that are able to recognize when they should not automate but instead should consider a fallback scenario.

[amueller]

Thanks for all the replies folks!
I'm not sure if having a different class for VotingClassifier is useful, but we might need to document it better. I think there was another library that had reimplemented it because they overlooked it. What name would you suggest? I wonder if we can somehow fix that with documentation? It's definitely not very discoverable.

Re SLEP: the first ever slep actually includes something like that https://scikit-learn-enhancement-proposals.readthedocs.io/en/latest/slep001/proposal.html
There's another slep on resampling that also addresses this a bit:
scikit-learn/enhancement_proposals#15

My opinion is here:
scikit-learn/enhancement_proposals#15 (comment)

Basically for resampling you have the same problem, that's why you can't do resampling in sklearn.

Re upstreaming: sure, open an issue! There might already be an issue for logging in pipelines?

I'm not entirely convinced by automatic outlier detection - not because I think this is not a problem, but because I'm not sure that we can automatically detect whether the problem happens. But I'm happy to be convinced otherwise ;) Why can't you do that right now with a meta-estimator?

[koaning]

@amueller We're certainly investigating a meta-estimator but we're not super sure on what the best behavior is.

We could do something like:

mod = WontPredict(classifier_pipeline, outlier_pipeline)
mod.predict(X)

But what is good behavior now when it detects an outlier? Should we throw an error? Should we assign a class of -1 to outliers, overwriting the classes? Should we instead use np.nan? Can we do the same thing with regression/clustering?

It should be said though, there can also be other reasons to raise WontPredict. There's usecases for fairness that may also be valid. If a fair model and an accurate model disagree, then maybe that's also a good proxy not to automate a decision.

[amueller]

Well this seems most relevant in a production setting, right? There you probably don't want your model to crash (unless the surrounding code catches it). So you could either predict a placeholder like -1 or predict the mean or most common class.
Probably the best way to deal with this is handle it downstream in your application, I'd think.

[MBrouns]

I'm not entirely convinced by automatic outlier detection - not because I think this is not a problem, but because I'm not sure that we can automatically detect whether the problem happens.

Fair enough, but some basic domain checks can already help in the case that an upstream system suddenly changes the scale of their data. There's at least a grey area where having some protection against simply making a prediction could be useful.

Why can't you do that right now with a meta-estimator?

I think the main issue is with metrics. If we consider it an outlier / WontPredict instance, do we want to consider it an error for scoring? For a classifier you could argue that it's a form of mislabelling but for a regression that doesn't really work. So probably you'd want to calculate the error on the non-outlier cases and report on how many cases are classified as outliers. AFAIK this is a bit hard atm with for example a GridSearch as the y_true for the metric functions is the y that gets passed into the pipeline

[koaning]

@amueller one question out of curiosity. How did you find this repo?

[amueller]

The comment on the fairness package for scikit-learn-contrib ;)

[koaning]

@amueller I've added an issue where we can discuss the addition of a sampling strategy in scikit-learn. I'd love to contribute.