SLEP025: Losing Accuracy in Scikit-Learn Score

[lorentzenchr]

What

This SLEP is for finding a consensus on how to removing accuracy as a default metric for classifiers which currently is: classifiers.score.

Why

Because accuracy has many severe weaknesses and we even baked a probability threashold of 50% into it.

What else

I hope the irony of the title is not lost by its passive aggressiveness.

This SLEP is based on @amueller's proposal scikit-learn/scikit-learn#28995.

@scikit-learn/core-devs @scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/documentation-team ping

[jjerphan]

Thank you for writing the SLEP.

While discussing the rectification of this design choice is appropriate (I really think it is valuable), I find the title a bit aggressive or at least detrimental to the intent of the proposal (improving scikit-learn's theoretical consistency): how about "Redefining default metrics"?

I think this could be implemented as part of scikit-learn 2.0.

[reshamas]

@lorentzenchr How about: Removing the accuracy metric in scikit-learn scoring

[betatim]

Do we need to choose the same metric for all classifiers?

I think the answer is yes because people will use the results of est1.score(X, y) and est2.score(X, y) to evaluate which one is the better estimator. It seems very hard to educate people that they can't compare scores from different estimators

(This is almost a rhetorical question, but I wanted to double check my thinking)

[lorentzenchr]

Given your assumption that users will continue to compare score results of different estimators, and given that a generally satisfying metric does not exist, the conclusion is to remove the score method.

My currently best choice for a general classifier metric is the skill score (R2) variant of the Brier score. Classifiers and regressors would then have the same metric, which is nice.

[betatim]

I'm not sure I am ready to remove score().

[lorentzenchr]

It is listed as alternative, not as proposal!

[lucyleeow]

What about 'Replacing accuracy' - it reads clearer IMO and could mean that we replace it with another metric or replace it with 'nothing'.

[thomasjpfan]

• I think we need to decide on a default by step 2, so we can tell users what to set to emulate the new default.
• I am okay with 1 and 2 happening at the same time as long as we choose a default. If we have not chosen a new default, then only 1 can happen.

[betatim]

I think if we can't agree a new default we should not start the process. I think this because agreeing the new default is the hard part of this task and if we start it we are on the hook for finishing the transition, which will be impossible without agreement.

Otherwise I agree with Thomas that we can do 1 and 2 at the same time.

[lorentzenchr]

I included it as proposal.

[thomasjpfan]

I feel like this needs a longer than the usual 2 minor releases. Personally, I'll be okay with 3 minor releases or 1 major release.

[betatim]

My assumption is that most people who blindly use score() do not know better. It is unclear to me if forcing them to make a decision is going to improve the quality of the decision they make. scikit-learn is about "machine learning without the learning curve", so we are on the hook for making a "not unreasonable" decision for the beginner user.

It doesn't stop us from extending our documentation and educational material to increase the chances of people reading it (eg we could have a blog post about this topic when the deprecation starts to explain why this is a much bigger deal than it might seem) and hopefully making a better decision than the default argument to score().

[lorentzenchr]

if forcing them to make a decision is going to improve the quality of the decision they make

Let's go through the options:

• a lucky user chooses a strictly consistent scoring function like log-loss or brier score: situation improved
• an informed user chooses a metric close to a business/application metric he has: situation improved
• a stubborn (or unlucky) user chooses accuracy or something similar (balanced accuracy, F2, you name it): situation is not worse.

Qunitessence: Situation can only improve, but never worsens.

[lorentzenchr]

I improved (I hope so) the writing, clearness of proposal, reasoning and alternatives. May I ask someone to approve? Or can/shall I merge myself?

I'd like to call for a vote soon.

[thomasjpfan]

For the multi-class case, will the brier_score_loss be normalized to be in the [0, 1] range?

Specifically, do we set scale_by_half=True in brier_score_loss?

[lorentzenchr]

Yes, I would do that.

[lorentzenchr]

On a second thought: it does not matter. The R^2 version is invariant to scaling: Brier(model) / Brier(mean of data) = MSE(model) / MSE(mean of data).