Inconsistent functions in ConformityScore class: seeking explanation

[jesuinovieira]

Hey there! I was wondering if someone could kindly help me understand why I'm encountering the following issue:

ValueError: The two functions get_conformity_scores and get_estimation_distribution of the ConformityScore class are not consistent. To resolve this, the equation self.get_estimation_distribution(y_pred, self.get_conformity_scores(y, y_pred)) == y must hold true. Currently, the maximum conformity score is 3.0517578125e-05. If you're confident that the two methods should be consistent, you might need to increase the eps attribute.

I would greatly appreciate it if someone could shed some light on this matter and provide some insights or possible reasons behind this issue. Thank you so much!

[jesuinovieira]

Discussed in #321. Feel free to delete this discussion, it seems that I can't do it.

[thibaultcordier]

Copy/paste the issue #321 to answer satisfactorily:

Interest of this exception

This exception has been raised for several reasons. First of all, what is the motivation behind this exception?

• Every MapieRegressor object must instantiate a ConformityScore object to calculate the non-conformity score (see the conformal prediction theory).
• The method that raises this exception is check_consistency in the ConformityScore class (see here), which checks consistency between the following methods get_estimation_distribution and get_signed_conformity_scores.
• In other words, it verifies that from the actual target, we can calculate the predicted target based on the non-conformity score and vice-versa without loss of precision (we're checking the bijective property of the method in fact).
• The purpose of this check is to ensure that if a user wants to create a new ConformityScore, he must check that his implementation is mathematically consistent (based on the computer precision). This is not the case for you, as you are using the default value proposed by the MAPIE library.

Particular interpretation of your exception

In your case, this exception tells us that some of your data is having difficulty being recalculated with high precision in the ConformityScore object (by default, MapieRegressor uses AbsoluteConformityScore (see here).

If the error occurs about 15% of software executions, it's because you haven't defined a random seed: the train_test_split function doesn't produce the same split in your data and so some toxic data isn't in your calibration set.

random_state = 42

train, calib = train_test_split(data, test_size=0.25, shuffle=True, random_state=random_state)

After all, I could assume that some of your data is extremely small or large compared to the computer precision and that this causes problems in the consistency check.

Resolution of the exception

But don't worry, because the method proposed by MAPIE is mathematically valid, and this is an extreme error involving computer precision, we can manually remove this check. Here's the code to do just that:

from mapie.conformity_score import AbsoluteConformityScore

my_conformity_score = AbsoluteConformityScore()
my_conformity_score.consistency_check = False

mapie = MapieRegressor(model, cv="prefit", conformity_score=my_conformity_score)

Conclusion

I admit I've never seen this kind of exception raised in this kind of situation. This is why the way to solve the issue is not aesthetic. But this issue could be the subject of a future readjustment to more easily control this parameter as soon as the object is initialised.

I hope I've been able to help you solve your issue. Don't hesitate to let me know if the solution works for you.