ENH: Double-double arithmetic for precision operations with float64s

[eicchen]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

I was working on #59652 where I have run into issues with float64 losing precision, causing our Pearson correlation to go over the [-1,1] bound due to loss of precision. I currently have some data-recentering, but we should probably implement something to better support edgecases like this long-term.

Feature Description

I propose implementing some form of double-double operations (Veltkamp-Dekker) so we can implement it wherever needed. The main benefit are the improved precision and being opt-in and minimally intrusive, with no new library needed.

We can't use np.longdouble or np.float128 as they are unavailable on Windows so it could help it work everywhere.
Additionally, this would be more wildly applicable than just addition operations (Kahan)

Alternative Solutions

Alternatively, we could use decimal.Decimal, mpmath, or the QD library, but these are either significantly slower or require external dependencies. I'd expect the primary use case to be in Cython though, so I'm not sure what the best way forward there would be.

Additional Context

If people think its a good idea, I can take it as it ties directly to a PR I'm currently working on

[rhshadrach]

I would be cautious here. There will always be precision issues, and there will always be something more one can do about it. The question becomes how much is too much, especially considering how many users run into problems. It seems to me, just based on my own experience, that for 99.9% of usages, pandas current algorithm is sufficient. That said, if there are improvements that could improve stability without significant performance or maintenance cost, I think they can definitely be considered.