LinAlgError: singular matrix after 2to3

[rgaiacs]

I run 2to3, see rgaiacs@1e10d5d, but when I rerun notebooks/tutorial.ipynb I got

---------------------------------------------------------------------------
LinAlgError                               Traceback (most recent call last)
<ipython-input-9-00bdbb9c2e31> in <module>()
      3 ax1 = entrofy.plotting.plot_correlation(df, "age", "age", ax=ax1,
      4                                         xtype="continuous",
----> 5                                         ytype="continuous", cont_type="kde")
      6 ax2 = entrofy.plotting.plot_correlation(df, "age", "age", ax=ax2,
      7                                         xtype="continuous",

/home/raniere/anaconda3/lib/python3.5/site-packages/entrofy-0.0.0-py3.5.egg/entrofy/plotting.py in plot_correlation(df, xlabel, ylabel, xmapper, ymapper, ax, xtype, ytype, cmap, prefac, cat_type, cont_type, s)
    515     elif ((xtype == "continuous") & (ytype == "continuous")):
    516         ax = _plot_continuous(df, xlabel, ylabel, ax, plottype=cont_type,
--> 517                               n_levels=10, cmap="YlGnBu", shade=True)
    518 
    519     else:

/home/raniere/anaconda3/lib/python3.5/site-packages/entrofy-0.0.0-py3.5.egg/entrofy/plotting.py in _plot_continuous(df, xlabel, ylabel, ax, plottype, n_levels, cmap, shade)
    368     if plottype == "kde":
    369         sns.kdeplot(x_clean, y_clean, n_levels=n_levels, shade=shade,
--> 370                     ax=ax, cmap=cmap)
    371 
    372     elif plottype == "scatter":

/home/raniere/anaconda3/lib/python3.5/site-packages/seaborn-0.8.1-py3.5.egg/seaborn/distributions.py in kdeplot(data, data2, shade, vertical, kernel, bw, gridsize, cut, clip, legend, cumulative, shade_lowest, cbar, cbar_ax, cbar_kws, ax, **kwargs)
    651         ax = _bivariate_kdeplot(x, y, shade, shade_lowest,
    652                                 kernel, bw, gridsize, cut, clip, legend,
--> 653                                 cbar, cbar_ax, cbar_kws, ax, **kwargs)
    654     else:
    655         ax = _univariate_kdeplot(data, shade, vertical, kernel, bw,

/home/raniere/anaconda3/lib/python3.5/site-packages/seaborn-0.8.1-py3.5.egg/seaborn/distributions.py in _bivariate_kdeplot(x, y, filled, fill_lowest, kernel, bw, gridsize, cut, clip, axlabel, cbar, cbar_ax, cbar_kws, ax, **kwargs)
    383         xx, yy, z = _statsmodels_bivariate_kde(x, y, bw, gridsize, cut, clip)
    384     else:
--> 385         xx, yy, z = _scipy_bivariate_kde(x, y, bw, gridsize, cut, clip)
    386 
    387     # Plot the contours

/home/raniere/anaconda3/lib/python3.5/site-packages/seaborn-0.8.1-py3.5.egg/seaborn/distributions.py in _scipy_bivariate_kde(x, y, bw, gridsize, cut, clip)
    442     """Compute a bivariate kde using scipy."""
    443     data = np.c_[x, y]
--> 444     kde = stats.gaussian_kde(data.T)
    445     data_std = data.std(axis=0, ddof=1)
    446     if isinstance(bw, string_types):

/home/raniere/anaconda3/lib/python3.5/site-packages/scipy/stats/kde.py in __init__(self, dataset, bw_method)
    170 
    171         self.d, self.n = self.dataset.shape
--> 172         self.set_bandwidth(bw_method=bw_method)
    173 
    174     def evaluate(self, points):

/home/raniere/anaconda3/lib/python3.5/site-packages/scipy/stats/kde.py in set_bandwidth(self, bw_method)
    497             raise ValueError(msg)
    498 
--> 499         self._compute_covariance()
    500 
    501     def _compute_covariance(self):

/home/raniere/anaconda3/lib/python3.5/site-packages/scipy/stats/kde.py in _compute_covariance(self)
    508             self._data_covariance = atleast_2d(np.cov(self.dataset, rowvar=1,
    509                                                bias=False))
--> 510             self._data_inv_cov = linalg.inv(self._data_covariance)
    511 
    512         self.covariance = self._data_covariance * self.factor**2

/home/raniere/anaconda3/lib/python3.5/site-packages/scipy/linalg/basic.py in inv(a, overwrite_a, check_finite)
    817         inv_a, info = getri(lu, piv, lwork=lwork, overwrite_lu=1)
    818     if info > 0:
--> 819         raise LinAlgError("singular matrix")
    820     if info < 0:
    821         raise ValueError('illegal value in %d-th argument of internal '

LinAlgError: singular matrix

for

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16,6))

ax1 = entrofy.plotting.plot_correlation(df, "age", "age", ax=ax1,
                                        xtype="continuous",
                                        ytype="continuous", cont_type="kde")
ax2 = entrofy.plotting.plot_correlation(df, "age", "age", ax=ax2,
                                        xtype="continuous",
                                        ytype="continuous", cont_type="scatter")

[dhuppenkothen]

Huh, weird. I'm currently on Python 3, too, and I get a few other errors, so I'll chase those up collectively. Will report back ASAP.

[dhuppenkothen]

This seems to maybe a be a problem with the underlying seaborn kdeplot implementation (and possibly the underlying scipy?). Could you make a set of random variables and try sticking them into the seaborn kdeplot function directly?

Just use the same arrays for data1 and data2.

Also, if you let me know what your versions of python, scipy, matplotlib and seaborn are, I can try to reproduce it here, which I currently can't (on python3.6).

[rgaiacs]

Could you make a set of random variables and try sticking them into the seaborn kdeplot function directly?

I can try later.

Also, if you let me know what your versions of python, scipy, matplotlib and seaborn are, I can try to reproduce it here, which I currently can't (on python3.6).

$ python --version
Python 3.5.3 :: Continuum Analytics, Inc.

and

>>> print(numpy.__version__)
1.13.0
>>> print(scipy.__version__)
0.19.0
>>> print(pandas.__version__)
0.20.2
>>> print(matplotlib.__version__)
2.0.2
>>> print(seaborn.__version__)
0.8.1

I can test with Python 3.6 later.

[dhuppenkothen]

I tried with your package versions, but I still can't reproduce the error on my system. :-(

[bmcfee]

Please include the output of numpy.show_config(). I suspect this is going to boil down to discrepancies in the underlying BLAS implementation.

[rgaiacs]

@dhuppenkothen and @bmcfee Thanks very much for the help.

Please include the output of numpy.show_config().

blis_info:
  NOT AVAILABLE
openblas_info:
    library_dirs = ['/usr/local/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    libraries = ['openblas', 'openblas']
blas_mkl_info:
  NOT AVAILABLE
blas_opt_info:
    library_dirs = ['/usr/local/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    libraries = ['openblas', 'openblas']
openblas_lapack_info:
    library_dirs = ['/usr/local/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    libraries = ['openblas', 'openblas']
lapack_mkl_info:
  NOT AVAILABLE
lapack_opt_info:
    library_dirs = ['/usr/local/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    libraries = ['openblas', 'openblas']

[dhuppenkothen]

just FYI: You can change the KDE plot into a scatter plot by setting cont_type="scatter" in plot_correlations, which doesn't fix the error, of course, but if you currently need that function for something, at least it means it won't block you from continuing.

[rgaiacs]

@dhuppenkothen Thanks for all the support. I will see if I have time to contribute over Xmas.