Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pin pandas to prevent recursive loop triggered by test #149

Merged
merged 1 commit into from
Sep 6, 2023

Conversation

thomas-fred
Copy link
Contributor

@thomas-fred thomas-fred commented Sep 6, 2023

on running test_network_components.py pandas fails with recursion error in new and exciting way

error in with pandas 2.1.0, test passes with 2.0.3

change: pin pandas to most recent but one release (2.0.3)

to reproduce failure, install pandas 2.1.0 and run: python -m pytest tests/integration/test_network_components.py

failing test output (including traceback with recursion annotated) included below:

[Wed Sep  6 10:48:58 2023]
rule network_components:
    input: results/djibouti-latest_filter-road/nodes.geoparquet, results/djibouti-latest_filter-road/edges.geoparquet
    output: results/djibouti-latest_filter-road/component_population.svg, results/djibouti-latest_filter-road/network_map_by_component.png, results/djibouti-latest_filter-road/components.parquet
    jobid: 0
    reason: Missing output files: results/djibouti-latest_filter-road/components.parquet, results/djibouti-latest_filter-road/component_population.svg, results/djibouti-latest_filter-road/network_map_by_component.png
    wildcards: OUTPUT_DIR=results, DATASET=djibouti-latest, FILTER_SLUG=filter-road
    resources: tmpdir=/tmp

/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pyarrow/pandas_compat.py:373: FutureWarning: is_sparse is deprecated and will be removed in a future version. Check `isinstance(dtype, pd.SparseDtype)` instead.
  if _pandas_api.is_sparse(col):

Traceback (most recent call last):
  File "/tmp/tmpvpafwtp3/workdir/.snakemake/scripts/tmpkp2k6mgv.network_components.py", line 151, in <module>
    map_image = plot_components_map(network)
  File "/tmp/tmpvpafwtp3/workdir/.snakemake/scripts/tmpkp2k6mgv.network_components.py", line 56, in plot_components_map
    agg = cvs.line(edges, geometry="geometry", agg=ds.mean("component_id"))
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/datashader/core.py", line 449, in line
    return bypixel(source, self, glyph, agg, antialias=glyph.antialiased)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/datashader/core.py", line 1247, in bypixel
    source, dshape = _bypixel_sanitise(source, glyph, agg)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/datashader/core.py", line 1287, in _bypixel_sanitise
    source = source[cols_to_keep]
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/frame.py", line 3911, in __getitem__
    data = self._take_with_is_copy(indexer, axis=1)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 4088, in _take_with_is_copy
    result = self.take(indices=indices, axis=axis)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 4073, in take
    return self._constructor_from_mgr(new_data, axes=new_data.axes).__finalize__(
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/frame.py", line 649, in _constructor_from_mgr
    return self._constructor(df, copy=False)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geodataframe.py", line 11, in __new__
    return GeoDataFrame(*args, **kwargs)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geodataframe.py", line 31, in __init__
    self[col] = GeoSeries(self[col])
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geoseries.py", line 35, in __init__
    super().__init__(data, index=index, name=name, **kwargs)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 471, in __init__
    data = data.reindex(index, copy=copy)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 4982, in reindex
    return super().reindex(
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 5514, in reindex
    return self.copy(deep=copy)
*** Recursive loop ***
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 6685, in copy
    return self._constructor_from_mgr(data, axes=data.axes).__finalize__(
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 589, in _constructor_from_mgr
    return self._constructor(ser, copy=False)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geoseries.py", line 12, in __new__
    return series_cls(data, *args, **kwargs)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geoseries.py", line 35, in __init__
    super().__init__(data, index=index, name=name, **kwargs)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 471, in __init__
    data = data.reindex(index, copy=copy)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 4982, in reindex
    return super().reindex(
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 5514, in reindex
    return self.copy(deep=copy)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 6685, in copy
    return self._constructor_from_mgr(data, axes=data.axes).__finalize__(
*** Recursive loop ***

on running test_network_components.py pandas fails with recursion error
in new and exciting way

error in with pandas 2.1.0, test passes with 2.0.3

change: pin pandas to most recent but one release (2.0.3)

to reproduce failure, install pandas 2.1.0 and run:
python -m pytest tests/integration/test_network_components.py

failing test output (including traceback with recursion annotated)
included below:

[Wed Sep  6 10:48:58 2023]
rule network_components:
    input: results/djibouti-latest_filter-road/nodes.geoparquet, results/djibouti-latest_filter-road/edges.geoparquet
    output: results/djibouti-latest_filter-road/component_population.svg, results/djibouti-latest_filter-road/network_map_by_component.png, results/djibouti-latest_filter-road/components.parquet
    jobid: 0
    reason: Missing output files: results/djibouti-latest_filter-road/components.parquet, results/djibouti-latest_filter-road/component_population.svg, results/djibouti-latest_filter-road/network_map_by_component.png
    wildcards: OUTPUT_DIR=results, DATASET=djibouti-latest, FILTER_SLUG=filter-road
    resources: tmpdir=/tmp

/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pyarrow/pandas_compat.py:373: FutureWarning: is_sparse is deprecated and will be removed in a future version. Check `isinstance(dtype, pd.SparseDtype)` instead.
  if _pandas_api.is_sparse(col):

Traceback (most recent call last):
  File "/tmp/tmpvpafwtp3/workdir/.snakemake/scripts/tmpkp2k6mgv.network_components.py", line 151, in <module>
    map_image = plot_components_map(network)
  File "/tmp/tmpvpafwtp3/workdir/.snakemake/scripts/tmpkp2k6mgv.network_components.py", line 56, in plot_components_map
    agg = cvs.line(edges, geometry="geometry", agg=ds.mean("component_id"))
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/datashader/core.py", line 449, in line
    return bypixel(source, self, glyph, agg, antialias=glyph.antialiased)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/datashader/core.py", line 1247, in bypixel
    source, dshape = _bypixel_sanitise(source, glyph, agg)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/datashader/core.py", line 1287, in _bypixel_sanitise
    source = source[cols_to_keep]
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/frame.py", line 3911, in __getitem__
    data = self._take_with_is_copy(indexer, axis=1)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 4088, in _take_with_is_copy
    result = self.take(indices=indices, axis=axis)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 4073, in take
    return self._constructor_from_mgr(new_data, axes=new_data.axes).__finalize__(
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/frame.py", line 649, in _constructor_from_mgr
    return self._constructor(df, copy=False)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geodataframe.py", line 11, in __new__
    return GeoDataFrame(*args, **kwargs)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geodataframe.py", line 31, in __init__
    self[col] = GeoSeries(self[col])
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geoseries.py", line 35, in __init__
    super().__init__(data, index=index, name=name, **kwargs)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 471, in __init__
    data = data.reindex(index, copy=copy)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 4982, in reindex
    return super().reindex(
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 5514, in reindex
    return self.copy(deep=copy)
*** Recursive loop ***
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 6685, in copy
    return self._constructor_from_mgr(data, axes=data.axes).__finalize__(
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 589, in _constructor_from_mgr
    return self._constructor(ser, copy=False)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geoseries.py", line 12, in __new__
    return series_cls(data, *args, **kwargs)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geoseries.py", line 35, in __init__
    super().__init__(data, index=index, name=name, **kwargs)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 471, in __init__
    data = data.reindex(index, copy=copy)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 4982, in reindex
    return super().reindex(
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 5514, in reindex
    return self.copy(deep=copy)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 6685, in copy
    return self._constructor_from_mgr(data, axes=data.axes).__finalize__(
*** Recursive loop ***
@thomas-fred thomas-fred merged commit a1d6190 into main Sep 6, 2023
1 check failed
@thomas-fred thomas-fred deleted the bugfix/pandas_recursion_error branch September 6, 2023 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant