Skip to content

Commit

Permalink
pin pandas to prevent recursive loop triggered by test
Browse files Browse the repository at this point in the history
on running test_network_components.py pandas fails with recursion error
in new and exciting way

error in with pandas 2.1.0, test passes with 2.0.3

change: pin pandas to most recent but one release (2.0.3)

to reproduce failure, install pandas 2.1.0 and run:
python -m pytest tests/integration/test_network_components.py

failing test output (including traceback with recursion annotated)
included below:

[Wed Sep  6 10:48:58 2023]
rule network_components:
    input: results/djibouti-latest_filter-road/nodes.geoparquet, results/djibouti-latest_filter-road/edges.geoparquet
    output: results/djibouti-latest_filter-road/component_population.svg, results/djibouti-latest_filter-road/network_map_by_component.png, results/djibouti-latest_filter-road/components.parquet
    jobid: 0
    reason: Missing output files: results/djibouti-latest_filter-road/components.parquet, results/djibouti-latest_filter-road/component_population.svg, results/djibouti-latest_filter-road/network_map_by_component.png
    wildcards: OUTPUT_DIR=results, DATASET=djibouti-latest, FILTER_SLUG=filter-road
    resources: tmpdir=/tmp

/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pyarrow/pandas_compat.py:373: FutureWarning: is_sparse is deprecated and will be removed in a future version. Check `isinstance(dtype, pd.SparseDtype)` instead.
  if _pandas_api.is_sparse(col):

Traceback (most recent call last):
  File "/tmp/tmpvpafwtp3/workdir/.snakemake/scripts/tmpkp2k6mgv.network_components.py", line 151, in <module>
    map_image = plot_components_map(network)
  File "/tmp/tmpvpafwtp3/workdir/.snakemake/scripts/tmpkp2k6mgv.network_components.py", line 56, in plot_components_map
    agg = cvs.line(edges, geometry="geometry", agg=ds.mean("component_id"))
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/datashader/core.py", line 449, in line
    return bypixel(source, self, glyph, agg, antialias=glyph.antialiased)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/datashader/core.py", line 1247, in bypixel
    source, dshape = _bypixel_sanitise(source, glyph, agg)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/datashader/core.py", line 1287, in _bypixel_sanitise
    source = source[cols_to_keep]
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/frame.py", line 3911, in __getitem__
    data = self._take_with_is_copy(indexer, axis=1)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 4088, in _take_with_is_copy
    result = self.take(indices=indices, axis=axis)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 4073, in take
    return self._constructor_from_mgr(new_data, axes=new_data.axes).__finalize__(
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/frame.py", line 649, in _constructor_from_mgr
    return self._constructor(df, copy=False)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geodataframe.py", line 11, in __new__
    return GeoDataFrame(*args, **kwargs)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geodataframe.py", line 31, in __init__
    self[col] = GeoSeries(self[col])
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geoseries.py", line 35, in __init__
    super().__init__(data, index=index, name=name, **kwargs)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 471, in __init__
    data = data.reindex(index, copy=copy)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 4982, in reindex
    return super().reindex(
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 5514, in reindex
    return self.copy(deep=copy)
*** Recursive loop ***
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 6685, in copy
    return self._constructor_from_mgr(data, axes=data.axes).__finalize__(
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 589, in _constructor_from_mgr
    return self._constructor(ser, copy=False)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geoseries.py", line 12, in __new__
    return series_cls(data, *args, **kwargs)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/spatialpandas/geoseries.py", line 35, in __init__
    super().__init__(data, index=index, name=name, **kwargs)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 471, in __init__
    data = data.reindex(index, copy=copy)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/series.py", line 4982, in reindex
    return super().reindex(
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 5514, in reindex
    return self.copy(deep=copy)
  File "/home/runner/micromamba-root/envs/open-gira/lib/python3.10/site-packages/pandas/core/generic.py", line 6685, in copy
    return self._constructor_from_mgr(data, axes=data.axes).__finalize__(
*** Recursive loop ***
  • Loading branch information
thomas-fred committed Sep 6, 2023
1 parent 6fdf149 commit d99d09b
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ dependencies:
- openpyxl # Excel file format
- osmium-tool==1.14.0 # openstreetmap extracts
- pathos # multiprocessing
# test_network_components.py fails with 2.1.0, see commit msg
- pandas==2.0.3 # tabular data
- pyarrow # parquet file format, Arrow data exchange
- aws-sdk-cpp=1.8.186
# workaround for a bug in aws-sdk-cpp, which is used by pyarrow
Expand Down

0 comments on commit d99d09b

Please sign in to comment.