xarray - compatibility with latest versions

[johnomotani]

Some change in xarray-2022.9.0 introduced a significant performance regression (see #213 (comment)). Possibly something related to the default option used to combine coordinates/metadata when combining two DataArrays/Datasets.

There also seem to be occasional errors when using the latest xarray versions - possibly some deprecated things that we need to update in xBOUT?

It is not good to not be able to use the latest versions of xarray. It would be really good if someone could work out what these issues are and how to fix them!

#234, #220, #213 are related.

The current workaround is to restrict xarray to versions <2022.9.0 #274.

[ZedThree]

I can rustle up some RSE effort to look at this, but it might not be for a little while.

[johnomotani]

@ZedThree that would be awesome. I don't have time to look myself. It's not super urgent as the last good version (xarray-2022.6.0) is fine for now.

[johnomotani]

Would be interesting to see if the performance regression is fixed by the latest xarray release - pydata/xarray#7824 may help things.

[dschwoerer]

Output of pytest with xarray 2023.7.0 and python 3.12:

+ /usr/bin/pytest xbout --long
============================= test session starts ==============================
platform linux -- Python 3.12.0b4, pytest-7.3.2, pluggy-1.0.0
rootdir: /builddir/build/BUILD/xbout-0.3.5
collected 796 items
xbout/calc/tests/test_turbulence.py .....                                [  0%]
xbout/tests/test_against_collect.py ........                             [  1%]
xbout/tests/test_animate.py .......................s...............      [  6%]
xbout/tests/test_boutdataarray.py ...................................... [ 11%]
..................                                                       [ 13%]
xbout/tests/test_boutdataset.py ........................................ [ 18%]
..........................ss...................                          [ 24%]
xbout/tests/test_fastoutput.py .                                         [ 24%]
xbout/tests/test_geometries.py ..                                        [ 24%]
xbout/tests/test_grid.py .....                                           [ 25%]
xbout/tests/test_init.py .                                               [ 25%]
xbout/tests/test_load.py ......................................s........ [ 31%]
.s...................................................................... [ 40%]
............                                                             [ 42%]
xbout/tests/test_plot.py ............................................... [ 47%]
........................................................................ [ 57%]
.........................                                                [ 60%]
xbout/tests/test_region.py ............................................. [ 65%]
........................................................................ [ 74%]
........................................................................ [ 83%]
........................................................................ [ 92%]
...........................................                              [ 98%]
xbout/tests/test_utils.py .............                                  [100%]
=============================== warnings summary ===============================
../../../../usr/lib/python3.12/site-packages/pytz/tzinfo.py:27
  /usr/lib/python3.12/site-packages/pytz/tzinfo.py:27: DeprecationWarning: datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.fromtimestamp(timestamp, datetime.UTC).
    _epoch = datetime.utcfromtimestamp(0)
xbout/tests/test_animate.py::TestAnimate::test_animate2D_controls_arg[controls4]
xbout/tests/test_animate.py::TestAnimate::test_animate1D
xbout/tests/test_animate.py::TestAnimate::test_animate1D_controls_arg[controls3]
xbout/tests/test_animate.py::TestAnimate::test_animate1D_controls_arg[controls4]
xbout/tests/test_animate.py::TestAnimate::test_animate1D_controls_arg[controls5]
xbout/tests/test_animate.py::TestAnimate::test_animate_list
xbout/tests/test_animate.py::TestAnimate::test_animate_list_controls_arg[controls3]
xbout/tests/test_animate.py::TestAnimate::test_animate_list_controls_arg[controls4]
xbout/tests/test_boutdataarray.py::TestBoutDataArrayMethods::test_to_dataset
  /usr/lib64/python3.12/site-packages/matplotlib/animation.py:884: UserWarning: Animation was deleted without rendering anything. This is most likely not intended. To prevent deletion, assign the Animation to a variable, e.g. `anim`, that exists until you output the Animation using `plt.show()` or `anim.save()`.
    warnings.warn(
xbout/tests/test_boutdataarray.py: 32 warnings
xbout/tests/test_boutdataset.py: 22 warnings
  /builddir/build/BUILD/xbout-0.3.5/xbout/geometries.py:390: UserWarning: rename 't' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    ds = ds.rename(t=coordinates["t"])
xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_slab
xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_CENTRE]
xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_XLOW]
xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_YLOW]
xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_ZLOW]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_from_restart_to_restart
xbout/tests/test_load.py::TestOpen::test_restarts
  /builddir/build/BUILD/xbout-0.3.5/xbout/load.py:349: UserWarning: No geometry type found, no physical coordinates will be added
    warn("No geometry type found, no physical coordinates will be added")
xbout/tests/test_boutdataset.py::TestSave::test_reload_all[toroidal]
  /builddir/build/BUILD/xbout-0.3.5/xbout/load.py:245: UserWarning: open_boutdataset() called with geometry="toroidal", but we are reloading a Dataset that was saved after being loaded with geometry="toroidal". Applying geometry="toroidal" from the argument.
    warn(
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[None]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes0]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes1]
  /builddir/build/BUILD/xbout-0.3.5/xbout/utils.py:376: UserWarning: rename 't' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    ds = ds.rename({ds.metadata[key]: d})
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[None]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes0]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_from_restart_to_restart
  /builddir/build/BUILD/xbout-0.3.5/xbout/utils.py:376: UserWarning: rename 'x' to 'x' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    ds = ds.rename({ds.metadata[key]: d})
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[None]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes0]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_from_restart_to_restart
  /builddir/build/BUILD/xbout-0.3.5/xbout/utils.py:376: UserWarning: rename 'y' to 'y' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    ds = ds.rename({ds.metadata[key]: d})
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[None]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes0]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_from_restart_to_restart
  /builddir/build/BUILD/xbout-0.3.5/xbout/utils.py:376: UserWarning: rename 'z' to 'z' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    ds = ds.rename({ds.metadata[key]: d})
xbout/tests/test_load.py::TestOpen::test_single_file
  /builddir/build/BUILD/xbout-0.3.5/xbout/tests/test_load.py:838: UserWarning: rename 't_array' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    expected = expected.set_coords(["t_array", "dx", "dy", "dz"]).rename(
xbout/tests/test_load.py::TestOpen::test_squashed_file
  /builddir/build/BUILD/xbout-0.3.5/xbout/tests/test_load.py:864: UserWarning: rename 't_array' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    expected = expected.set_coords(["t_array", "dx", "dy", "dz"]).rename(
xbout/tests/test_load.py::TestOpen::test_combine_along_x
  /builddir/build/BUILD/xbout-0.3.5/xbout/tests/test_load.py:978: UserWarning: rename 't_array' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    expected = expected.set_coords(["t_array", "dx", "dy", "dz"]).rename(
xbout/tests/test_load.py::TestOpen::test_combine_along_y
  /builddir/build/BUILD/xbout-0.3.5/xbout/tests/test_load.py:1012: UserWarning: rename 't_array' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    expected = expected.set_coords(["t_array", "dx", "dy", "dz"]).rename(
xbout/tests/test_load.py::TestOpen::test_combine_along_xy[lengths0-False-False]
xbout/tests/test_load.py::TestOpen::test_combine_along_xy[lengths0-True-False]
xbout/tests/test_load.py::TestOpen::test_combine_along_xy[lengths0-True-True]
xbout/tests/test_load.py::TestOpen::test_combine_along_xy[lengths1-False-False]
xbout/tests/test_load.py::TestOpen::test_combine_along_xy[lengths1-True-False]
xbout/tests/test_load.py::TestOpen::test_combine_along_xy[lengths1-True-True]
  /builddir/build/BUILD/xbout-0.3.5/xbout/tests/test_load.py:1076: UserWarning: rename 't_array' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    expected = expected.set_coords(["t_array", "dx", "dy", "dz"]).rename(
xbout/tests/test_load.py::TestOpen::test_salpha
xbout/tests/test_load.py::TestOpen::test_salpha
  /builddir/build/BUILD/xbout-0.3.5/xbout/geometries.py:483: UserWarning: rename 'x' to 'r' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    ds = ds.rename(x="r")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========== 791 passed, 5 skipped, 111 warnings in 18885.95s (5:14:45) =========

Good thing is no errors. And there is also some potential to improve performance ;-)

[dschwoerer]

https://src.fedoraproject.org/rpms/python-xbout/blob/rawhide/f/python-xbout.spec#_13

This was done with #278 #283 #291