Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regridding functionalities (powered by xESMF) #243

Merged
merged 180 commits into from
Nov 23, 2023
Merged
Show file tree
Hide file tree
Changes from 145 commits
Commits
Show all changes
180 commits
Select commit Hold shift + click to select a range
ff698e5
put in basic structure and notes for regrid
ellesmith88 Mar 4, 2021
4869291
change params to strings
ellesmith88 Mar 4, 2021
963ad64
import xarray
ellesmith88 Mar 4, 2021
d8d250c
add halo comment
ellesmith88 Mar 4, 2021
1ae4120
A first version of the Grid class for the regrid-prototype.
sol1105 Mar 18, 2021
ccccdc7
Merge branch 'master' into regrid-main-elle
ellesmith88 Mar 19, 2021
7476750
Start adding tests for Grid class
ellesmith88 Mar 19, 2021
be13855
Updates to clisops.core regrid.py Grid
sol1105 Mar 19, 2021
7bac759
merge most recent update
ellesmith88 Mar 22, 2021
7b49c24
update tests
ellesmith88 Mar 22, 2021
efaf5e0
update tests to include bounds and grid
ellesmith88 Mar 22, 2021
e6cb2f3
Added basic Weight class and regrid function to clisops.core regrid.py
sol1105 Mar 23, 2021
50cb102
Updated Regrid class in ops/regrid.py and added Grid.remove_halo() in…
sol1105 Mar 30, 2021
b4ae320
update dependencies and add tests for Weights
ellesmith88 Mar 30, 2021
5e944fb
Added a comment to Grid.grid_remove_halo() in core/regrid.py
sol1105 Mar 31, 2021
a1136e7
remove bottleneck
ellesmith88 Mar 31, 2021
3d12c73
update tests
ellesmith88 Mar 31, 2021
23a75fb
undo rename
ellesmith88 Mar 31, 2021
56654fe
merge master
sol1105 May 4, 2021
8925d0b
Merged master
agstephens Jun 28, 2021
c0bd806
Updated to get regrid tests working to regular grid
agstephens Jul 1, 2021
054537c
Removed unused variable in test
agstephens Jul 1, 2021
46947e8
Tidied.
agstephens Jul 1, 2021
8d9e55e
Fixed attribute bug in tests.
agstephens Jul 1, 2021
f4f3bbc
run pre-commit
Jul 1, 2021
4f2845e
remove scipy from requirements
Jul 1, 2021
b6fc19e
Add scipy into requirements and environment.yml
Jul 1, 2021
2a57286
require scipy>=1.5.4 to work with python 3.6
Jul 1, 2021
84bfa69
Adjusted tests, code and requirements in relation to xesmf / regridding
sol1105 Jul 9, 2021
4ce0342
Added newlines
agstephens Jul 21, 2021
89e12be
Merge branch 'regrid-main-ag' of https://github.com/roocs/clisops int…
agstephens Jul 21, 2021
7f5505e
Changed method of catching/checking xesmf import so tests and code st…
agstephens Jul 22, 2021
bf0934b
Fixed comparison with None
agstephens Jul 22, 2021
5c6cf38
Flake8 linting fix.
agstephens Jul 22, 2021
0ba7983
black linting fixes
agstephens Jul 22, 2021
3ee5dab
Added xesmf to requirements.
agstephens Jul 22, 2021
bbda7b8
Updated xesmf in requirements.txt
agstephens Jul 22, 2021
979faf5
Updated conda github action, using: python -m pytest
agstephens Jul 22, 2021
7cb0414
requirements.txt
agstephens Jul 27, 2021
bd8b228
Updated regrid tests to fix CI failure
agstephens Jul 30, 2021
fb533e5
linted
agstephens Jul 30, 2021
1dc7f72
Marked CI problematic test as xfail
agstephens Aug 31, 2021
ca0c23a
Fixed python version to py3.8 in environment.yml
agstephens Aug 31, 2021
d5615f6
Fixed python version to py3.8 in environment.yml
agstephens Aug 31, 2021
c21275f
Modified use of `ds.mean()` to overcome dask error:
agstephens Sep 6, 2021
eaad713
Fixed tests
agstephens Sep 6, 2021
b497a89
Moved imports to resolve seg fault in tests
agstephens Sep 6, 2021
8290344
Fixed last average test
agstephens Sep 6, 2021
db946f3
Updated github CI workflow
agstephens Sep 6, 2021
86f9d82
Fixed linting
agstephens Sep 6, 2021
9e24bbd
Merge pull request #179 from roocs/regrid-main-ag
agstephens Sep 6, 2021
fbeba9d
Updates to the regrid functionality
sol1105 Sep 8, 2021
ffa6b58
Updates for the Grid class
sol1105 Oct 19, 2021
76cc8b0
Added rudimentary local remapping weights cache
sol1105 Oct 26, 2021
3e0ef65
Added/updated core/ops regrid tests, core Grid class and regrid funct…
sol1105 Oct 27, 2021
a24b950
Grid method to detect collapsing grid cells
sol1105 Oct 28, 2021
e2875ed
Update of compare_grid and grid_from_instructor
sol1105 Oct 29, 2021
1ee708c
Updated clisops.core.regrid.py
sol1105 Nov 1, 2021
4b4d38e
Added curvilinear bounds computation and tests
sol1105 Nov 2, 2021
7301231
Updated behaviour regarding bounds computation
sol1105 Nov 4, 2021
bdf8663
Added test for core regrid weights_cache_init/flush
sol1105 Nov 5, 2021
9cf80dc
weights cache: added source and tracking_id to collected weights.json…
sol1105 Nov 5, 2021
da073ed
Fixes for tests and weights cache.
sol1105 Nov 5, 2021
7681c67
Added basic support of irregular grids in core Grid. Applied a few fi…
sol1105 Nov 5, 2021
d0368d2
Update requirements.txt
sol1105 Nov 8, 2021
0beefe8
Added remapping support for irregular grids (nearest_s2d with xesmf l…
sol1105 Nov 8, 2021
06b937c
core regrid: Transferring now also data_vars attrs when keep_attrs is…
sol1105 Nov 10, 2021
6fd09fe
Update requirements.txt
sol1105 Nov 10, 2021
ee134e1
Update requirements.txt
sol1105 Nov 10, 2021
efea84a
Update requirements.txt
sol1105 Nov 10, 2021
ff6e87b
merge master
sol1105 Jan 21, 2022
1d4f5a9
Updated requirements/env
sol1105 Jan 21, 2022
a2a0292
Skipping core/test_regrid/test_cache_init_and_flush if xESMF not inst…
sol1105 Jan 21, 2022
6e3e587
Added regrid notebook and environment.yml for binder
sol1105 Jan 21, 2022
f4562ed
resolve conflicts
ellesmith88 Feb 1, 2022
384b36b
Updates to the weight cache
sol1105 Feb 1, 2022
7c3a5b7
Merge remote-tracking branch 'origin/regrid-main' into regrid-main-ma…
Feb 2, 2022
36f5bbb
Update .pre-commit-config.yaml
sol1105 Feb 3, 2022
7089342
Fix linting
sol1105 Feb 3, 2022
393eef9
Fix keep_attrs for regrid and add test, update notebook
sol1105 Feb 3, 2022
8e14b95
Update to core and ops regrid
sol1105 Feb 3, 2022
01a89ab
Added tests for grid halo and lock mechanism, fixed lock mechanism
sol1105 Feb 4, 2022
9bf2e82
Fix tests
sol1105 Feb 4, 2022
d8d2d12
Update ops regrid test keep_attrs, fix core regrid keep attrs, contin…
sol1105 Feb 8, 2022
f41f84b
Updates to Weights class and weights cache
sol1105 Feb 10, 2022
9a5e95c
Added test for Grid.to_netcdf(), minor adjustments in core regrid.py …
sol1105 Feb 16, 2022
3a77cca
Updated Grid.to_netcdf and adjusted relevant tests
sol1105 Feb 17, 2022
2e231df
Updates to Grid class
sol1105 Feb 22, 2022
5e69bde
Updates to Grid class
sol1105 Feb 25, 2022
afc4caf
merge master
sol1105 Jul 4, 2022
7b83753
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 4, 2022
2aad0e4
Edit import order core/__init__.py
sol1105 Jul 4, 2022
b57083e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 4, 2022
6058eb9
Updating environment.yml - adding numpy requirements of numba, exempt…
sol1105 Jul 5, 2022
6798264
Fix typo in environment.yml
sol1105 Jul 5, 2022
1d7afa8
merge master
sol1105 Jul 7, 2022
8804df8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 7, 2022
cb5333c
Merge pull request #236 from roocs/regrid-main-merge-master
sol1105 Jul 8, 2022
4e0bc35
Merge branch 'regrid-main' into regrid-main-martin to resolve conflicts.
sol1105 Jul 11, 2022
4c0211c
Restructured title levels in regrid.ipynb
sol1105 Jul 11, 2022
1da97d1
Fix indentation in core/regrid docstrings.
sol1105 Jul 11, 2022
f1bb5fd
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 11, 2022
27a3de9
Preliminary changes to HISTORY.rst, README.rst. Exempting regrid.ipyn…
sol1105 Jul 11, 2022
856d796
Setting readthedocsbuild to conda/mamba
sol1105 Jul 22, 2022
af44b98
Setting readthedocsbuild to conda/mamba #2
sol1105 Jul 22, 2022
4b115e7
Adjusting readthedocs environment.yml
sol1105 Jul 22, 2022
d5c61b0
Adjusting environment.yml and minor changes to regrid notebook
sol1105 Aug 5, 2022
05e39bc
Adding comment to setup.py
sol1105 Aug 5, 2022
c1a4fc0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 5, 2022
994e37c
Adjusting environment.yml and requirements.txt
sol1105 Aug 5, 2022
4c13830
regrid.ipynb rework captions
sol1105 Aug 8, 2022
d7b6159
regrid.ipynb fix anchors
sol1105 Aug 8, 2022
a0f3e33
regrid.ipynb fix list
sol1105 Aug 8, 2022
98c7006
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 8, 2022
38c0ca0
regrid.ipynb fix list 2nd try
sol1105 Aug 8, 2022
e329b57
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 8, 2022
08f0774
core.regrid.regrid adjusting attrs transfer
sol1105 Aug 8, 2022
d4114c7
Merge pull request #196 from roocs/regrid-main-martin
sol1105 Aug 8, 2022
0311240
Update for redundant _FillValue and coordinates attributes removal (o…
sol1105 Aug 16, 2022
b6ce49c
Adjustments and corrections regarding previous commit.
sol1105 Aug 16, 2022
a6d72af
Setting lower limit for cf-xarray, installing from git main branch un…
sol1105 Aug 17, 2022
15a8c39
Corrected requirements
sol1105 Aug 17, 2022
67e8683
Updating dependencies, fix to regrid.ipynb formatting
sol1105 Aug 17, 2022
6784ecf
Updating dependencies
sol1105 Aug 18, 2022
0624d3c
Pin gdal at 3.4.1 for conda env only.
sol1105 Aug 18, 2022
9de8ba4
Merge branch 'master' into regrid-main
Zeitsperre Aug 18, 2022
999ae99
Merge branch 'master' into regrid-main
Zeitsperre Sep 7, 2022
bf11086
Merge branch 'master' into regrid-main
Zeitsperre Oct 25, 2022
0f79877
Merge branch 'master' into regrid-main
Zeitsperre Nov 29, 2022
8bc8114
ignore does not allow `#` anymore
Zeitsperre Nov 29, 2022
b51dce9
Altered treatment of duplicated cells, added tests, altered requirements
sol1105 Dec 13, 2022
dc54d2a
Merge branch 'master' into regrid-main
sol1105 Dec 13, 2022
4ceb2d5
Updated regrid function, added tests
sol1105 Dec 14, 2022
7f20f35
Merge branch 'master' into regrid-main
Zeitsperre Dec 14, 2022
5fa6b7b
Adjust environment, fix failing tests
sol1105 Dec 16, 2022
3458eaa
Merge branch 'master' into regrid-main
Zeitsperre Dec 19, 2022
1848c02
Merge branch 'master' into regrid-main
Zeitsperre Jan 31, 2023
f1dda7a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 31, 2023
1dbe44d
Merge branch 'master' into regrid-main
Zeitsperre Jan 31, 2023
d7077cf
Merge branch 'master' into regrid-main
sol1105 Feb 13, 2023
b473f54
Latest ESMPy requires newest xesmf version 0.7.0. Fix tests.
sol1105 Feb 13, 2023
f352463
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 13, 2023
4b07cf4
Merge branch 'master' into regrid-main
sol1105 Apr 6, 2023
bc921bb
Adding roocs-grids to requirements.txt
sol1105 Apr 6, 2023
cf58b5b
Pin pandas and xarray. Update binder and docs environment.yml
sol1105 Apr 20, 2023
25d757f
Merge branch 'master' into regrid-main
Zeitsperre Jun 20, 2023
8a82255
Merge branch 'master' into regrid-main
Zeitsperre Aug 23, 2023
7259447
Adapt to new structure, add roocs-grids main branch, flatten tests, u…
Zeitsperre Aug 23, 2023
e70c189
remove library-like import
Zeitsperre Aug 23, 2023
bca6bcd
minor cleanup
Zeitsperre Aug 23, 2023
799af2c
Merge branch 'master' into regrid-main
Zeitsperre Aug 23, 2023
ca6582c
Merge branch 'master' into regrid-main
sol1105 Oct 10, 2023
7a23977
Merge branch 'master' into regrid-main
sol1105 Oct 11, 2023
c00ad6a
Update environment.yml
sol1105 Oct 12, 2023
3855a8d
Update environment.yml
sol1105 Oct 12, 2023
8052060
Update main.yml
sol1105 Oct 13, 2023
4e3f5be
Update pyproject.toml
sol1105 Oct 13, 2023
f506931
Fix regrid.ipynb xesmf import
sol1105 Oct 16, 2023
ca4b5df
Fix regrid.ipynb xesmf import 2
sol1105 Oct 16, 2023
858335f
Updated dependencies related to roocs-grids
sol1105 Oct 20, 2023
96dc684
Merge branch 'master' into regrid-main
sol1105 Oct 25, 2023
7656244
Update dependencies regarding cf-xarray and roocs-grids
sol1105 Oct 25, 2023
4eaf174
Merge branch 'master' into regrid-main
Zeitsperre Nov 2, 2023
447011d
Update requirements
sol1105 Nov 10, 2023
dcadea5
Update requirements
sol1105 Nov 10, 2023
de4d0fe
Update requirements
sol1105 Nov 13, 2023
7035a17
Merge branch 'master' into regrid-main
Zeitsperre Nov 13, 2023
c560e48
Update dependencies (roocs_grids) and HISTORY.rst
sol1105 Nov 20, 2023
da34513
Update dependencies (roocs_grids)
sol1105 Nov 20, 2023
16ca1d3
Update pyproject.toml
sol1105 Nov 20, 2023
7a0d988
docstring cleanup and static typing
Zeitsperre Nov 20, 2023
21f7260
move upstream builds to push on main, cronjob, or workflow_dispatch
Zeitsperre Nov 20, 2023
192ee2e
refactoring fixes
Zeitsperre Nov 20, 2023
80a681d
refactor black to lint, naming fixes
Zeitsperre Nov 20, 2023
5b2ea42
remove self from staticmethods
Zeitsperre Nov 20, 2023
8281f8a
remove deprecated pkg_resources calls, update pre-commit
Zeitsperre Nov 20, 2023
51620a0
simplify flake8 ignores
Zeitsperre Nov 20, 2023
de831fe
update HISTORY.rst
Zeitsperre Nov 20, 2023
98fef4d
Updates to environment.yml and core.regrid
sol1105 Nov 21, 2023
fc7b31c
Update docs/environment.yml to be consistent with environment.yml
sol1105 Nov 21, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ repos:
hooks:
- id: isort
args: ['--profile', 'black']
exclude: clisops/core/__init__.py
#- repo: https://github.com/pycqa/pydocstyle
# rev: 6.1.1
# hooks:
Expand Down
5 changes: 4 additions & 1 deletion .readthedocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,7 @@ python:
build:
os: ubuntu-20.04
tools:
python: "3.8"
python: "mambaforge-4.10"

conda:
environment: docs/environment.yml
7 changes: 7 additions & 0 deletions HISTORY.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
Version History
===============

v0.9.7 (unreleased)
-------------------

New Features
^^^^^^^^^^^^
* ``clisops.ops.regrid``, ``clisops.core.regrid``, ``clisops.core.Weights`` and ``clisops.core.Grid`` added (#TBA). Allowing the remapping of geospatial data on various grids by applying the `xESMF <https://pangeo-xesmf.readthedocs.io/en/latest/>`_ regridder.

v0.9.6 (2023-04-05)
-------------------

Expand Down
18 changes: 18 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,24 @@ The package provides the following operations:
* average
* regrid

Online Demo
-----------

..
todo: Links have to be adjusted to the master or respective branch!

You can try clisops online using Binder (just click on the binder link below),
or view the notebooks on NBViewer.

.. image:: https://mybinder.org/badge_logo.svg
:target: https://mybinder.org/v2/gh/roocs/clisops/master?filepath=notebooks
:alt: Binder Launcher

.. image:: https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg
:target: https://nbviewer.jupyter.org/github/roocs/clisops/tree/master/notebooks/
:alt: NBViewer
:height: 20

Credits
-------

Expand Down
36 changes: 36 additions & 0 deletions binder/environment.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: clisops
channels:
- conda-forge
dependencies:
- python >=3.8
- bottleneck >=1.3.1
- cf_xarray >=0.7.5
- cftime >=1.4.1
- dask >=2.6.0
- gdal >=3.0
- geopandas >=0.11
- loguru >=0.5.3
- netCDF4 >=1.4
- numba # needed for xesmf v0.6.3, see: https://github.com/conda-forge/xesmf-feedstock/pull/24
- numpy >=1.16
- packaging
- pandas >=1.0.3,<2.0
- poppler >=0.67
- pyproj >=3.3.0
- pooch
- requests >=2.0
- roocs-utils >=0.6.4,<0.7
- shapely >=1.9
- sparse >=0.8.0 # needed for xesmf v0.6.3, see: https://github.com/conda-forge/xesmf-feedstock/pull/24
- xarray >=0.21,<2023.3.0
- xesmf >=0.7.1
- jupyterlab
- matplotlib>=3.5.2
- cartopy>=0.20.2
- GitPython
- pip:
- psy-maps
- clisops @ git+https://github.com/roocs/clisops/@regrid-main#egg=clisops
- roocs-grids @ git+https://github.com/roocs/roocs-grids.git@main#egg=roocs-grids
# - cf-xarray @ git+https://github.com/xarray-contrib/cf-xarray/@main#egg=cf-xarray
# - roocs-utils @ git+https://github.com/roocs/roocs-utils.git@master#egg=roocs-utils
5 changes: 3 additions & 2 deletions binder/postBuild
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
pip install matplotlib
pip install .
set -e

git clone https://github.com/roocs/mini-esgf-data ~/.mini-esgf-data
2 changes: 2 additions & 0 deletions clisops/core/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,5 @@
subset_time_by_components,
subset_time_by_values,
)

from .regrid import Grid, Weights, regrid, weights_cache_init, weights_cache_flush
Loading