Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add functionality for download of subregion #70

Merged
merged 23 commits into from
Mar 4, 2021
Merged

add functionality for download of subregion #70

merged 23 commits into from
Mar 4, 2021

Conversation

bvreede
Copy link
Contributor

@bvreede bvreede commented Feb 18, 2021

This draft pull request is a sequel to #66, and preemptively includes #67, so should not be considered until after the latter is merged.
The work was started by @aytacpacal and aims to allow users to download data from a subregion of the globe by adding coordinates to the command.

Requested changes from #66 have been included, and tests have been added. Some tests in test_cli.py have been set to xfail as I am unsure what error message to use; help would be appreciated!

Closes #44.

@bvreede
Copy link
Contributor Author

bvreede commented Feb 18, 2021

Things to fix and add:

  • Make documentation consistent
  • Include coordinates in output file name
  • Fix cli tests currently set to xfail
  • Fix Travis build
  • Fix code coverage

@codecov
Copy link

codecov bot commented Feb 19, 2021

Codecov Report

Merging #70 (aec813a) into master (ff1bb4d) will increase coverage by 0.11%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #70      +/-   ##
==========================================
+ Coverage   98.20%   98.31%   +0.11%     
==========================================
  Files           7        7              
  Lines         335      357      +22     
==========================================
+ Hits          329      351      +22     
  Misses          6        6              
Impacted Files Coverage Δ
era5cli/cli.py 96.87% <100.00%> (+0.03%) ⬆️
era5cli/fetch.py 98.80% <100.00%> (+0.17%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ff1bb4d...aec813a. Read the comment docs.

era5cli/fetch.py Outdated Show resolved Hide resolved
bvreede and others added 2 commits February 19, 2021 10:06
…want-python-argparse-to-throw-an-exception-rather-than-usage
Co-authored-by: Peter Kalverla <[email protected]>
@bvreede
Copy link
Contributor Author

bvreede commented Feb 19, 2021

Some things to pay attention to when reviewing:

  • Is using SystemExit as an error type appropriate in testing (see test_cli.py)
  • Because the area coordinates are floats, they always have decimal points. This looks ugly in particular in file names, especially when the original input was integers. Should this be addressed?

@bvreede bvreede marked this pull request as ready for review February 19, 2021 17:18
Copy link
Collaborator

@Peter9192 Peter9192 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @bvreede ,
Great progress! I have a couple more suggestions, also in response to your questions. Keep up the good work :-)

era5cli/fetch.py Outdated
and -180 <= W <= 180
and -180 <= E <= 180
and N > S
and W != E
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm surprised as I'd also expect W < E, but indeed the online CDS checker just accepts that. Does that lead to unexpected filenames?

And maybe we can issue a warning in this case (instead of an error)? Something like: "Be aware, this will probably work but we recommend changing it to avoid confusion"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assumed that W-E was different, because they are the western and eastern coordinates of a grid that wraps around the world. You could e.g. have a grid from 120W to -120E to get the pacific ocean, or from -120W to 120E to get most continents.
The filenames are probably confusing if you would read them as degrees on the map, which is what W-E-N-S may do, rather than numeric coordinates of the globe, with the W/E/N/S indicating the borders of the map you have chosen.

I can't even express this properly here, let alone in a filename :/ really not sure what is the best approach...

Copy link
Contributor Author

@bvreede bvreede Mar 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps instead of W/E/N/S we could work with Left/Right/Top/Down, as that is the information we need. [edit: or use the ymax xmin ymin xmax terminology, which is already in the cli, and remove all references to WENS as this is simply confusing]

era5cli/fetch.py Show resolved Hide resolved
era5cli/fetch.py Outdated Show resolved Hide resolved
tests/test_cli.py Outdated Show resolved Hide resolved
tests/test_cli.py Outdated Show resolved Hide resolved
tests/test_cli.py Outdated Show resolved Hide resolved
tests/test_cli.py Show resolved Hide resolved
era5cli/fetch.py Outdated Show resolved Hide resolved
tests/test_fetch.py Outdated Show resolved Hide resolved
tests/test_fetch.py Outdated Show resolved Hide resolved
@bvreede bvreede requested a review from Peter9192 March 2, 2021 11:55
era5cli/fetch.py Outdated
Comment on lines 170 to 171
coords = [str(int(c)) for c in self.area]
fname += '_[' + ']['.join(coords) + ']'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beware that round(1.5) = 2 while int(1.5)=1.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about something like:

lon = lambda x: f"{x}E" if x>0 else f"{abs(x)}W"
lat = lambda y:  f"{y}N" if y>0 else f"{abs(x)}S"
xmin, xmax = -10, 10
ymin, ymax = 45, 55
name = f"{lon(xmin)}-{lon(xmax)}_{lat(ymin)}-{lat(ymax)}"
--> '10W-10E_45N-55N'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks muchly! Incorporated.
Questions (to pay attention to in the new commit):

  • I added that it does not add NS/EW notation if the coordinate is 0. Is this correct? [added a test for this as well; line 223 in test_fetch.py gives an example].
  • Are they in the right order like this?

I liked having a direct relationship between the user input and the file name; that is the only thing I'm sad to lose...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely done 👍 . I think I'd prefer 0N and 0E over just plain zeros.

@bvreede bvreede requested a review from Peter9192 March 4, 2021 06:24
@Peter9192 Peter9192 merged commit 4d4fce6 into master Mar 4, 2021
@Peter9192 Peter9192 deleted the subregion branch March 4, 2021 10:05
@bvreede bvreede mentioned this pull request Mar 4, 2021
@Peter9192
Copy link
Collaborator

Thanks @bvreede and @aytacpacal for adding this awesome feature!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for regional subselection
3 participants