CDS moving to CDS-beta

[jannesvaningen]

As per 3rd of September, the old CDS will be shutdown and users are urged to move to CDS-beta.
https://forum.ecmwf.int/t/the-new-climate-data-store-beta-cds-beta-is-now-live/3315
Users should move to cdsapi version 0.7.0
Can you support this in a new version of era5cli? The current version 1.4.2 requires cdsapi == 0.5.1

[BSchilperoort]

Thanks for the heads up. I started making the package compatible with the new CDS, however there are still some issues:

• Only the most basic download functionality is implemented
• If you haven't accepted a specific license yet, the new CDS currently gives you a very unhelpful error
• NetCDF downloading seems to be messed up (at least for our example download from the docs). Grib downloads OK... seems OK actually

It's a bit strange how they have a beta now (which is currently barely working) but are also going to remove the old CDS without a proper transitioning period.

Anyway, WIP pull request is in #166

[jannesvaningen]

Thanks @BSchilperoort! Hope the new beta will work properly soon.

[malininae]

Thanks for your work @BSchilperoort! We use era5cli in our pipeline for the quasi-operational rapid extreme event attribution that runs every day and we'd need to find alternative way in the next week if era5cli won't get updated by the 3rd. Again, appreciating your efforts so much! Let mw know if any sort of assistance needed.

[BSchilperoort]

@malininae I should be able put out a release to support the beta this week. But already a heads up: netCDF files will be formatted differently compared to the old CDS.

For example:

<xarray.Dataset>
Dimensions:     (valid_time: 744, latitude: 30, longitude: 43)
Coordinates:
    number      int64 ...
  * valid_time  (valid_time) datetime64[ns] 2000-01-01 ... 2000-01-31T23:00:00
  * latitude    (latitude) float64 53.6 53.5 53.4 53.3 ... 51.0 50.9 50.8 50.7
  * longitude   (longitude) float64 3.3 3.4 3.5 3.6 3.7 ... 7.1 7.2 7.3 7.4 7.5
    expver      (valid_time) object ...
Data variables:
    t2m         (valid_time, latitude, longitude) float32 ...
Attributes:
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2024-08-27 13:19:40 UTC by era5cli 1.4.2: reanal...

The time coordinate is now called valid_time, there is an additional number coordinate, and the attributes (as well as data variable attributes) are formatted differently.

[malininae]

@malininae I should be able put out a release to support the beta this week. But already a heads up: netCDF files will be formatted differently compared to the old CDS.

For example:

<xarray.Dataset>
Dimensions:     (valid_time: 744, latitude: 30, longitude: 43)
Coordinates:
    number      int64 ...
  * valid_time  (valid_time) datetime64[ns] 2000-01-01 ... 2000-01-31T23:00:00
  * latitude    (latitude) float64 53.6 53.5 53.4 53.3 ... 51.0 50.9 50.8 50.7
  * longitude   (longitude) float64 3.3 3.4 3.5 3.6 3.7 ... 7.1 7.2 7.3 7.4 7.5
    expver      (valid_time) object ...
Data variables:
    t2m         (valid_time, latitude, longitude) float32 ...
Attributes:
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2024-08-27 13:19:40 UTC by era5cli 1.4.2: reanal...

The time coordinate is now called valid_time, there is an additional number coordinate, and the attributes (as well as data variable attributes) are formatted differently.

Thanks so much for the heads up!!! That's very helpful, we'll start adjusting right away!

[BSchilperoort]

The beta-version of era5cli 2.0 is now available. You can install it with:
pip install era5cli==2.0.0b1

or any pre-release (in a new environment) with:
pip install --pre era5cli

Please report it if you find any issues!

[jannesvaningen]

Hey @BSchilperoort, I believe era5cli==2.0.0b2 is not pip installable yet. Can you check? It is indicated that it should be on https://github.com/eWaterCycle/era5cli

[BSchilperoort]

Oh yeah, the GH action broke... https://github.com/eWaterCycle/era5cli/actions/runs/10960177444/job/30434145852

Would have to migrate to a new gh action