updated with new data access

Author: stewartchrisecmwf

glofas-bangladesh-floods.ipynb

@@ -31,10 +31,10 @@
     "The tutorial is structured as follows:\n",
     "\n",
     "0. (Install and import the required python packages)\n",
-    "1. Download the data of interest from the C3S Climate Data Store (CDS);\n",
-    "1. Explore the data with xarray, apply mask using the auxilary data, and produce demonstrative map plots;\n",
-    "1. Compute and plot time-series of the June 2022 river discharge against the 10-year period;\n",
-    "1. Compute and display a map of the number of days in June 2022 that the river discharge exceeded the\n",
+    "1. Download the data of interest from the Early warning Data Store (EWDS);;\n",
+    "2. Explore the data with xarray, apply mask using the auxilary data, and produce demonstrative map plots;\n",
+    "3. Compute and plot time-series of the June 2022 river discharge against the 10-year period;\n",
+    "4. Compute and display a map of the number of days in June 2022 that the river discharge exceeded the\n",
     "  90th percentile of the 10-year period.\n",
     "\n",
     "The steps and methods here serve as demonstrative examples and could be applied to other events and datasets,\n",
@@ -84,11 +84,11 @@
    "source": [
     "## Setup your environment\n",
     "\n",
-    "Before we begin we must prepare our environment. This includes installing the Application Programming Interface (API) of the CDS, and importing the various python libraries that we will need.\n",
+    "Before we begin we must prepare our environment. This includes installing the Application Programming Interface (API) of the EWDS, and importing the various python libraries that we will need.\n",
     "\n",
-    "### Install CDS API\n",
+    "### Install EWDS API\n",
     "\n",
-    "To install the CDS API, run the following command. We use an exclamation mark to pass the command to the shell (not to the Python interpreter)."
+    "To install the EWDS API, run the following command. We use an exclamation mark to pass the command to the shell (not to the Python interpreter)."
    ]
   },
   {
@@ -97,7 +97,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!pip -q install cdsapi\n",
+    "!pip -q install cdsapi #'cdsapi>=0.7.2'\n",
     "\n",
     "# Other requirements:\n",
     "!pip -q install cfgrib ecmwflibs\n",
@@ -112,14 +112,25 @@
    "source": [
     "### Import libraries\n",
     "\n",
-    "We will be working with data in NetCDF format. To best handle this data we will use libraries for working with multidimensional arrays, in particular Xarray. We will also need libraries for plotting and viewing data, in this case we will use Matplotlib and Cartopy."
+    "We will be working with data in Grib and Grib format. To best handle this data we will use libraries for working with multidimensional arrays, in particular Xarray. We will also need libraries for plotting and viewing data, in this case we will use Matplotlib and Cartopy."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "ename": "",
+     "evalue": "",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[1;31mRunning cells with 'Python 3.12.3' requires the ipykernel package.\n",
+      "\u001b[1;31mRun the following command to install 'ipykernel' into the Python environment. \n",
+      "\u001b[1;31mCommand: '/bin/python3.12 -m pip install ipykernel -U --user --force-reinstall'"
+     ]
+    }
+   ],
    "source": [
     "import os\n",
     "\n",
@@ -148,11 +159,11 @@
    "metadata": {},
    "source": [
     "## Search and download the data\n",
-    "### Enter your CDS API key\n",
+    "### Enter your EWDS API key\n",
     "\n",
-    "We will request data from the Climate Data Store (CDS) programmatically with the help of the CDS API.\n",
+    "We will request data from the Early Warning Data Store (EWDS) programmatically with the help of the EWDS API.\n",
     "\n",
-    "If you have not set up your CDS API credentials with a `~/.cdsapirc` file, it is possible to provide the credentials when initialising the `cdsapi.Client`. To do this we must define the two variables needed to establish a connection: `URL` and `KEY` . The `URL` for the cds api is common and you do not need to modify that field. The `KEY` is string of characters made up of your your personal User ID and CDS API key. To obtain these, first register or login to the CDS (https://cds-beta.climate.copernicus.eu/), then visit https://cds-beta.climate.copernicus.eu/how-to-api and copy the string of characters listed after \"key:\". Replace the `#########` below with this string.\n",
+    "If you have not set up your EWDS API credentials with a `~/.cdsapirc` file, it is possible to provide the credentials when initialising the `cdsapi.Client`. To do this we must define the two variables needed to establish a connection: `URL` and `KEY` . The `URL` for the EWDS api is common and you do not need to modify that field. The `KEY` is string of characters made up of your your personal User ID and EWDS API key. To obtain these, first register or login to the EWDS (https://ewds.climate.copernicus.eu/), then visit https://ewds.climate.copernicus.eu/how-to-api and copy the string of characters listed after \"key:\". Replace the `#########` below with this string.\n",
     "\n",
     "NOTE: If you have set up your cdsapi key using a ~/.cdsapirc you do not need to add your key below."
    ]
@@ -166,7 +177,7 @@
     "if os.path.isfile(\"~/.cdsapirc\"):\n",
     "    cdsapi_kwargs = {}\n",
     "else:\n",
-    "    URL = 'https://cds-beta.climate.copernicus.eu/api'\n",
+    "    URL = 'https://ewds.climate.copernicus.eu/api'\n",
     "    KEY = '##################################'\n",
     "    cdsapi_kwargs = {\n",
     "        'url': URL,\n",
@@ -200,7 +211,7 @@
     "### Find the river discharge data we want\n",
     "\n",
     "The historical GLOFAS data can be found in \n",
-    "[CDS catalogue entry](https://cds.climate.copernicus.eu/cdsapp#!/dataset/cems-glofas-historical).\n",
+    "[EWDS catalogue entry](https://ewds.climate.copernicus.eu/datasets).\n",
     "\n",
     "We are going to look at the Bangladesh flood in June 2022. For our analysis we are going to compare June 2022\n",
     "against all the June data from the 10-year reference period: 2012 to 2021. This period has been chosen for\n",
@@ -220,16 +231,17 @@
     "- **Product type**: Consolidated\n",
     "- **Variable**: River discharge in the last 24 hours'\n",
     "- **Year**: 2012 to 2022\n",
-    "- **Month**: June\n",
+    "- **Month**: 06\n",
     "- **Day**: All days\n",
-    "- **Area**: [30, 85, 20, 95,] = A bounding box for Bangladesh\n",
-    "- **Format**: Grib\n",
+    "- **Area**: [30, 85, 20, 95] = A bounding box for Bangladesh\n",
+    "- **data_format**: Grib\n",
+    "- **download_format**: unarchived\n",
     "\n",
     "At the end of the download form, select **\"Show API request\"**. This will reveal a block of code, which you can simply copy and paste into a cell of your Jupyter Notebook (see cells below).\n",
     "\n",
     "#### Download data\n",
     "\n",
-    "... having copied the API request into the cell below, running this will retrieve and download the data you requested into your local directory. However, before you run the cell below, the **terms and conditions** of this particular dataset need to have been accepted in the CDS. The option to view and accept these conditions is given at the end of the download form, just above the **\"Show API request\"** option."
+    "... having copied the API request into the cell below, running this will retrieve and download the data you requested into your local directory. However, before you run the cell below, the **terms and conditions** of this particular dataset need to have been accepted in the EWDS. The option to view and accept these conditions is given at the end of the download form, just above the **\"Show API request\"** option."
    ]
   },
   {
@@ -240,23 +252,21 @@
    "source": [
     "download_file = f\"{DATADIR}/glofas-2012_2022.grib\"\n",
     "if not os.path.isfile(download_file):\n",
-    "    c = cdsapi.Client(\n",
-    "        **cdsapi_kwargs\n",
-    "    )\n",
-    "    c.retrieve(\n",
-    "        'cems-glofas-historical',\n",
-    "        {\n",
-    "            'system_version': 'version_4_0',\n",
-    "            'hydrological_model': 'lisflood',\n",
-    "            'product_type': 'consolidated',\n",
-    "            'variable': 'river_discharge_in_the_last_24_hours',\n",
-    "            'hyear': [f\"{year}\" for year in range(2012, 2023)],\n",
-    "            'hmonth': 'june',\n",
-    "            'hday': [f\"{day:02d}\" for day in range(1,31)],\n",
-    "            'format': 'grib',\n",
-    "            'area': [30, 85, 20, 95,],\n",
-    "        },\n",
-    "    ).download(download_file)"
+    "    c = cdsapi.Client()\n",
+    "    DATASET=\"cems-glofas-historical\"\n",
+    "    REQUEST= {\n",
+    "            \"system_version\": [\"version_4_0\"],\n",
+    "            \"hydrological_model\": [\"lisflood\"],\n",
+    "            \"product_type\": [\"consolidated\"],\n",
+    "            \"variable\": [\"river_discharge_in_the_last_24_hours\"],\n",
+    "            \"hyear\": [f\"{year}\" for year in range(2012, 2023)],\n",
+    "            \"hmonth\": [\"06\"],\n",
+    "            \"hday\": [f\"{day:02d}\" for day in range(1,31)],\n",
+    "            \"data_format\": \"grib2\",\n",
+    "            \"download_format\": \"unarchived\",\n",
+    "            \"area\": [30, 85, 20, 95]\n",
+    "             }\n",
+    "    c.retrieve(DATASET,REQUEST).download(download_file)"
    ]
   },
   {
@@ -716,7 +726,7 @@
     }
    ],
    "source": [
-    "glofas_data = xr.open_dataset(download_file)\n",
+    "glofas_data = xr.open_dataset(download_file, engine=\"cfgrib\")\n",
     "glofas_data"
    ]
   },
@@ -1273,7 +1283,7 @@
     "upstream_area_file = os.path.join(DATADIR, upstream_area_fname)\n",
     "# If we have not already downloaded the file, we download it\n",
     "if not os.path.isfile(upstream_area_file):\n",
-    "   u_version=2 # file version\n",
+    "   u_version=4 # file version\n",
     "   upstream_data_url = (\n",
     "      f\"https://confluence.ecmwf.int/download/attachments/242067380/{upstream_area_file}?\"\n",
     "      f\"version={u_version}&modificationDate=1668604690076&api=v2&download=true\"\n",
@@ -2066,7 +2076,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "Python 3.11.8-01",
    "language": "python",
    "name": "python3"
   },
@@ -2080,7 +2090,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.9"
+   "version": "3.11.8"
   },
   "vscode": {
    "interpreter": {