Merge pull request #36 from andrewdelman/v4r4

V4r4

Author: ifenty

ECCO-ACCESS/Downloading_ECCO_datasets_from_PODAAC/README.md

@@ -0,0 +1,13 @@
+# Instructions for Downloading ECCO Datasets hosted on PODAAC
+
+## Using Python 3 & Jupyter Notebooks
+
+[Python 3 Jupyter Notebook (ipynb)](Tutorial_Python3_Jupyter_Notebook_Downloading_ECCO_Datasets_from_PODAAC.ipynb)
+
+[Python 3 Jupyter Notebook (pdf)](Tutorial_Python3_Jupyter_Notebook_Downloading_ECCO_Datasets_from_PODAAC.pdf)
+
+## Using Command Line *_wget_* and NASA Earthdata
+
+[_wget_ command line instructions (markdown)](Tutorial_wget_Command_Line_HTTPS_Downloading_ECCO_Datasets_from_PODAAC.md)
+
+[_wget_ command line instructions (pdf)](Tutorial_wget_Command_Line_HTTPS_Downloading_ECCO_Datasets_from_PODAAC.md)

ECCO-ACCESS/Downloading_ECCO_datasets_from_PODAAC/Tutorial_Python3_Jupyter_Notebook_Downloading_ECCO_Datasets_from_PODAAC.ipynb

@@ -0,0 +1,94 @@
+# Tutorial: Using Command Line _wget_ to Download ECCO Datasets from PO.DAAC
+
+Version 1.0 2021-06-25
+
+By Jack McNelis and Ian Fenty
+
+## Step 1: Create an account with NASA Earthdata
+
+Please visit https://urs.earthdata.nasa.gov/home to make an account and be ready with your EOSDIS login and password. 
+
+**The Earthdata Login provides a single mechanism for user registration and profile management for all EOSDIS system components (DAACs, Tools, Services). Your Earthdata login also helps the EOSDIS program better understand the usage of EOSDIS services to improve user experience through customization of tools and improvement of services. EOSDIS data are openly available to all and free of charge except where governed by international agreements.**
+
+> **Note!**  _some Earthdata password characters may cause problems depending on your system_. To be safe, do not use any of the following characters in your password: backslash (\\), space, hash (#), quotes (single or double), or greater than (>).  Set/change your Earthdata password here: [https://urs.earthdata.nasa.gov/change_password](https://urs.earthdata.nasa.gov/change_password)
+
+## Step 2: Set up your ```netrc``` and ```urs_cookies``` files
+
+1. Create a file called ```.netrc``` in your home directory (linux, Mac):
+```
+/home/<username>/.netrc
+```
+or ```_netrc``` (Windows):
+```
+C:\Users\<username>\_netrc
+```
+
+The ```netrc``` file must have the following structure and must include your Earthdata account login name and password:
+
+```
+machine urs.earthdata.nasa.gov
+    login <your username>
+    password <your password>
+```
+
+2. Set permissions on your ```netrc``` file to be readable only by the current user.  If not, you will receive the error "netrc access too permissive." 
+
+```shell
+$ chmod 0600 ~/.netrc
+```
+
+3. Create an ```urs_cookies``` "cookie" file. This will be used to persist sessions across individual _wget_ calls, making it more efficient.
+
+```
+  > cd ~
+  > touch .urs_cookies
+```
+
+## Step 3: Prepare a list of granules (files) to download
+
+Now the only step that remains is to get a list of URLs to pass to *_wget_* for downloading. There's a lot of ways to do this -- even more so for ECCO datasets data because the files/datasets follow well-structured naming conventions -- but we will rely on Earthdata Search to do this from the browser for the sake of simplicity.
+
+**1. Find the collection/dataset of interest in Earthdata Search.**
+
+Start from this [complete list of ECCO collections](https://search.earthdata.nasa.gov/portal/podaac-cloud/search?fpj=ECCO) in Earthdata Search, and refine the results until you see your dataset of interest.
+
+In this example we will download all of the granules for the collection [ECCO Version 4 Release 4 (V4r4) monthly sea surface height on a 0.5 degree lat-lon grid](https://search.earthdata.nasa.gov/portal/podaac-cloud/search/granules?p=C1990404799-POCLOUD).
+
+
+**2. Choose your collection, then click the green *Download All* button on the next page.**
+
+Click the big green button identified by the red arrow/box in the screenshot below.
+
+<img src="https://raw.githubusercontent.com/ECCO-GROUP/ECCO-ACCESS/master/PODAAC/Images/edsc1.png" width="70%" />
+
+That will add all the granules in the collection to your "shopping cart" and then redirect you straight there and present you with the available options for customizing the data prior to download.  In this example we ignore the other download options those because they are in active development. 
+
+**3. Click *Download Data* to get your list of download urls (bottom-left, another green button)**
+
+The *Download Data* button takes you to one final page that provides the list of urls from which to download the files matching your search parameters and any customization options that you selected in the steps that followed. This page will be retained in your User History in case you need to return to it later. 
+
+<img src="https://raw.githubusercontent.com/ECCO-GROUP/ECCO-ACCESS/master/PODAAC/Images/edsc2.png" width="70%" />
+<center><i>The screenshot above shows the download customization interface (i.e. "shopping cart")</i></center>
+
+There are several ways that you could get the list of urls into a text file that's accessible from Jupyter or your local shell. I simply clicked the **Save** button in my browser and downloaded them as a text file. (You could also copy them into a new notebook cell and write them to a file like we did with the ```netrc``` file above.). 
+
+<img src="https://raw.githubusercontent.com/ECCO-GROUP/ECCO-ACCESS/master/PODAAC/Images/edsc3.png" width="70%" />
+
+> **Note!**  _As of 2021-06-25 the option "Download Script" also produces a functioning script for batch downloading._)
+
+## Step 4: Download files in a batch with GNU *_wget_*
+
+I find *_wget_* options to be convenient and easy to remember. There are only a handful that I use with any regularity.
+
+The most important _wget_ option for our purpose is set by the `-i` argument, which takes a path to the input text file containing our download urls. Another nice feature of _wget_ is the ability to continue downloads where you left of during a previously-interrupted download session. That option is turned on by passing the `-c` argument.
+
+Go ahead and create a *data/* directory to keep the downloaded files, and then start the downloads into that location by including the `-P` argument:
+
+```sh
+$mkdir -p data
+
+$wget --no-verbose \
+     --no-clobber \
+     --continue \
+     -i 5237392644-download.txt -P data/
+```

ECCO-ACCESS/Downloading_ECCO_datasets_from_PODAAC/Tutorial_Python3_Jupyter_Notebook_Downloading_ECCO_Datasets_from_PODAAC.pdf

@@ -0,0 +1,220 @@
+def ecco_podaac_download(ShortName,StartDate,EndDate,download_root_dir=None,n_workers=6,force_redownload=False):
+    """
+    This routine downloads ECCO datasets from PO.DAAC. It is adapted from the Jupyter notebooks created by Jack McNelis and Ian Fenty (https://github.com/ECCO-GROUP/ECCO-ACCESS/blob/master/PODAAC/Downloading_ECCO_datasets_from_PODAAC/README.md) and modified by Andrew Delman (https://ecco-v4-python-tutorial.readthedocs.io).
+    
+    Parameters
+    ----------
+    ShortName: the ShortName of the dataset (can be identified from https://search.earthdata.nasa.gov/search?fpj=ECCO, selecting the "i" information button and the ShortName will appear in a gray box in the upper-left corner)
+    
+    StartDate: the start of the time range to be downloaded, expressed in the format "YYYY-MM-DD"
+    
+    EndDate: the end of the time range to be downloaded, expressed in the format "YYYY-MM-DD"
+    
+    download_root_dir: path of the parent directory to download ECCO files
+    
+    n_workers: number of workers to use in concurrent downloads
+    
+    force_redownload: if True, existing files will be redownloaded and replaced; if False, existing files will not be replaced
+    """
+    
+    
+    ## Initalize Python libraries
+    import numpy as np
+    import pandas as pd
+    import requests
+    import shutil
+    import time as time
+    
+    # for concurrent simulatenous downloads
+    from concurrent.futures import ThreadPoolExecutor
+    from getpass import getpass
+    from http.cookiejar import CookieJar
+    from io import StringIO
+    from itertools import repeat
+    from pathlib import Path
+    from platform import system
+    from netrc import netrc
+    from os.path import basename, isfile, isdir, join
+    # progress bar
+    from tqdm import tqdm
+    # library to download files
+    from urllib import request
+    
+    # if no download directory specified, set directory under user's home directory
+    if download_root_dir==None:
+        import sys
+        from os.path import expanduser
+        user_home_dir = expanduser('~')
+        download_root_dir = Path(user_home_dir + '/Downloads/ECCO_V4r4_PODAAC')
+    else:
+        download_root_dir = Path(download_root_dir)
+    
+    # Predict the path of the netrc file depending on os/platform type.
+    _netrc = join(expanduser('~'), "_netrc" if system()=="Windows" else ".netrc")
+    
+    ## Define Helper Subroutines
+    
+    ### Helper subroutine to log into NASA EarthData
+    
+    # not pretty but it works
+    def setup_earthdata_login_auth(url: str='urs.earthdata.nasa.gov'):
+        # look for the netrc file and use the login/password
+        try:
+            username, _, password = netrc(file=_netrc).authenticators(url)
+    
+        # if the file is not found, prompt the user for the login/password
+        except (FileNotFoundError, TypeError):
+            print('Please provide Earthdata Login credentials for access.')
+            username, password = input('Username: '), getpass('Password: ')
+        
+        manager = request.HTTPPasswordMgrWithDefaultRealm()
+        manager.add_password(None, url, username, password)
+        auth = request.HTTPBasicAuthHandler(manager)
+        jar = CookieJar()
+        processor = request.HTTPCookieProcessor(jar)
+        opener = request.build_opener(auth, processor)
+        request.install_opener(opener)
+    
+    ### Helper subroutines to make the API calls to search CMR and parse response
+    def set_params(params: dict):
+        params.update({'scroll': "true", 'page_size': 2000})
+        return {par: val for par, val in params.items() if val is not None}
+    
+    def get_results(params: dict, headers: dict=None):
+        response = requests.get(url="https://cmr.earthdata.nasa.gov/search/granules.csv", 
+                                params=set_params(params),
+                                headers=headers)
+        return response, response.headers
+    
+    
+    def get_granules(params: dict):
+        response, headers = get_results(params=params)
+        scroll = headers['CMR-Scroll-Id']
+        hits = int(headers['CMR-Hits'])
+        if hits==0:
+            raise Exception("No granules matched your input parameters.")
+        df = pd.read_csv(StringIO(response.text)) 
+        while hits > df.index.size:
+            response, _ = get_results(params=params, headers={'CMR-Scroll-Id': scroll})
+            data = pd.read_csv(StringIO(response.text))
+            df = pd.concat([df, data])
+        return df
+    
+    ### Helper subroutine to gracefully download single files and avoids re-downloading if file already exists.
+    # To force redownload of the file, pass **True** to the boolean argument *force* (default **False**)\n,
+    def download_file(url: str, output_dir: str, force: bool=False):
+        """url (str): the HTTPS url from which the file will download
+        output_dir (str): the local path into which the file will download
+        force (bool): download even if the file exists locally already
+        """
+        if not isdir(output_dir):
+            raise Exception(f"Output directory doesnt exist! ({output_dir})")
+        
+        target_file = join(output_dir, basename(url))
+        
+        # if the file has already been downloaded, skip    
+        if isfile(target_file) and force is False:
+            print(f'\n{basename(url)} already exists, and force=False, not re-downloading')
+            return 0
+        
+        with requests.get(url) as r:
+            if not r.status_code // 100 == 2: 
+                raise Exception(r.text)
+                return 0
+            else:
+                with open(target_file, 'wb') as f:
+                    total_size_in_bytes= int(r.headers.get('content-length', 0))
+                    for chunk in r.iter_content(chunk_size=1024):
+                        if chunk:
+                            f.write(chunk)
+    
+                    return total_size_in_bytes
+    
+    ### Helper subroutine to download all urls in the list `dls`
+    def download_files_concurrently(dls, download_dir, force=False):
+        start_time = time.time()
+    
+        # use 3 threads for concurrent downloads
+        with ThreadPoolExecutor(max_workers=max_workers) as executor:
+    
+            # tqdm makes a cool progress bar
+            results = list(tqdm(executor.map(download_file, dls, repeat(download_dir), repeat(force)), total=len(dls)))
+        
+            # add up the total downloaded file sizes
+            total_download_size_in_bytes = np.sum(np.array(results))
+            # calculate total time spent in the download
+            total_time = time.time() - start_time
+    
+            print('\n=====================================')
+            print(f'total downloaded: {np.round(total_download_size_in_bytes/1e6,2)} Mb')
+            print(f'avg download speed: {np.round(total_download_size_in_bytes/1e6/total_time,2)} Mb/s')
+    
+    # define root directory for downloaded NetCDF files
+    download_root_dir = Path(user_home_dir + '/Downloads/ECCO_V4r4_PODAAC')
+    
+    # define the directory where the downloaded files will be saved
+    download_dir = download_root_dir / ShortName
+    
+    # create the download directory
+    download_dir.mkdir(exist_ok = True, parents=True)
+    
+    print(f'created download directory {download_dir}')
+    
+    ## Log into Earthdata using your username and password
+    
+    # actually log in with this command:
+    setup_earthdata_login_auth()
+    
+    # Query the NASA Common Metadata Repository to find the URL of every granule associated with the desired ECCO Dataset and date range of interest.
+    
+    # create a Python dictionary with our search criteria:  `ShortName` and `temporal`
+    input_search_params = {'ShortName': ShortName,
+                           'temporal': ",".join([StartDate, EndDate])}
+    
+    print(input_search_params)
+    
+    ### Query CMR for the desired ECCO Dataset
+    
+    # grans means 'granules', PO.DAAC's term for individual files in a dataset
+    grans = get_granules(input_search_params)
+    
+    # grans.info()
+    
+    num_grans = len( grans['Granule UR'] )
+    print (f'\nTotal number of matching granules: {num_grans}')
+    
+    
+    ## Download the granules
+    
+    # convert the rows of the 'Online Access URLS' column to a Python list
+    dls = grans['Online Access URLs'].tolist()
+    
+    try:
+        # Attempt concurrent downloads, but if error arises switch to sequential downloads
+        ### Method 1: Concurrent downloads
+        
+        # Define the maximum number of concurrent downloads (benefits typically taper off above 5-6)
+        max_workers = 6
+        
+        # Force redownload (or not) depending on value of force_redownload
+        download_files_concurrently(dls, download_dir, force_redownload)
+        
+    except:
+        ### Method 2: Sequential Downloads
+        
+        # Download each URL sequentially in a for loop.
+        total_download_size_in_bytes = 0
+        start_time = time.time()
+        
+        # loop through all urls in dls
+        for u in dls:
+            u_name = u.split('/')[-1]
+            print(f'downloading {u_name}')
+            total_download_size_in_bytes += download_file(url=u, output_dir=download_dir, force=force_redownload)
+        
+        # calculate total time spent in the download
+        total_time = time.time() - start_time
+        
+        print('\n=====================================')
+        print(f'total downloaded: {np.round(total_download_size_in_bytes/1e6,2)} Mb')
+        print(f'avg download speed: {np.round(total_download_size_in_bytes/1e6/total_time,2)} Mb/s')

ECCO-ACCESS/Downloading_ECCO_datasets_from_PODAAC/Tutorial_wget_Command_Line_HTTPS_Downloading_ECCO_Datasets_from_PODAAC.md

@@ -4,7 +4,7 @@
 
 This repository contains a Python tutorial for using the [ECCO Central Production version 4](https://ecco.jpl.nasa.gov/) ocean and sea-ice state estimate.  Directories within the repository include the ([tutorial documentation](http://ecco-v4-python-tutorial.readthedocs.io/)) and individiual lessons from the tutorial as Juypter notebooks ([model settings ([Tutorials_as_Jupyter_Notebooks/](Tutorials_as_Jupyter_Notebooks/) and [Tutorials_as_Python_Files/](Tutorials_as_Python_Files/)).  
 
-The tutorials were written for ECCO version 4 release 3 but should be applicable to any ECCO v4 solution. If user support is needed, please contact <[email protected]>.
+The tutorials were written for ECCO version 4 release 3 but should be applicable to any ECCO v4 solution, and are currently being updated for version 4 release 4. If user support is needed, please contact <[email protected]>.
 
 [Estimating the Circulation and Climate of the Ocean]: http://ecco.jpl.nasa.gov, http://ecco-group.org/