Extract_GCprof_station_run-example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extract tracer profiles and compute total columns above a given station from GEOS-Chem outputs : Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a 'Run' version. For more details about what is computed and how it is implemented, see de [development notebook](Profiles_Station_GEOSChem_dev.ipynb).\n",
    "\n",
    "**USAGE**: A good practice to ensure [reproducibility](http://en.wikipedia.org/wiki/Reproducibility) is:\n",
    "\n",
    "1. First, copy this notebook as a new notebook (one notebook per run).\n",
    "2. Rename it by changing the short suffix that clearly identifies the purpose of this run. Change also the suffix in the title.\n",
    "3. Fill the comments section below with a longer description.\n",
    "4. Run cells in the Setup section for pre-processing, creating run directories, writing inputs, etc...\n",
    "5. Run cells in the Run section to set the command and start the process.\n",
    "6. The Monitor section is for checking/controlling the process while it is running. Check the process status as many times as you want.\n",
    "7. Run cells in the Output section to clean temp files, inspect the outputs, etc... when the process has finished.\n",
    "8. Write some comments about the outcomes of this run in the Comments section below.\n",
    "9. Save and close the notebook.\n",
    "\n",
    "**WARNING 1**: **Don't run all cells at once**. Some cells may be executed several times and other may be executed optionally. It is also better to run the notebook cell by cell for carefull inputs/outputs verification.\n",
    "\n",
    "**WARNING 2**: **Don't shutdown the kernel associated with this notebook** (or 'close and halt' or restart the kernel) **until the process has finished**. The process will run in the background and thus will normally not be affected by the kernel shutdown, but it will not be possible to get information from the process anymore.   \n",
    "\n",
    "**NOTE**: It is here possible to take advantage of multiple CPUs and automatically split the process into several, parallel jobs (it uses the IPython parallel system). To activate this, set a value > 1 for the argument ncpu (nengines) of the command line.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write here a more detailled description about the purpose of this run, the inputs used, etc..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outcomes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write here your comments about the process outcomes (how look the outputs, etc...)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Get some infos from this notebook and store it as variables in the kernel (needed for the next step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "var notebook_name = document.getElementById('notebook_name').innerHTML;\n",
       "var notebook_root_dir = document.body.getAttribute('data-project');\n",
       "var notebook_rel_path = document.body.getAttribute('data-notebook-path');\n",
       "\n",
       "var kernel = IPython.notebook.kernel;\n",
       "\n",
       "kernel.execute(\"notebook_name = '\" + notebook_name + \"'\");\n",
       "kernel.execute(\"notebook_root_dir = '\" + notebook_root_dir + \"'\");\n",
       "kernel.execute(\"notebook_rel_path = '\" + notebook_rel_path + \"'\");"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript at 0x7fe690215150>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%javascript\n",
    "\n",
    "var notebook_name = document.getElementById('notebook_name').innerHTML;\n",
    "var notebook_root_dir = document.body.getAttribute('data-project');\n",
    "var notebook_rel_path = document.body.getAttribute('data-notebook-path');\n",
    "\n",
    "var kernel = IPython.notebook.kernel;\n",
    "\n",
    "kernel.execute(\"notebook_name = '\" + notebook_name + \"'\");\n",
    "kernel.execute(\"notebook_root_dir = '\" + notebook_root_dir + \"'\");\n",
    "kernel.execute(\"notebook_rel_path = '\" + notebook_rel_path + \"'\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Set up the working directories.\n",
    "    - Locate the directory of this notebook\n",
    "    - Create a new directory for this run, which will have the name of this notebook and which will be located in \\$HOME/IPYRuns (plus the same relative path from \\$HOME/IPYNotebooks to this notebook).\n",
    "    - 'cd' into this run directory. The process will be run from within this directory. Unless the absolute path is specified explicitly, any file (input, output, log file) will be saved into this directory.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "run directory is: /home/bovy/IPYRuns/Extract_GCprof_station_run-example\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    " \n",
    "notebook_rel_path = os.path.dirname(notebook_rel_path)\n",
    "notebook_rel_path = notebook_rel_path.replace('IPYNotebooks/', '')\n",
    "this_notebook_dir = os.path.join(os.path.expanduser('~'), 'IPYNotebooks',\n",
    "                                 notebook_rel_path)\n",
    "notebook_runs_dir = os.path.join(os.path.expanduser('~'), 'IPYRuns')\n",
    "this_run_dir = os.path.join(notebook_runs_dir, notebook_rel_path, notebook_name)\n",
    "\n",
    "if not os.path.exists(this_run_dir):\n",
    "    os.makedirs(this_run_dir)\n",
    "os.chdir(this_run_dir)\n",
    "\n",
    "print(\"run directory is: \" + this_run_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Edit/Create input file: all the content of the cell below will be written to the file specified at the 1st line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile input.yaml\n",
    "\n",
    "# This is an input file for the script 'extract_gcprof_station.py'\n",
    "# Format of the file is YAML\n",
    "\n",
    "\n",
    "# ------------------\n",
    "# IN-OUT FILES\n",
    "# ------------------\n",
    "\n",
    "# Input main directory\n",
    "#   Should contain GEOS-Chem ouput datafields\n",
    "in_dir: ../../../IPYNotebooks/nb_geoschem/data/ts_example\n",
    "\n",
    "# GEOS-Chem output file(s) (netCDF and/or bpch)\n",
    "#   May be either \n",
    "#   (1) the name of a single file present in `in_dir` \n",
    "#   (2) an absolute path to a single file\n",
    "#   (3) a file-matching pattern using de wildcard character.\n",
    "#   (4) a list of any combination of (1), (2) and (3)\n",
    "#\n",
    "#   Mixing CTM outputs and ND49 outputs (time series) may work \n",
    "#   (though not tested yet), but datafields must not overlap in time.\n",
    "#   All datafields contained in the files must use the same horizontal\n",
    "#   grid (or a subset of this grid)!\n",
    "in_files: 'ts.joch.200401*'\n",
    "\n",
    "# Path to save output files where extracted data will be written\n",
    "#   If '~' is given, output files will be saved in the directory\n",
    "#   from where the script is run\n",
    "out_dir: ~      \n",
    "\n",
    "# Basename of the output files for profiles\n",
    "#   Should not include the file extension\n",
    "#   Any wildcard \"*\" will be replaced by the `station_name` parameter\n",
    "out_profiles_basename: '*_profiles_200401'   \n",
    "\n",
    "# Basename of output file for columns\n",
    "out_columns_basename: '*_columns_200401'     \n",
    "\n",
    "# Format of output files\n",
    "#   One of the following: \"csv\", \"hdf5\", \"xls\", \"xlsx\"\n",
    "#   In addtion, netCDF files will be created (iris cubes).\n",
    "out_format: xlsx                 \n",
    "\n",
    "\n",
    "# ------------------\n",
    "# DATAFIELDS TO LOAD\n",
    "# ------------------\n",
    "\n",
    "# List of tracers/diagnostics for which profiles and columns\n",
    "# will be extracted/computed\n",
    "tracers: [PAN, CO, ACET, C3H8, CH2O, C2H6, NH3]\n",
    "\n",
    "# List of diagnostic categories to load\n",
    "#   Should be \"IJ-AVG-$\" for tracers\n",
    "categories: [IJ-AVG-$]               \n",
    "\n",
    "# Additional fields names to load (format: 'diagnostic_category')\n",
    "#   Must at least include datafields required for columns calculation,\n",
    "#   i.e., 'PSURF_PEDGE-$', 'BXHEIGHT_BXHGHT-$',\n",
    "#   'AIRDEN_TIME-SER' or 'N(AIR)_BXHGHT-$', 'TMPU_DAO-3D-$\n",
    "other_fields: [PSURF_PEDGE-$, BXHEIGHT_BXHGHT-$, AIRDEN_TIME-SER, N(AIR)_BXHGHT-$, TMPU_DAO-3D-$]\n",
    "\n",
    "\n",
    "# ------------------\n",
    "# STATION PARAMETERS\n",
    "# ------------------\n",
    "\n",
    "# Name of the station\n",
    "station_name: JungfrauJoch      \n",
    "\n",
    "# Latitude of the station [degrees_north]\n",
    "station_lat: 46.54806              \n",
    "\n",
    "# Longitude of the station [degress_east]\n",
    "station_lon: 7.98389               \n",
    "\n",
    "# Elevation a.s.l at the station [meters],\n",
    "station_altitude: 3580.            \n",
    "\n",
    "# Path to the file (CF-netCDF) that contains the altitude values\n",
    "# of the vertical grid on which data will be regridded.\n",
    "station_vertical_grid_file: /home/bovy/Grids/NDACC_vertical_Jungfraujoch_39L_2x2.5.nc\n",
    "\n",
    "\n",
    "# ------------------\n",
    "# GRID INFO\n",
    "# ------------------\n",
    "\n",
    "# Grid model name\n",
    "#   All GEOS-Chem ouputs that will be loaded must use this grid.\n",
    "#   See :prop:`pygchem.grid.CTMGrid.models`\n",
    "#   for a list of available lodels\n",
    "grid_model_name: GEOS57_47L          \n",
    "\n",
    "# Grid horizontal resolution (lon, lat) [degrees]\n",
    "#   All GEOS-Chem ouputs must use this resolution\n",
    "grid_model_resolution: [2.5, 2]          \n",
    "\n",
    "# Grid indices (min, max) of the 3D region box of interest\n",
    "#   i: longitude, j: latitude, l: vertical levels\n",
    "#   Must match the extent that was defined for any ND49\n",
    "#   diagnostic output specified in `in_files`.\n",
    "#   Must emcompass the position of the station (see below).\n",
    "#   Used either to define the coordinates of ND49 outputs or to\n",
    "#   extract a subset from the global CTM datafields.\n",
    "iminmax: [76, 77]\n",
    "jminmax: [69, 70]\n",
    "lminmax: [1, 47]\n",
    "\n",
    "\n",
    "# ------------------\n",
    "# TOPOGRAPHY\n",
    "# ------------------\n",
    "\n",
    "# Path to the file of global topography needed for resampling\n",
    "# the tracer profiles on a vertical grid with fixed altitude values.\n",
    "#   The global topography grid must be compatible with the\n",
    "#   GEOS-Chem grid used by the output GEOS-Chem files.\n",
    "global_topography_datafile: /home/bovy/Grids/dem_GEOS57_2x2.5_awm.nc\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Set the command line to be executed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "cmd = \"{executable} {script} input.yaml --loglevel={loglevel} --nengines={ncpu}\"\n",
    "\n",
    "cmd = cmd.format(\n",
    "    # path to executable (same python interpreter than the one used to run the notebook server)\n",
    "    executable=sys.executable,\n",
    "    # path to the script\n",
    "    script=os.path.join(this_notebook_dir, 'run_scripts', 'extract_gcprof_station.py'),\n",
    "    # number of CPU to use\n",
    "    ncpu=4,\n",
    "    # loglevel ('CRITICAL', 'ERROR', 'WARNING', 'INFO' or 'DEBUG')\n",
    "    loglevel='INFO',\n",
    ")\n",
    "\n",
    "print(\"Command to execute: \" + cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Execute the command in a new process in the background (only if no process is already running)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import subprocess\n",
    "import os\n",
    "import sys\n",
    "import shlex\n",
    "\n",
    "# prevent running a new process if a process is already running.\n",
    "try:\n",
    "    if process.poll() is None:\n",
    "        raise RuntimeError('A process is already running')\n",
    "except NameError:\n",
    "    pass\n",
    "\n",
    "# split the command into a sequence\n",
    "cmd = shlex.split(cmd)\n",
    "# comment the line above and use the command string instead of sequence if shell is True\n",
    "# http://stackoverflow.com/questions/16840427/python-on-linux-subprocess-popen-works-weird-with-shell-true\n",
    "\n",
    "with open('process.log', 'w') as log:\n",
    "    process = subprocess.Popen(cmd, shell=False, stdout=log, stderr=log)\n",
    "\n",
    "print(\"New process started. PID: {}\".format(process.pid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Monitor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Check the status of the process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "try:\n",
    "    if process.poll() is None:\n",
    "        print(\"process is running\")\n",
    "        status = 'running'\n",
    "    \n",
    "    elif process.poll() == 0:\n",
    "        print(\"process has terminated succesfully\")\n",
    "        status = 'success'\n",
    "        \n",
    "    else:\n",
    "        sys.stderr.write(\"process has terminated with errors\\n\")\n",
    "        status = 'error'\n",
    "        \n",
    "except NameError:\n",
    "    print(\"no process is running! \"\n",
    "          \"(or connection with the process loosed due \"\n",
    "          \"to a kernel issue or kernel shutdown/restart)\")\n",
    "    status = None\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Display the log file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if status is not None:\n",
    "    %cat process.log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- If using the IPython parallel cluster, display ouputs (stdout, stderr) of all engines as they are printed out (debug)\n",
    "    - open a terminal session in the server\n",
    "    - activate the virtual environment\n",
    "    - while the process is running, run the script `iopubwatcher.py` in the `run_scripts` directory\n",
    "\n",
    "    $ python iopubwatcher.py\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **(WARNING) Use the cell below to terminate the process if needed**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import signal\n",
    "\n",
    "if status == 'running':\n",
    "    process.send_signal(signal.SIGINT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **(WARNING) Use the cell below to stop the IPython cluster if needed**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import getpass\n",
    "import sys\n",
    "\n",
    "user = getpass.getuser()\n",
    "ipy_profile = 'nb_{}'.format(user)\n",
    "ipcluster_exe = os.path.join(\n",
    "    os.path.dirname(sys.executable),\n",
    "    'ipcluster'\n",
    ")\n",
    "\n",
    "os.system('{} stop --profile={}'.format(ipcluster_exe, ipy_profile))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Clean / Inspect outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Check if output files were created by listing them (with size, last modified date...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!ls -all -h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- (NOT WORKING YET REMOTELY) generate links for output files (only file saved in the current working directory) so that they can be downloaded from here. TODO: add a tornado handler to the ipython notebook server, which verifies user authentication and redirects (X-Accel_Redirect) to an internal nginx location? see nginx/tornado example [here](https://groups.google.com/forum/#!topic/python-tornado/sgadmx8Hd_s) Use NotebookApp.webapp_settings ? see http://www.tornadoweb.org/en/stable/web.html?highlight=web.application#tornado.web.Application for a list of available settings. See [handlers used by IPython](http://ipython.org/ipython-doc/dev/api/generated/IPython.html.base.handlers.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import glob\n",
    "from IPython.display import display, FileLink\n",
    "\n",
    "for fout in glob.glob('*'):\n",
    "    if os.path.isdir(fout):\n",
    "        continue\n",
    "    fout_link = FileLink(fout)\n",
    "    display(fout_link)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PyGChem (py27)",
   "language": "python",
   "name": "pygchem_py27"
  },
  "language_info": {
   "name": "python",
   "version": "2.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}