Skip to content

Using GTG

Jump to bottom
Cameron Goodale edited this page Jan 20, 2016 · 21 revisions

Using GTG

Install all the Python Modules listed in the what is needed section of the Project Home page.

Download the latest release from the Release Page

Grab the release you want in either tar.gz or zip format and uncompress them on your local machine.

Note the following files

config.json: contains the variables for the GTG. The config variables are broken up into two types, simple variables you'd change to adjust the temporal and spatial domains and the DIRS to input files and the advanced variables that will change how the algorithm detects MCCs.

Simple Variables (you want to change these before running the code):

LATMAX
LATMIN
LONMIN
LONMAX
DIRS.CEoriDirName
DIRS.mainDirStr
DIRS.TRMMdirName
startDateTime
endDateTime

baselineDataDir.zip: contains sample data.

code/

  • UserInterface.ipynb: notebook to use the GTG

  • iomethods.py: contains the I/O functions needed for the GTG

  • metrics.py: contains the core (atmospheric) metrics performed by the GTG

  • mccSearch.py contains all the functions needed

  • mccSearchUI.py contains a command line wizard for running the program

  • mainProg.py contains a sample of the general workflow of the order the modules should be called.

  • mainProgTemplate.py the general workflow of the order the modules should be called in order to complete an evaluation.

  • plotting.py: contains the core plotting functions for the GTG

  • utils.py: contains misc. functions needed by the GTG

  • variables.py: contains the classes for to set up the variables needed by the GTG

  • Scripts for developers *

  • gtg_lint_file: the config file for the GTG linting standards.

  • code/baselineTiming.py: script for timing individual components of the GTG

  • code/checkMERGBaseline.py: script for checking the differences between outputs from various GTG runs

How to run the GTG

New Users (Juypter Notebook)

You will need to start up the Juypter Notebook server once you have all the python dependencies installed. From the main root directory of the unzipped file run the following commands:

Unzip the sample data to help you get started

grab-tag-graph$ unzip baselineDataDir.zip
grab-tag-graph$ ls datadir
MERG	TRMM

** With the sample data unzipped you want to start up the Notebook**

grab-tag-graph$ cd code
grab-tag-graph/code$ ipython notebook
[I 17:58:45.452 NotebookApp] Using MathJax from CDN: https://cdn.mathjax.org/mathjax/latest/MathJax.js
[I 17:58:45.491 NotebookApp] Serving notebooks from local directory: /Users/cgoodale/grab-tag-graph/code
[I 17:58:45.491 NotebookApp] 0 active kernels 
[I 17:58:45.491 NotebookApp] The IPython Notebook is running at: http://localhost:8888/
[I 17:58:45.491 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Now your default browser should take you to a list of available notebooks on your local machine. Select the UserInterface.ipynb from the list and it will launch that notebook in a new browser tab.

Setup the DIR paths to point at your local machine paths

Advanced Users (Python Command Line)

Run mccSearchUI.py

python mccSearchUI.py

mccSearchUI.py is the use of GTG for the weather application of finding mesoscale convective complexes (MCCs) in When you run the command line, mccSearchUI.py, there are a number of inputs that will be required. There are the main inputs you will have to supply (as well as their variable names in the code - especially useful for editing mainProg.py or tracing the code:

  • mainDirStr : this is the working directory where you wish all the output files –images, textfiles, clipped netCDF files- to be store
  • There is an option to preprocess data. The program
  • CEoriDirName : this is the directory where the original MERG data in netCDF format is stored
  • TRMMdirName : this is the directory where the original TRMM data in netCDF format is stored
  • Start date and time in the format yyyymmddhr
  • End date and time in the format yyyymmddhr

The following assumptions are made:

  • input data are in one folder. For MERG data this is CEoriDirName and for the TRMM data this is TRMMdirName in mainProg.py. These directories cannot be the same.
  • THERE IS NO FILE CHECKING. So please ensure ALL your files are there in netCDF format.

GTG workflow in weather application mccSearchUI.py

The general workflow of the program. The dashed lines indicate optional paths.

The findCloudElements (find_cloud_elements) can be completed in a parallel fashion leveraging Python's multiprocessing libraries on the vectorization branch of the code. This design was contributed by Gabriel Mel de Fontenay, who was a final year CS student at USC during his capstone project. The design is illustrated below.The parallelized implementation of find_cloud_elements that leverages the Python multiprocessing library.

Successful run

Once everything went well, the directory you indicated where outputs should be stored when prompted in the CL will be generated, and four folders should appear in it.

  • image/ stores all images generated from plots.
  • textFiles/ stores all the text files generated during the run, e.g. cloudElementsUserFile.txt that contains information about each cloud element identified
  • MERGnetcdfCEs/ contains the infrared data in masked netCDF files that have been generated for each cloud element identified
  • TRMMnetcdfCEs/ contains the precipitation data in clipped netCDF files that have been generated for each cloud element identified.
Clone this wiki locally