Update to 13 Sep version

Author: Peter Hitchcock

_images/Tcp.png

@@ -1,6 +1,6 @@
-==================
+==========================
 Creating a custom variable
-==================
+==========================
 
 Suppose one wants to perform an operation on data represented by some pygeode
 variable that cannot be done by existing pygeode variables. One can always

_images/contour.png

@@ -6,10 +6,28 @@ File Input and Output
 Serialized data stored in files on disk (or over the network) can be imported
 into PyGeode with the routines :func:`open` (for single files), :func:`openall`
 (for small numbers of files), and :func:`open_multi` (for large numbers of
-files).  Variables and datasets can be saved to disk with :func:`save`. Several
-formats are supported natively by pygeode, including NetCDF (versions 3 and 4),
-HDF (versions 4 and 5) and grib files, though support for NetCDF and HDF is the
-most complete.
+files).  Variables and datasets can be saved to disk with :func:`save`. 
+
+.. _formats:
+
+Several formats are supported natively by pygeode, including NetCDF (versions 3
+and 4), HDF (versions 4 and 5) and grib files, though support for NetCDF and
+HDF is the most complete. By default the format is detected through the file extension;
+however, many of the methods below accept an optional ``format`` argument which can be used to 
+specify the format explicitly. 
+
+========= ============  ================= =========
+Format    Extension     String Identifier Notes
+========= ============  ================= =========
+NetCDF    .nc           netcdf
+HDF5                    netcdf            Uses the same library as NetCDF
+HDF4      .hdf          hdf4
+GRIB      .grib         grib
+========= ============  ================= =========
+
+Code also exists to read the native binary format used by the Canadian Centre
+for Climate Modeling and Analysis, but this is not distributed with PyGeode by
+default.
 
 .. autofunction:: open
 
@@ -19,6 +37,6 @@ most complete.
 
 .. autofunction:: save
 
-.. currentmodule:: pygeode.format
+.. currentmodule:: pygeode.formats
 
 .. autofunction:: autodetectformat

_images/line.png

@@ -37,7 +37,7 @@ This is the recommended approach for Ubuntu systems, since it will automatically
 
 
 RPM-based Linux distributions (Fedora, CentOS, openSUSE, etc.)
-=============================================
+===============================================================
 
 Pre-built RPM packages are available via the `openSUSE Build Service <https://build.opensuse.org/package/show/home:neishm/python-pygeode>`_
 

_images/log_t1Temp.png

@@ -25,6 +25,8 @@ Plot module
 
 .. module:: pygeode.plot
 
-.. autoclass:: AxesWrapper
+.. class:: AxesWrapper
+
+   A PyGeode wrapper class for a matplotlib figure or subplot
 
 .. autofunction:: grid

_images/math/700d4b0a3714bb317206b353f69b69a7b735694a.png

@@ -0,0 +1,43 @@
+Advanced Variable Operations
+============================
+
+.. currentmodule:: pygeode
+
+Climatological statistics
+.........................
+
+A variety of time averaging operations are possible. In addition to computing
+climatologies, one can compute monthly means, daily means, diurnal means, and
+trends. These should generally work as expected; however, PyGeode also
+implements special mapping rules for time axes so that anomalies from these time
+averages can be easily defined. In general these are based on the 
+
+A common operation is to
+compute climatologies. As described in the first of these tutorials this is
+simply performed as follows
+
+.. ipython::
+  
+  @suppress
+  In [0]: import pygeode as pyg
+
+  In [10]: from pygeode.tutorial import t2
+
+  In [10]: Tc = pyg.climatology(t2.Temp)    # Compute the climatology
+
+Time averaging
+..............
+
+Other Operations
+----------------
+
+EOFs, correlation, and regression analysis
+..........................................
+
+include lag correlation
+
+Integration, Differentiation, Interpolation and Smoothing
+.........................................................
+
+Composites
+..........

_images/math/ed8152d6bd3bfd1d1aca7b55fab8aceaaf7e5801.png

@@ -1,8 +1,69 @@
-Advanced Variable Operations
+============================
+Working with Axes
 ============================
 
 .. currentmodule:: pygeode
 
+Constructing variables in memory
+--------------------------------
+
+It is also often useful to create PyGeode variables directly in memory; for
+instance, to evaluate analytical expressions on a given geophysical grid. A very
+useful set of building blocks for creating variables in this way are axes
+objects. If you already have a variable on the relevant grid (defined from a
+file for instance), it's easy to simply use the existing axes, as has been done
+in many cases in these tutorials. However, most axes objects have simple
+constructors to create them directly.
+
+.. ipython::
+
+  @suppress
+  In [0]: import pygeode as pyg, numpy as np
+  
+  # The simplest, default case
+  In [1]: print pyg.NamedAxis(np.arange(15), 'myaxis')
+
+  # Some simple examples
+  In [1]: lon = pyg.Lon(180)
+  
+  In [1]: print lon
+
+  In [2]: lat = pyg.Lat(92)
+
+  In [2]: print lat
+
+  In [3]: pres = pyg.Pres([1000, 900, 800, 700, 500, 300, 200, 100, 50, 30, 10])
+
+  # Gaussian latitudes (with appropriate weights)
+  In [2]: lat2 = pyg.gausslat(64)
+
+  In [2]: print lat2
+
+Time axes are somewhat more complicated, as you need to specify the calendar,
+the reference date (``startdate``), offsets, and the native unit:
+
+.. ipython::
+
+  In [2]: time = pyg.ModelTime365(values=np.arange(3650), units='days', startdate=dict(year=2000, month=1))
+
+Usually the easiest approach to creating variables in memory is to apply the
+relevant mathematical operations to the axes themselves:
+
+.. ipython::
+
+  In [2]: pyg.sin(2*np.pi*time / 365) * pyg.exp(-(lat2 / 20.)**2) 
+
+  In [2]: pyg.sin(2*np.pi*time / 365) * pyg.exp(-(lat2 / 20.)**2)
+
+However, if you have a numpy array that you want to turn in to a PyGeode
+variable, this can also be done.
+
+.. ipython::
+
+  In [2]: x = np.ones((64, 180), 'd')
+
+  In [3]: print pyg.Var((lat2, lon), name = 'myvar', values=x)
+
 .. _timeaxisops:
 
 Time Axis Operations
@@ -109,8 +170,6 @@ simply performed as follows
 
   In [10]: Tc = pyg.climatology(t2.Temp)    # Compute the climatology
 
-  In [10]: 
-
 
 Time axis utilities
 ...................
@@ -119,21 +178,4 @@ There are a number of other useful utility functions for working with time axes
 in the timeutils module. For instance, it is sometimes convenient to convert
 data on a a standardtime axis 
 
-
-EOFs, correlation and regression analysis
------------------------------------------
-
-include lag correlation
-
-Integration and Differentiation
--------------------------------
-
-Interpolation
--------------
-
-Composites
-----------
-
-Smoothing
----------
-
+lagged variables

_images/sind_t1lat.png

@@ -1,17 +1,21 @@
-Basic Variable Operations
-=========================
+================
+Basic Operations
+================
 
 .. currentmodule:: pygeode
 
 Much of the functionality of PyGeode is found in the operations you can perform
 on Var objects. We'll start with the basics of slicing variables, then move on
-to more complicated operations. One important thing to keep in mind, as
-discussed in :doc:`gettingstarted`, is that these operations all delayed until
+to more complicated operations. An important thing to keep in mind, as
+discussed in :doc:`tut_gettingstarted`, is that these operations all delayed until
 the data is specifically requested. For instance, one can compute an average
 over longitude as follows:
 
 .. ipython::
 
+  @suppress
+  In [6]: import pylab as pyl; pyl.ion();
+
   In [1]: import pygeode as pyg
 
   In [2]: from pygeode.tutorial import t1
@@ -166,30 +170,33 @@ axes are removed in this unlike numpy slicing, degenerate axes are removed.
 That is, ``t1.Temp[0, 1]`` returns a scalar, while ``t1.Temp(i_lat=0, i_lon=1)``
 returns a two dimensional variable.
 
-Arithmetic operations 
----------------------
+Arithmetic operations and broadcasting rules
+--------------------------------------------
 
 Reference: :doc:`ufunc` and :doc:`var.arith`
 
-Arithmetic and mathematical operations are also supported by PyGeode. The
+Arithmetic and other mathematical operations are also supported by PyGeode. The
 simplest are unary operations, which are performed elementwise. Most standard
-mathematical functions (powers, exponentials, trigonometric functions, etc.) are
-supported, and can be found in the main ``pygeode`` module:
+mathematical functions (powers, exponentials, trigonometric functions, etc.)
+are supported, and can be found in the main ``pygeode`` module:
 
 .. ipython::
 
   @suppress
   In [6]: import pylab as pyl; pyl.ion()
 
-  @showfig log_t1Temp.png width=4in
+  @suppress
+  In [6]: import matplotlib as mpl; mpl.rc('figure', figsize=(5, 3))
+
+  @savefig log_t1Temp.png width=4in
   In [8]: pyg.showvar(pyg.log(t1.Temp))     # Plot the natural logarithm of Temp
 
 There are some convenience functions included, for instance, most trig functions
 have a version which takes arguments in degrees rather than radians:
 
 .. ipython::
 
-  @showfig sind_t1lat.png width=4in
+  @savefig sind_t1lat.png width=4in
   In [10]: pyg.showvar(pyg.sind(t1.lat))     # Compute the sine of latitude
 
 In most cases the underlying operation is performed by the numpy equivalent,
@@ -199,13 +206,23 @@ variables, so one should get accustomed to including the ``pyg`` prefix. This
 is also a good reason to import pygeode into its own namespace, rather than
 into the top-level namespace itself.
 
-Standard Python arithmetic operations (``+``, ``-``, ``*``, ``/``, ``**``,
-etc.) are also supported and work as one would expect if all the variables are
-defined on the same axes.  If the variables do not share the same axes, PyGeode
-follows a set of rules for automatically broadcasting them so that the
-operations behave as one might typically desire - it's important to be aware of
-these rules as you can sometimes end up with some unexpected results, so they
-are described here in some detail.
+Standard Python binary arithmetic operations (``+``, ``-``, ``*``, ``/``,
+``**``, etc.) are supported and work as one would expect if all the
+variables are defined on the same axes. Comparison operators (``>``, ``<``, etc.)
+are also supported and produce a boolean variable on the same axes with the result of
+the operation performed elementwise. This can be very useful for masking data, 
+since when cast to scalar values, ``True`` is equal to 1 and ``False`` is equal to 0:
+
+.. ipython::
+
+  @savefig t1mask.png width=4in
+  In [11]: pyg.showvar(2 + 3 * (t1.Temp > 280.))
+
+If the variables do not share the same axes, PyGeode follows a set of rules for
+automatically broadcasting them so that the operations behave as one might
+typically desire - it's important to be aware of these rules as you can
+sometimes end up with some unexpected results, so they are described here in
+some detail.
 
 .. _broadcasting:
 
@@ -343,18 +360,17 @@ be specified; those that are not will be appended in their present order.
 
 .. ipython::
 
-  In[8]: t2.Temp.axes
+  In [8]: t2.Temp.axes
 
-  In[9]: print t2.Temp.transpose('lon', 'lat', 'pres', 'time').axes
+  In [9]: print t2.Temp.transpose('lon', 'lat', 'pres', 'time').axes
 
 :func:`Var.replace_axes()` replaces any or all axes of a variable. The new
 axes must have the same length as those they are replacing.
 
 .. ipython::
 
   # The logPAxis method returns a log-pressure axis with a given scale height
-  In[8]: print t2.Temp.replace_axes(pres=t2.pres.logPAxis(H=7000)).axes
-
+  In [8]: print t2.Temp.replace_axes(pres=t2.pres.logPAxis(H=7000)).axes
 
 :func:`Var.squeeze()` removes degenerate axes (those with one or fewer elements).
 As mentioned above, the default selection behaviour is to always return a
@@ -365,11 +381,17 @@ all degenerate axes. Several other methods of calling are also available:
 
 .. ipython::
 
-  In[8]: print t2.Temp(time = 4, lon = 20).squeeze().axes
+  # Squeeze all degenerate axes
+  In [8]: print t2.Temp(time = 4, lon = 20).squeeze()
+
+  # Squeeze only the time axis (if it is degenerate)
+  In [8]: print t2.Temp(time = 4, lon = 20).squeeze('time')
 
-  In[8]: print t2.Temp(time = 4, lon = 20).squeeze('time').axes
+  # Select a single value then squeeze the time axis
+  In [8]: print t2.Temp.squeeze(time = 4)
 
-  In[8]: print t2.Temp.squeeze(i_time = 3, lon = (20, 40))
+  # Select a single value and sqeeze the time axis using a selection prefix
+  In [8]: print t2.Temp(s_time = 4, lon = (20, 40))
 
 There are also commands to rename variables and their axes
 (:func:`Var.rename()`, :func:`Var.rename_axes()`), for adding axes to a

_images/t1Temp.png

@@ -0,0 +1,9 @@
+======================
+Working with Datasets
+======================
+
+.. currentmodule:: pygeode
+
+selections on whole datasets
+combining datasets
+operating on datasets