10-Getting-at-Data.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function, division\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting at Data\n",
    "\n",
    "Both SunPy and Astropy have utilities for downloading data for your delectation. They are simple and easy to use, however increasing levels of computing will allow a great deal of personalisation and selection. Let us begin with SunPy\n",
    "\n",
    "## Aquiring Data with SunPy\n",
    "\n",
    "### VS0\n",
    "This is the main interface which SunPy to search for and find solar data. VSO stands for the Virtual Solar Observatory, a service which presents a homogenous interface to heterogenous data set.\n",
    "\n",
    "So what do we need?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sunpy.net import vso\n",
    "client = vso.VSOClient()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is your client object. This is effectively the intermediary between yourself and the treasure chest of solar data available. You query VSO, then VSO querys all data providers which fit the limiations you imposed during your search command. The VSO client also handles the particulars of dowloading the data onto your machiene. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making a query\n",
    "\n",
    "Lets kick off with an example, lets ask the veteran of solar imaging, SoHO for some EIS data, between the dates of between January 1st and 2nd, 2001,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "qr = client.query(vso.attrs.Time('2001/1/1', '2001/1/2'), \n",
    "                  vso.attrs.Instrument('eit'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`qr` is a Python list of response onjects, each one a record that the VSO has found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "122\n",
      "   Start Time [1]       End Time [1]    Source Instrument   Type  \n",
      "------------------- ------------------- ------ ---------- --------\n",
      "2001-01-01 01:00:14 2001-01-01 01:00:21   SOHO        EIT FULLDISK\n",
      "2001-01-01 07:00:14 2001-01-01 07:00:21   SOHO        EIT FULLDISK\n",
      "2001-01-01 13:00:15 2001-01-01 13:00:22   SOHO        EIT FULLDISK\n",
      "2001-01-01 19:00:14 2001-01-01 19:00:21   SOHO        EIT FULLDISK\n",
      "2001-01-01 00:00:42 2001-01-01 00:00:54   SOHO        EIT FULLDISK\n",
      "2001-01-01 00:12:10 2001-01-01 00:12:23   SOHO        EIT FULLDISK\n",
      "2001-01-01 00:24:10 2001-01-01 00:24:22   SOHO        EIT FULLDISK\n",
      "2001-01-01 00:36:10 2001-01-01 00:36:22   SOHO        EIT FULLDISK\n",
      "2001-01-01 00:48:10 2001-01-01 00:48:22   SOHO        EIT FULLDISK\n",
      "2001-01-01 01:13:45 2001-01-01 01:13:57   SOHO        EIT FULLDISK\n",
      "                ...                 ...    ...        ...      ...\n",
      "2001-01-01 23:24:10 2001-01-01 23:24:22   SOHO        EIT FULLDISK\n",
      "2001-01-01 23:36:10 2001-01-01 23:36:22   SOHO        EIT FULLDISK\n",
      "2001-01-01 23:48:10 2001-01-01 23:48:22   SOHO        EIT FULLDISK\n",
      "2001-01-01 01:06:03 2001-01-01 01:08:05   SOHO        EIT FULLDISK\n",
      "2001-01-01 07:06:04 2001-01-01 07:08:06   SOHO        EIT FULLDISK\n",
      "2001-01-01 13:06:06 2001-01-01 13:08:08   SOHO        EIT FULLDISK\n",
      "2001-01-01 19:06:04 2001-01-01 19:08:06   SOHO        EIT FULLDISK\n",
      "2001-01-01 01:19:39 2001-01-01 01:20:11   SOHO        EIT FULLDISK\n",
      "2001-01-01 07:19:40 2001-01-01 07:20:12   SOHO        EIT FULLDISK\n",
      "2001-01-01 13:19:44 2001-01-01 13:20:16   SOHO        EIT FULLDISK\n",
      "2001-01-01 19:19:41 2001-01-01 19:20:13   SOHO        EIT FULLDISK\n",
      "Length = 122 rows\n"
     ]
    }
   ],
   "source": [
    "print(len(qr))\n",
    "print(qr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Break it down\n",
    "\n",
    "So we can pass many attributes to the VSO, in this case we started with time\n",
    "\n",
    "<pre><code> vso.attrs.Time('2001/1/1','2001/1/2')</code></pre>\n",
    "\n",
    "Start and end times for the query as strings, any date/time function can be understood by SunPy's parse_time function e.g. the datetime onjects we will look at later. Next we give it the instrument we want:\n",
    "\n",
    "<pre><code> vso.attrs.Instrument('eit') </code></pre>\n",
    "\n",
    "You don't have to pass it an instrument, the client will find all available missions in the parameter you've defined if you like. Next, wavelength:\n",
    "\n",
    "<pre><code> vso.attrs.Wave(14.2*u.nm, 12.3*u.nm)</code></pre> \n",
    "\n",
    "We pass it a min and max wavelength. This has to be an astropy units quantity (in SI for the love of coffee). If you don't you will get an error.\n",
    " \n",
    "For a full list of attributes that vso can take use:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we can use multiple instument queries and define get smaller sample times but narrowing down the query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3361"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qr = client.query(vso.attrs.Time('2001/1/1T12:00:00', '2001/1/2T13:00:00'), vso.attrs.Instrument('eit') | vso.attrs.Instrument('mdi'))\n",
    "len(qr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HEK\n",
    "\n",
    "The Heliophysics Event Knowledgebase (HEK) is a repository of feature and event information about the Sun. Entries are generated both by automated algorithms and human observers.\n",
    "\n",
    "We need to set up HEK in a similar way to VSO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sunpy.net import hek\n",
    "hek_client = hek.HEKClient()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a very similar client as we saw with VSO above.\n",
    "\n",
    "Given that HEK is a database of solar events of interest, the query has different requirements to VSO. It needs start and end times, and an event type. Again time objects can be defined as datetime objects or correctly formatted strings.\n",
    "\n",
    "Event types are specified as uppercase two letter strings found on [the HEK website](http://www.lmsal.com/hek/VOEvent_Spec.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tstart = '2011/08/09 07:23:56'\n",
    "tend = '2011/08/09 12:40:29'\n",
    "event_type = 'FL'\n",
    "result = hek_client.query(hek.attrs.Time(tstart,tend),\n",
    "                          hek.attrs.EventType(event_type))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the HEK query is extremely similar to the VSO query style, with our attributes defined accordingly.\n",
    "\n",
    "Instead of returning a list, HEK returns a list of dictionary objects. Each entry in the dictionary sis a pair of key-value pairs that exactly correspond to the parameters. We can return the key words using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'concept',\n",
       " u'frm_versionnumber',\n",
       " u'gs_movieurl',\n",
       " u'hrc_coord',\n",
       " u'fl_peakflux',\n",
       " u'hpc_bbox',\n",
       " u'area_atdiskcenter',\n",
       " u'event_mapurl',\n",
       " u'event_c1error',\n",
       " u'obs_dataprepurl',\n",
       " u'hgc_coord',\n",
       " u'frm_identifier',\n",
       " u'intensmean',\n",
       " u'boundbox_c2ur',\n",
       " u'event_coordunit',\n",
       " u'gs_thumburl',\n",
       " u'fl_peakfluxunit',\n",
       " u'obs_meanwavel',\n",
       " u'bound_ccnsteps',\n",
       " u'skel_curvature',\n",
       " u'hgs_coord',\n",
       " u'bound_chaincode',\n",
       " u'frm_daterun',\n",
       " u'fl_peakem',\n",
       " u'intensmedian',\n",
       " u'fl_efoldtimeunit',\n",
       " u'bound_ccstartc1',\n",
       " u'frm_paramset',\n",
       " u'bound_ccstartc2',\n",
       " u'event_coord2',\n",
       " u'event_coord3',\n",
       " u'event_coord1',\n",
       " u'event_importance',\n",
       " u'kb_archivdate',\n",
       " u'event_title',\n",
       " u'fl_peaktempunit',\n",
       " u'hgc_bbox',\n",
       " u'skel_chaincode',\n",
       " u'intenstotal',\n",
       " u'hrc_a',\n",
       " u'area_atdiskcenteruncert',\n",
       " u'event_probability',\n",
       " u'hrc_boundcc',\n",
       " u'event_description',\n",
       " u'eventtype',\n",
       " u'obs_channelid',\n",
       " u'ar_mcintoshcls',\n",
       " u'frm_institute',\n",
       " u'frm_contact',\n",
       " u'ar_noaaclass',\n",
       " u'boundbox_c1ur',\n",
       " u'hgs_boundcc',\n",
       " u'boundbox_c2ll',\n",
       " u'area_unit',\n",
       " u'intensskew',\n",
       " u'hpc_coord',\n",
       " u'frm_name',\n",
       " u'obs_levelnum',\n",
       " u'area_uncert',\n",
       " u'ar_zurichcls',\n",
       " u'active',\n",
       " u'event_importance_num_ratings',\n",
       " u'obs_includesnrt',\n",
       " u'event_testflag',\n",
       " u'hpc_y',\n",
       " u'hpc_x',\n",
       " u'fl_fluence',\n",
       " u'hpc_boundcc',\n",
       " u'event_score',\n",
       " u'obs_lastprocessingdate',\n",
       " u'fl_halphaclass',\n",
       " u'refs_orig',\n",
       " u'ar_numspots',\n",
       " u'intensvar',\n",
       " u'fl_goescls',\n",
       " u'event_avg_rating',\n",
       " u'frm_url',\n",
       " u'fl_efoldtime',\n",
       " u'ar_compactnesscls',\n",
       " u'comment_count',\n",
       " u'event_npixels',\n",
       " u'event_clippedspatial',\n",
       " u'obs_wavelunit',\n",
       " u'frm_humanflag',\n",
       " u'hcr_checked',\n",
       " u'hrc_r',\n",
       " u'event_expires',\n",
       " u'noposition',\n",
       " u'event_peaktime',\n",
       " u'kb_archivist',\n",
       " u'SOL_standard',\n",
       " u'event_coordsys',\n",
       " u'hgc_boundcc',\n",
       " u'gs_galleryid',\n",
       " u'event_maskurl',\n",
       " u'skel_startc1',\n",
       " u'skel_startc2',\n",
       " u'obs_title',\n",
       " u'event_type',\n",
       " u'refs',\n",
       " u'hgc_x',\n",
       " u'hgc_y',\n",
       " u'chaincodetype',\n",
       " u'kb_archivid',\n",
       " u'ar_penumbracls',\n",
       " u'obs_firstprocessingdate',\n",
       " u'event_endtime',\n",
       " u'hrc_bbox',\n",
       " u'skel_nsteps',\n",
       " u'rasterscan',\n",
       " u'ar_mtwilsoncls',\n",
       " u'event_pixelunit',\n",
       " u'ar_noaanum',\n",
       " u'revision',\n",
       " u'hgs_x',\n",
       " u'hgs_y',\n",
       " u'intenskurt',\n",
       " u'ar_polarity',\n",
       " u'obs_instrument',\n",
       " u'frm_specificid',\n",
       " u'boundbox_c1ll',\n",
       " u'area_raw',\n",
       " u'rasterscantype',\n",
       " u'obs_observatory',\n",
       " u'intensunit',\n",
       " u'gs_imageurl',\n",
       " u'hgs_bbox',\n",
       " u'event_starttime',\n",
       " u'fl_peakemunit',\n",
       " u'event_clippedtemporal',\n",
       " u'fl_fluenceunit',\n",
       " u'intensmax',\n",
       " u'fl_peaktemp',\n",
       " u'intensmin',\n",
       " u'hpc_geom',\n",
       " u'event_c2error']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result[0].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, the HEK query we made returns all the flares in the time-range stored in the HEK, regardless of the feature recognition method. The HEK parameter which stores the the feature recognition method is called “frm_name”. Using list comprehensions (which are very cool), it is easy to get a list of the feature recognition methods used to find each of the flares in the result object, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "asainz\n",
      "asainz\n",
      "asainz\n",
      "asainz\n",
      "asainz\n",
      "asainz\n",
      "asainz\n",
      "SWPC\n",
      "SSW Latest Events\n",
      "Flare Detective - Trigger Module\n",
      "Flare Detective - Trigger Module\n",
      "SWPC\n",
      "SSW Latest Events\n",
      "Flare Detective - Trigger Module\n",
      "Flare Detective - Trigger Module\n",
      "Flare Detective - Trigger Module\n",
      "Flare Detective - Trigger Module\n",
      "Flare Detective - Trigger Module\n",
      "Flare Detective - Trigger Module\n"
     ]
    }
   ],
   "source": [
    "for elem in result: \n",
    "    print(elem[\"frm_name\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This way we can avoid troublesome doubling up of results. We can do the same `help(hek.attrs)` command as VSO to fins out further options. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aquiring data with AstroQuery\n",
    "\n",
    "Astroquery supports a plethora of [services](https://astroquery.readthedocs.org/en/latest/#using-astroquery), all of which follow roughly the same API (application program interface). In its simplest for the API involves queries based on coordinates or object names e.g. using SIMBAD:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAIN_ID      RA         DEC     ... COO_QUAL COO_WAVELENGTH     COO_BIBCODE    \n",
      "          \"h:m:s\"     \"d:m:s\"   ...                                            \n",
      "------- ----------- ----------- ... -------- -------------- -------------------\n",
      "  M   1 05 34 31.94 +22 00 52.2 ...        C              R 2011A&A...533A..10L\n"
     ]
    }
   ],
   "source": [
    "from astroquery.simbad import Simbad\n",
    "result_table = Simbad.query_object(\"m1\")\n",
    "result_table.pprint(show_unit=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case the query is looking at a specific set of coordinates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        MAIN_ID              RA      ... COO_WAVELENGTH     COO_BIBCODE    \n",
      "                          \"h:m:s\"    ...                                   \n",
      "----------------------- ------------ ... -------------- -------------------\n",
      "        NAME ORI REGION  05 35 17.30 ...              O                    \n",
      "                    ...          ... ...            ...                 ...\n",
      "2MASS J05353130-0519533  05 35 31.31 ...              I 2003yCat.2246....0C\n",
      "           V* V2114 Ori 05 35 01.671 ...              I 2003yCat.2246....0C\n",
      "Length = 2977 rows\n"
     ]
    }
   ],
   "source": [
    "from astropy import coordinates\n",
    "import astropy.units as u\n",
    "c = coordinates.SkyCoord(\"05h35m17.3s -05d23m28s\", frame='icrs')\n",
    "r = 5 * u.arcminute\n",
    "result = Simbad.query_region(c, radius=r)\n",
    "result.pprint(show_unit=True, max_width=80, max_lines=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These methods can be expanded to all the following modules\n",
    "\n",
    "\n",
    "*    SIMBAD Queries (astroquery.simbad)\n",
    "*    IRSA Dust Extinction Service Queries (astroquery.irsa_dust)\n",
    "*    NED Queries (astroquery.ned)\n",
    "*    Splatalogue Queries (astroquery.splatalogue)\n",
    "*    IRSA Image Server program interface (IBE) Queries (astroquery.ibe)\n",
    "*    IRSA Queries (astroquery.irsa)\n",
    "*    UKIDSS Queries (astroquery.ukidss)\n",
    "*    MAGPIS Queries (astroquery.magpis)\n",
    "*    NRAO Queries (astroquery.nrao)\n",
    "*    Besancon Queries (astroquery.besancon)\n",
    "*    NIST Queries (astroquery.nist)\n",
    "*    NVAS Queries (astroquery.nvas)\n",
    "*    GAMA Queries (astroquery.gama)\n",
    "*    ESO Queries (astroquery.eso)\n",
    "*    Atomic Line List (astroquery.atomic)\n",
    "*    ALMA Queries (astroquery.alma)\n",
    "*    Skyview Queries (astroquery.skyview)\n",
    "*    NASA ADS Queries (astroquery.nasa_ads)\n",
    "*    HEASARC Queries (astroquery.heasarc)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python2",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}