Work%2Bwith%2Ba%2Bremote%2BSpark%2BSample%2BOne%2BLocal.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Working with an existing remote Spark via HTTP (sample 1)  \n",
    "\n",
    "IBM DSX Local provides the interface for Python notebooks to work with an existing remote Spark through HTTP connection and user-friendly sparkmagic commands. This sample notebook shows how to send a simple request to remote Spark.\n",
    "\n",
    "The installation of the remote Spark in this sample is using Horton Data Platform (HDP), which utilizes Livy HTTP REST API. Livy is an open source REST interface for interacting with [Apache Spark](http://spark.apache.org) from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in [Apache Hadoop YARN](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html).\n",
    "\n",
    "This notebook runs on Python 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "  ## Table of contents \n",
    "\n",
    "   1.  [Load sparkmagic](#load-sparkmagic)<br>\n",
    "   2.  [Create a connection to remote Spark](#connection-to-remote-spark)<br>\n",
    "   3.  [Send a request to remote Spark](#send-request-remote-spark)<br>\n",
    "   4.  [Delete the remote Spark session](#delete-session)<br>        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"load-sparkmagic\"></a>\n",
    "## 1. Load sparkmagic\n",
    "\n",
    "Sparkmagic is a set of tools for interactively working with remote Spark clusters through Livy, a Spark REST server, in Jupyter notebooks. The Sparkmagic project includes a set of magics for interactively running Spark code in multiple languages, as well as some kernels that you can use to turn Jupyter into an integrated Spark environment.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "11/02/2017 12:51:12 AM - proxy_util - INFO - Set custom headers.\n",
      "11/02/2017 12:51:12 AM - proxy_util - INFO - Set proxy user to be useed with Livy.\n",
      "11/02/2017 12:51:12 AM - proxy_util - INFO - proxy settings set.\n"
     ]
    }
   ],
   "source": [
    "%load_ext sparkmagic.magics\n",
    "import dsx_core_utils\n",
    "dsx_core_utils.setup_livy_sparkmagic()\n",
    "%reload_ext sparkmagic.magics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"connection-to-remote-spark\"></a>\n",
    "##  2. Create a connection to remote Spark \n",
    "\n",
    "Run the following cell to invoke the user interface for managing Spark. In the user interface, perform the following tasks to create a connection to the remote Spark:\n",
    " * Check **Manage Endpoints**. If you already see an endpoint defined, then your DSX Admin has configured a default DSX Endpoint.\n",
    " * Otherwise, select the **Add Endpoint** tab to create the endpoint of the Livy service URL. Type the Livy service URL in the **Address** field, select the authentication type, and specify the authentication credentials if required. Then, select the **Add endpoint** button.\n",
    " * Select the **Add Session** tab to create a session. Choose the endpoint, type the session name, and choose the language. Then, select the **Create Session** button. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting Spark application\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>Current session?</th></tr><tr><td>140</td><td>application_1509308217126_0034</td><td>pyspark</td><td>idle</td><td><a target=\"_blank\" href=\"http://vend1.fyre.ibm.com:8088/proxy/application_1509308217126_0034/\">Link</a></td><td><a target=\"_blank\" href=\"http://vend7.fyre.ibm.com:8042/node/containerlogs/container_e12_1509308217126_0034_01_000001/user1\">Link</a></td><td>✔</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SparkContext available as 'sc'.\n",
      "HiveContext available as 'sqlContext'.\n"
     ]
    }
   ],
   "source": [
    "%manage_spark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"send-request-remote-spark\"></a>\n",
    "## 3. Send a request to remote Spark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Run the following cell to send Python Spark codes that calculate and return the Pi number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pi is roughly 3.145640"
     ]
    }
   ],
   "source": [
    "%%spark \n",
    "import random\n",
    "NUM_SAMPLES = 100000\n",
    "\n",
    "def sample(p):\n",
    "    x, y = random.random(), random.random()\n",
    "    return 1 if x*x + y*y < 1 else 0\n",
    "\n",
    "count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample).reduce(lambda a, b: a + b)\n",
    "print \"Pi is roughly %f\" % (4.0 * count / NUM_SAMPLES)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a id=\"delete-session\"></a>\n",
    "## 4. Delete the remote Spark session\n",
    "\n",
    "On the **Manage Sessions** tab in step 2, select the **Delete** button to delete the session."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Summary\n",
    "\n",
    "In this notebook, you learned how to send a simple request to remote Spark by using the Livy HTTP REST API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div class=\"alert alert-block alert-info\"> Note: To save resources and get the best performance please use the code below to stop the kernel before exiting your notebook.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%javascript\n",
    "Jupyter.notebook.session.delete();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<hr>\n",
    "Copyright &copy; IBM Corp. 2017. Released as licensed Sample Materials."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  },
  "widgets": {
   "state": {
    "091c60fb23cc4b7ea1f448b57587d7a6": {
     "views": [
      {
       "cell_index": 5
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}