docs/README-CONNECT-TO-CEPH-FROM-PYSPARK-NB.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview\n",
    "\n",
    "This demo walks through connecting to Ceph from an EPIC cluster using pySpark."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup\n",
    "\n",
    "- On your client machine, run the script `./scripts/end_user_scripts/ceph/1_demo_server_setup.sh` to setup a ceph nano server on the RDP Server host\n",
    "- Add the EPIC Spark 2.4 image\n",
    "- Configure EPIC with Active Directory [see README](https://github.com/bluedata-community/bluedata-demo-env-aws-terraform/blob/master/docs/README-AD.md)\n",
    "- Setup Demo Tenant with Active Directory [see README](https://github.com/bluedata-community/bluedata-demo-env-aws-terraform/blob/master/docs/README-AD.md)\n",
    "- Provision a Spark 2.4 cluster in the Demo Tenant with:\n",
    "  - 1 x Spark Controller (small)\n",
    "  - 1 x Jupyter Hub (small)\n",
    "- Classic Jupyter notebook in Jupyterhub (Open Jupyterhub and nagivate to Help -> Launch Classic Notebook)\n",
    "- SSH into the RDP Host and upload a dataset:\n",
    "\n",
    "```\n",
    "wget https://raw.githubusercontent.com/fivethirtyeight/data/master/airline-safety/airline-safety.csv\n",
    "sed -i -e \"s/\\r/\\n/g\" airline-safety.csv # convert line endings\n",
    "s3cmd put ./airline-safety.csv s3://sandboxbucket/airline-safety.csv\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect\n",
    "\n",
    "- Verify that we are able to get a response from the ceph instance. We should see something like:\n",
    "\n",
    "```\n",
    "<?xml version=\"1.0\" encoding=\"UTF-8\"?><ListAllMyBucketsResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- test we are able to connect to the spark context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<SparkContext master=spark://bluedata-1.demo.bdlocal:7077 appName=IBM Spark Kernel>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- set connection to ceph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark import SparkContext\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark import SparkConf\n",
    "\n",
    "conf = (SparkConf(). \n",
    "        set(\"spark.executor.extraJavaOptions\",\"-Dcom.amazonaws.services.s3.enableV4=true\").\n",
    "        set(\"spark.driver.extraJavaOptions\",\"-Dcom.amazonaws.services.s3.enableV4=true\")\n",
    "       )\n",
    "\n",
    "sc.setSystemProperty(\"com.amazonaws.services.s3.enableV4\", \"true\")\n",
    "sc._jsc.hadoopConfiguration().set(\"fs.s3a.access.key\", \"sandboxAccessKey\")\n",
    "sc._jsc.hadoopConfiguration().set(\"fs.s3a.secret.key\", \"sandboxSecretKey\")\n",
    "sc._jsc.hadoopConfiguration().set(\"fs.s3a.endpoint\", \"10.1.0.216:8080\")   #### Change to the private IP of RDP server \n",
    "sc._jsc.hadoopConfiguration().set(\"fs.s3a.connection.ssl.enabled\", \"false\")\n",
    "sc._jsc.hadoopConfiguration().set(\"com.amazonaws.services.s3a.enableV4\", \"true\")\n",
    "sc._jsc.hadoopConfiguration().set(\"fs.s3a.impl\", \"org.apache.hadoop.fs.s3a.S3AFileSystem\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- retrieve some data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(_c0='airline', _c1='avail_seat_km_per_week', _c2='incidents_85_99', _c3='fatal_accidents_85_99', _c4='fatalities_85_99', _c5='incidents_00_14', _c6='fatal_accidents_00_14', _c7='fatalities_00_14')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sql = SparkSession(sc)\n",
    "csv_df = sql.read.csv(\"s3a://sandboxbucket/airline-safety.csv\")\n",
    "csv_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Apache Toree - PySpark",
   "language": "/opt/anaconda3/bin/python",
   "name": "apache_toree_pyspark"
  },
  "language_info": {
   "name": "scala",
   "version": "2.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}