SystemML Assignment 1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 4\n",
    "## Understaning scaling of linear algebra operations on Apache Spark using Apache SystemML\n",
    "\n",
    "In this assignment we want you to understand how to scale linear algebra operations from a single machine to multiple machines, memory and CPU cores using Apache SystemML. Therefore we want you to understand how to migrate from a numpy program to a SystemML DML program. Don't worry. We will give you a lot of hints. Finally, you won't need this knowledge anyways if you are sticking to Keras only, but once you go beyond that point you'll be happy to see what's going on behind the scenes. Please make sure you run this notebook from an Apache Spark 2.3 notebook.\n",
    "\n",
    "So the first thing we need to ensure is that we are on the latest version of SystemML, which is 1.2.0 (as of Feb. '19)\n",
    "Please use the code block below to check if you are already on 1.2.0 or higher."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for a Spark session to start...\n",
      "Spark Initialization Done! ApplicationId = app-20190903014230-0000\n",
      "KERNEL_ID = 9c578721-6a4b-4a98-81fd-c640fb364280\n"
     ]
    },
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'systemml'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-8222421bc53c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0msystemml\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mMLContext\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mml\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mMLContext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mspark\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mml\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mversion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'systemml'"
     ]
    }
   ],
   "source": [
    "from systemml import MLContext\n",
    "ml = MLContext(spark)\n",
    "ml.version()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are blow version 1.2.0 please execute the next two code blocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting systemml\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b1/94/62104cb8c526b462cd501c7319926fb81ac9a5668574a0b3407658a506ab/systemml-1.2.0.tar.gz (9.7MB)\n",
      "\u001b[K    100% |################################| 9.7MB 735kB/s ta 0:00:01\n",
      "\u001b[?25hCollecting numpy>=1.8.2 (from systemml)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/75/92/57179ed45307ec6179e344231c47da7f3f3da9e2eee5c8ab506bd279ce4e/numpy-1.17.1-cp36-cp36m-manylinux1_x86_64.whl (20.4MB)\n",
      "\u001b[K    100% |################################| 20.4MB 640kB/s ta 0:00:01    41% |#############                   | 8.5MB 47.7MB/s eta 0:00:01    69% |######################          | 14.1MB 1.3MB/s eta 0:00:05\n",
      "\u001b[?25hCollecting scipy>=0.15.1 (from systemml)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/29/50/a552a5aff252ae915f522e44642bb49a7b7b31677f9580cfd11bcc869976/scipy-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (25.2MB)\n",
      "\u001b[K    100% |################################| 25.2MB 510kB/s ta 0:00:011    98% |############################### | 24.8MB 36.6MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting pandas (from systemml)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/73/9b/52e228545d14f14bb2a1622e225f38463c8726645165e1cb7dde95bfe6d4/pandas-0.25.1-cp36-cp36m-manylinux1_x86_64.whl (10.5MB)\n",
      "\u001b[K    100% |################################| 10.5MB 852kB/s ta 0:00:01\n",
      "\u001b[?25hCollecting scikit-learn (from systemml)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a0/c5/d2238762d780dde84a20b8c761f563fe882b88c5a5fb03c056547c442a19/scikit_learn-0.21.3-cp36-cp36m-manylinux1_x86_64.whl (6.7MB)\n",
      "\u001b[K    100% |################################| 6.7MB 940kB/s ta 0:00:01\n",
      "\u001b[?25hCollecting Pillow>=2.0.0 (from systemml)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/14/41/db6dec65ddbc176a59b89485e8cc136a433ed9c6397b6bfe2cd38412051e/Pillow-6.1.0-cp36-cp36m-manylinux1_x86_64.whl (2.1MB)\n",
      "\u001b[K    100% |################################| 2.1MB 1.4MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting pytz>=2017.2 (from pandas->systemml)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/87/76/46d697698a143e05f77bec5a526bf4e56a0be61d63425b68f4ba553b51f2/pytz-2019.2-py2.py3-none-any.whl (508kB)\n",
      "\u001b[K    100% |################################| 512kB 1.2MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting python-dateutil>=2.6.1 (from pandas->systemml)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)\n",
      "\u001b[K    100% |################################| 235kB 1.3MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting joblib>=0.11 (from scikit-learn->systemml)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/cd/c1/50a758e8247561e58cb87305b1e90b171b8c767b15b12a1734001f41d356/joblib-0.13.2-py2.py3-none-any.whl (278kB)\n",
      "\u001b[K    100% |################################| 286kB 1.4MB/s ta 0:00:01\n",
      "\u001b[?25hCollecting six>=1.5 (from python-dateutil>=2.6.1->pandas->systemml)\n",
      "  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl\n",
      "Building wheels for collected packages: systemml\n",
      "  Building wheel for systemml (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /home/spark/shared/.cache/pip/wheels/cf/07/79/b3ed6f12afe06b2ab55d60dcfd62e66240f5d8c6088a518177\n",
      "Successfully built systemml\n",
      "\u001b[31mtensorflow 1.13.1 requires tensorboard<1.14.0,>=1.13.0, which is not installed.\u001b[0m\n",
      "Installing collected packages: numpy, scipy, pytz, six, python-dateutil, pandas, joblib, scikit-learn, Pillow, systemml\n",
      "Successfully installed Pillow-6.1.0 joblib-0.13.2 numpy-1.17.1 pandas-0.25.1 python-dateutil-2.8.0 pytz-2019.2 scikit-learn-0.21.3 scipy-1.3.1 six-1.12.0 systemml-1.2.0\n"
     ]
    }
   ],
   "source": [
    "!pip install systemml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to create two sym links that the newest version is picket up - this is a workaround and will be removed soon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ln -s -f ~/user-libs/python3/systemml/systemml-java/systemml-1.2.0-extra.jar ~/user-libs/spark2/systemml-1.2.0-extra.jar\n",
    "!ln -s -f ~/user-libs/python3/systemml/systemml-java/systemml-1.2.0.jar ~/user-libs/spark2/systemml-1.2.0.jar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now please restart the kernel and make sure the version is correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for a Spark session to start...\n",
      "Spark Initialization Done! ApplicationId = app-20190903014624-0001\n",
      "KERNEL_ID = 9c578721-6a4b-4a98-81fd-c640fb364280\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'1.2.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from systemml import MLContext\n",
    "ml = MLContext(spark)\n",
    "ml.version()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations, if you see version 1.2.0 or higher, please continue with the notebook..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from systemml import MLContext, dml\n",
    "import numpy as np\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we create an MLContext to interface with Apache SystemML. Note that we pass a SparkSession object as parameter so SystemML now knows how to talk to the Apache Spark cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml = MLContext(spark)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create some large random matrices to have numpy and SystemML crunch on it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "u = np.random.rand(1000,10000)\n",
    "s = np.random.rand(10000,1000)\n",
    "w = np.random.rand(1000,1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we implement a short one-liner to define a very simple linear algebra operation\n",
    "\n",
    "In case you are unfamiliar with matrxi-matrix multiplication: https://en.wikipedia.org/wiki/Matrix_multiplication\n",
    "\n",
    "sum(U' * (W . (U * S)))\n",
    "\n",
    "\n",
    "| Legend        |            |   \n",
    "| ------------- |-------------| \n",
    "| '      | transpose of a matrix | \n",
    "| * | matrix-matrix multiplication      |  \n",
    "| . | scalar multiplication      |   \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.5945181846618652\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "res = np.sum(u.T.dot(w * u.dot(s)))\n",
    "print (time.time()-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see this executes perfectly fine. Note that this is even a very efficient execution because numpy uses a C/C++ backend which is known for it's performance. But what happens if U, S or W get such big that the available main memory cannot cope with it? Let's give it a try:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#u = np.random.rand(10000,100000)\n",
    "#s = np.random.rand(100000,10000)\n",
    "#w = np.random.rand(10000,10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After a short while you should see a memory error. This is because the operating system process was not able to allocate enough memory for storing the numpy array on the heap. Now it's time to re-implement the very same operations as DML in SystemML, and this is your task. Just replace all ###your_code_goes_here sections with proper code, please consider the following table which contains all DML syntax you need:\n",
    "\n",
    "| Syntax        |            |   \n",
    "| ------------- |-------------| \n",
    "| t(M)      | transpose of a matrix, where M is the matrix | \n",
    "| %*% | matrix-matrix multiplication      |  \n",
    "| * | scalar multiplication      |   \n",
    "\n",
    "## Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to show you the advantage of SystemML over numpy we've blown up the sizes of the matrices. Unfortunately, on a 1-2 worker Spark cluster it takes quite some time to complete. Therefore we've stripped down the example to smaller matrices below, but we've kept the code, just in case you are curious to check it out. But you might want to use some more workers which you easily can configure in the environment settings of the project within Watson Studio. Just be aware that you're currently limited to free 50 capacity unit hours per month wich are consumed by the additional workers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "script = \"\"\"\n",
    "U = rand(rows=1000,cols=10000, seed=5)\n",
    "S = rand(rows=10000,cols=1000, seed=23)\n",
    "W = rand(rows=1000,cols=1000, seed=42)\n",
    "res = sum(t(U) %*% (W * (U %*% S)))\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get consistent results we switch from a random matrix initialization to something deterministic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SystemML Statistics:\n",
      "Total execution time:\t\t4.226 sec.\n",
      "Number of executed Spark inst:\t0.\n",
      "\n",
      "\n",
      "6252492444241.075\n"
     ]
    }
   ],
   "source": [
    "prog = dml(script).output('res')\n",
    "res = ml.execute(prog).get('res')\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If everything runs fine you should get *6252492444241.075* as result (or something in that bullpark). Feel free to submit your DML script to the grader now!\n",
    "\n",
    "### Submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-09-03 01:47:19--  https://raw.githubusercontent.com/romeokienzler/developerWorks/master/coursera/ai/rklib.py\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2289 (2.2K) [text/plain]\n",
      "Saving to: 'rklib.py'\n",
      "\n",
      "rklib.py            100%[===================>]   2.24K  --.-KB/s    in 0s      \n",
      "\n",
      "2019-09-03 01:47:19 (55.6 MB/s) - 'rklib.py' saved [2289/2289]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm -f rklib.py\n",
    "!wget https://raw.githubusercontent.com/romeokienzler/developerWorks/master/coursera/ai/rklib.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from rklib import submit\n",
    "key = \"esRk7vn-Eeej-BLTuYzd0g\"\n",
    "\n",
    "\n",
    "email = \"dennislamcv@gmail.com\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submission successful, please check on the coursera grader page for the status\n",
      "-------------------------\n",
      "{\"elements\":[{\"itemId\":\"P1p3F\",\"id\":\"tE4j0qhMEeecqgpT6QjMdA~P1p3F~2HuEMM3sEemAsw6NyXp6ZA\",\"courseId\":\"tE4j0qhMEeecqgpT6QjMdA\"}],\"paging\":{},\"linked\":{}}\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "part = \"fUxc8\"\n",
    "secret = \"4gEdAwtc8eIORdVM\"\n",
    "submit(email, secret, key, part, [part], script)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 with Spark",
   "language": "python3",
   "name": "python36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}