colab_rapids_umap_01.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "colab-rapids-umap-01.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/martin-fabbri/colab-notebooks/blob/master/colab_rapids_umap_01.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "scfLT2i0MLyD",
        "colab_type": "text"
      },
      "source": [
        "# Environment Sanity Check #\n",
        "\n",
        "Click the _Runtime_ dropdown at the top of the page, then _Change Runtime Type_ and confirm the instance type is _GPU_.\n",
        "\n",
        "Check the output of `!nvidia-smi` to make sure you've been allocated a Tesla T4."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B0C8IV5TQnjN",
        "colab_type": "code",
        "outputId": "b9ca77b2-d6a2-4a52-bc0d-85a3f853400d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 319
        }
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Thu Sep 12 16:55:32 2019       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 430.40       Driver Version: 418.67       CUDA Version: 10.1     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   55C    P8    29W / 149W |      0MiB / 11441MiB |      0%      Default |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                       GPU Memory |\n",
            "|  GPU       PID   Type   Process name                             Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ik8Wv2K0uDfx",
        "colab_type": "code",
        "outputId": "b440ffb8-b485-40b2-f7d8-7b044047ed3a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "import pynvml\n",
        "\n",
        "pynvml.nvmlInit()\n",
        "handle = pynvml.nvmlDeviceGetHandleByIndex(0)\n",
        "device_name = pynvml.nvmlDeviceGetName(handle)\n",
        "\n",
        "if device_name != b'Tesla T4':\n",
        "  raise Exception(\"\"\"\n",
        "    Unfortunately this instance does not have a T4 GPU.\n",
        "    \n",
        "    Please make sure you've configured Colab to request a GPU instance type.\n",
        "    \n",
        "    Sometimes Colab allocates a Tesla K80 instead of a T4. Resetting the instance.\n",
        "\n",
        "    If you get a K80 GPU, try Runtime -> Reset all runtimes...\n",
        "  \"\"\")\n",
        "else:\n",
        "  print('Woo! You got the right kind of GPU!')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Woo! You got the right kind of GPU!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CtNdk7PSafKP",
        "colab_type": "text"
      },
      "source": [
        "#Setup:\n",
        "\n",
        "1. Install most recent Miniconda release compatible with Google Colab's Python install  (3.6.7)\n",
        "2. Install RAPIDS libraries\n",
        "3. Set necessary environment variables\n",
        "4. Copy RAPIDS .so files into current working directory, a workaround for conda/colab interactions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m0jdXBRiDSzj",
        "colab_type": "code",
        "outputId": "947a9bf9-348e-4dcf-ca61-95b8fdb10c40",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 2703
        }
      },
      "source": [
        "# intall miniconda\n",
        "!wget -c https://repo.continuum.io/miniconda/Miniconda3-4.5.4-Linux-x86_64.sh\n",
        "!chmod +x Miniconda3-4.5.4-Linux-x86_64.sh\n",
        "!bash ./Miniconda3-4.5.4-Linux-x86_64.sh -b -f -p /usr/local\n",
        "\n",
        "# install RAPIDS packages\n",
        "!conda install -q -y --prefix /usr/local -c conda-forge \\\n",
        "  -c rapidsai-nightly/label/cuda10.0 -c nvidia/label/cuda10.0 \\\n",
        "  cudf cuml\n",
        "\n",
        "# set environment vars\n",
        "import sys, os, shutil\n",
        "sys.path.append('/usr/local/lib/python3.6/site-packages/')\n",
        "os.environ['NUMBAPRO_NVVM'] = '/usr/local/cuda/nvvm/lib64/libnvvm.so'\n",
        "os.environ['NUMBAPRO_LIBDEVICE'] = '/usr/local/cuda/nvvm/libdevice/'\n",
        "\n",
        "# copy .so files to current working dir\n",
        "for fn in ['libcudf.so', 'librmm.so']:\n",
        "  shutil.copy('/usr/local/lib/'+fn, os.getcwd())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-04-27 18:48:38--  https://repo.continuum.io/miniconda/Miniconda3-4.5.4-Linux-x86_64.sh\n",
            "Resolving repo.continuum.io (repo.continuum.io)... 104.18.201.79, 104.18.200.79, 2606:4700::6812:c84f, ...\n",
            "Connecting to repo.continuum.io (repo.continuum.io)|104.18.201.79|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 58468498 (56M) [application/x-sh]\n",
            "Saving to: ‘Miniconda3-4.5.4-Linux-x86_64.sh’\n",
            "\n",
            "\r          Miniconda   0%[                    ]       0  --.-KB/s               \r         Miniconda3  51%[=========>          ]  28.68M   143MB/s               \rMiniconda3-4.5.4-Li 100%[===================>]  55.76M   156MB/s    in 0.4s    \n",
            "\n",
            "2019-04-27 18:48:38 (156 MB/s) - ‘Miniconda3-4.5.4-Linux-x86_64.sh’ saved [58468498/58468498]\n",
            "\n",
            "PREFIX=/usr/local\n",
            "installing: python-3.6.5-hc3d631a_2 ...\n",
            "Python 3.6.5 :: Anaconda, Inc.\n",
            "installing: ca-certificates-2018.03.07-0 ...\n",
            "installing: conda-env-2.6.0-h36134e3_1 ...\n",
            "installing: libgcc-ng-7.2.0-hdf63c60_3 ...\n",
            "installing: libstdcxx-ng-7.2.0-hdf63c60_3 ...\n",
            "installing: libffi-3.2.1-hd88cf55_4 ...\n",
            "installing: ncurses-6.1-hf484d3e_0 ...\n",
            "installing: openssl-1.0.2o-h20670df_0 ...\n",
            "installing: tk-8.6.7-hc745277_3 ...\n",
            "installing: xz-5.2.4-h14c3975_4 ...\n",
            "installing: yaml-0.1.7-had09818_2 ...\n",
            "installing: zlib-1.2.11-ha838bed_2 ...\n",
            "installing: libedit-3.1.20170329-h6b74fdf_2 ...\n",
            "installing: readline-7.0-ha6073c6_4 ...\n",
            "installing: sqlite-3.23.1-he433501_0 ...\n",
            "installing: asn1crypto-0.24.0-py36_0 ...\n",
            "installing: certifi-2018.4.16-py36_0 ...\n",
            "installing: chardet-3.0.4-py36h0f667ec_1 ...\n",
            "installing: idna-2.6-py36h82fb2a8_1 ...\n",
            "installing: pycosat-0.6.3-py36h0a5515d_0 ...\n",
            "installing: pycparser-2.18-py36hf9f622e_1 ...\n",
            "installing: pysocks-1.6.8-py36_0 ...\n",
            "installing: ruamel_yaml-0.15.37-py36h14c3975_2 ...\n",
            "installing: six-1.11.0-py36h372c433_1 ...\n",
            "installing: cffi-1.11.5-py36h9745a5d_0 ...\n",
            "installing: setuptools-39.2.0-py36_0 ...\n",
            "installing: cryptography-2.2.2-py36h14c3975_0 ...\n",
            "installing: wheel-0.31.1-py36_0 ...\n",
            "installing: pip-10.0.1-py36_0 ...\n",
            "installing: pyopenssl-18.0.0-py36_0 ...\n",
            "installing: urllib3-1.22-py36hbe7ace6_0 ...\n",
            "installing: requests-2.18.4-py36he2e5f8d_1 ...\n",
            "installing: conda-4.5.4-py36_0 ...\n",
            "installation finished.\n",
            "WARNING:\n",
            "    You currently have a PYTHONPATH environment variable set. This may cause\n",
            "    unexpected behavior when running the Python interpreter in Miniconda3.\n",
            "    For best results, please verify that your PYTHONPATH only points to\n",
            "    directories of packages that are compatible with the Python interpreter\n",
            "    in Miniconda3: /usr/local\n",
            "Solving environment: ...working... done\n",
            "\n",
            "## Package Plan ##\n",
            "\n",
            "  environment location: /usr/local\n",
            "\n",
            "  added / updated specs: \n",
            "    - cudf\n",
            "    - cuml\n",
            "\n",
            "\n",
            "The following packages will be downloaded:\n",
            "\n",
            "    package                    |            build\n",
            "    ---------------------------|-----------------\n",
            "    python-3.6.7               |    h381d211_1004        34.5 MB  conda-forge\n",
            "    numpy-1.15.4               |py36h8b7e671_1002         4.2 MB  conda-forge\n",
            "    cryptography-2.6.1         |   py36h72c5cf5_0         606 KB  conda-forge\n",
            "    libblas-3.8.0              |       7_openblas           6 KB  conda-forge\n",
            "    python-dateutil-2.8.0      |             py_0         219 KB  conda-forge\n",
            "    librmm-0.7.0.dev0          |      cuda10.0_30          39 KB  rapidsai-nightly/label/cuda10.0\n",
            "    rmm-0.7.0.dev0             |          py36_30          14 KB  rapidsai-nightly/label/cuda10.0\n",
            "    llvmlite-0.28.0            |   py36hdbcaa40_0        20.2 MB  conda-forge\n",
            "    parquet-cpp-1.5.1          |                4           3 KB  conda-forge\n",
            "    conda-4.6.14               |           py36_0         2.1 MB  conda-forge\n",
            "    pandas-0.24.2              |   py36hf484d3e_0        11.1 MB  conda-forge\n",
            "    tk-8.6.9                   |    h84994c4_1001         3.2 MB  conda-forge\n",
            "    cudf-0.7.0.dev0            |         py36_874         1.9 MB  rapidsai-nightly/label/cuda10.0\n",
            "    libcuml-0.7.0.dev0         |     cuda10.0_391        21.6 MB  rapidsai-nightly/label/cuda10.0\n",
            "    libgcc-ng-8.2.0            |       hdf63c60_1         7.6 MB\n",
            "    libstdcxx-ng-8.2.0         |       hdf63c60_1         2.9 MB\n",
            "    libcudf-0.7.0.dev0         |     cuda10.0_874        17.7 MB  rapidsai-nightly/label/cuda10.0\n",
            "    sqlite-3.26.0              |    h67949de_1001         1.9 MB  conda-forge\n",
            "    openblas-0.3.5             |       ha44fe06_0        13.7 MB  conda-forge\n",
            "    bzip2-1.0.6                |    h14c3975_1002         415 KB  conda-forge\n",
            "    libcumlmg-0.0.0.dev0       |     cuda10.0_373         955 KB  nvidia/label/cuda10.0\n",
            "    ca-certificates-2019.3.9   |       hecc5488_0         146 KB  conda-forge\n",
            "    openssl-1.1.1b             |       h14c3975_1         4.0 MB  conda-forge\n",
            "    boost-cpp-1.68.0           |    h11c811c_1000        20.5 MB  conda-forge\n",
            "    liblapack-3.8.0            |       7_openblas           6 KB  conda-forge\n",
            "    libnvstrings-0.7.0.dev0    |     cuda10.0_102         9.3 MB  rapidsai-nightly/label/cuda10.0\n",
            "    libcblas-3.8.0             |       7_openblas           6 KB  conda-forge\n",
            "    libgfortran-3.0.0          |                1         281 KB  conda-forge\n",
            "    libprotobuf-3.6.1          |    hdbcaa40_1001         4.0 MB  conda-forge\n",
            "    cuml-0.7.0.dev0            |cuda10.0_py36_391         2.6 MB  rapidsai-nightly/label/cuda10.0\n",
            "    arrow-cpp-0.12.1           |   py36h0e61e49_0         6.9 MB  conda-forge\n",
            "    thrift-cpp-0.12.0          |    h0a07b25_1002         2.4 MB  conda-forge\n",
            "    icu-58.2                   |    hf484d3e_1000        22.6 MB  conda-forge\n",
            "    libcudf_cffi-0.7.0.dev0    |         py36_874          19 KB  rapidsai-nightly/label/cuda10.0\n",
            "    certifi-2019.3.9           |           py36_0         149 KB  conda-forge\n",
            "    pytz-2019.1                |             py_0         227 KB  conda-forge\n",
            "    pyarrow-0.12.1             |   py36hbbcf98d_0         2.2 MB  conda-forge\n",
            "    nvstrings-0.7.0.dev0       |         py36_102          86 KB  rapidsai-nightly/label/cuda10.0\n",
            "    numba-0.43.1               |   py36he1b5a44_0         2.9 MB  conda-forge\n",
            "    ------------------------------------------------------------\n",
            "                                           Total:       222.9 MB\n",
            "\n",
            "The following NEW packages will be INSTALLED:\n",
            "\n",
            "    arrow-cpp:       0.12.1-py36h0e61e49_0        conda-forge                    \n",
            "    boost-cpp:       1.68.0-h11c811c_1000         conda-forge                    \n",
            "    bzip2:           1.0.6-h14c3975_1002          conda-forge                    \n",
            "    cudf:            0.7.0.dev0-py36_874          rapidsai-nightly/label/cuda10.0\n",
            "    cuml:            0.7.0.dev0-cuda10.0_py36_391 rapidsai-nightly/label/cuda10.0\n",
            "    icu:             58.2-hf484d3e_1000           conda-forge                    \n",
            "    libblas:         3.8.0-7_openblas             conda-forge                    \n",
            "    libcblas:        3.8.0-7_openblas             conda-forge                    \n",
            "    libcudf:         0.7.0.dev0-cuda10.0_874      rapidsai-nightly/label/cuda10.0\n",
            "    libcudf_cffi:    0.7.0.dev0-py36_874          rapidsai-nightly/label/cuda10.0\n",
            "    libcuml:         0.7.0.dev0-cuda10.0_391      rapidsai-nightly/label/cuda10.0\n",
            "    libcumlmg:       0.0.0.dev0-cuda10.0_373      nvidia/label/cuda10.0          \n",
            "    libgfortran:     3.0.0-1                      conda-forge                    \n",
            "    liblapack:       3.8.0-7_openblas             conda-forge                    \n",
            "    libnvstrings:    0.7.0.dev0-cuda10.0_102      rapidsai-nightly/label/cuda10.0\n",
            "    libprotobuf:     3.6.1-hdbcaa40_1001          conda-forge                    \n",
            "    librmm:          0.7.0.dev0-cuda10.0_30       rapidsai-nightly/label/cuda10.0\n",
            "    llvmlite:        0.28.0-py36hdbcaa40_0        conda-forge                    \n",
            "    numba:           0.43.1-py36he1b5a44_0        conda-forge                    \n",
            "    numpy:           1.15.4-py36h8b7e671_1002     conda-forge                    \n",
            "    nvstrings:       0.7.0.dev0-py36_102          rapidsai-nightly/label/cuda10.0\n",
            "    openblas:        0.3.5-ha44fe06_0             conda-forge                    \n",
            "    pandas:          0.24.2-py36hf484d3e_0        conda-forge                    \n",
            "    parquet-cpp:     1.5.1-4                      conda-forge                    \n",
            "    pyarrow:         0.12.1-py36hbbcf98d_0        conda-forge                    \n",
            "    python-dateutil: 2.8.0-py_0                   conda-forge                    \n",
            "    pytz:            2019.1-py_0                  conda-forge                    \n",
            "    rmm:             0.7.0.dev0-py36_30           rapidsai-nightly/label/cuda10.0\n",
            "    thrift-cpp:      0.12.0-h0a07b25_1002         conda-forge                    \n",
            "\n",
            "The following packages will be UPDATED:\n",
            "\n",
            "    ca-certificates: 2018.03.07-0                                                 --> 2019.3.9-hecc5488_0  conda-forge\n",
            "    certifi:         2018.4.16-py36_0                                             --> 2019.3.9-py36_0      conda-forge\n",
            "    conda:           4.5.4-py36_0                                                 --> 4.6.14-py36_0        conda-forge\n",
            "    cryptography:    2.2.2-py36h14c3975_0                                         --> 2.6.1-py36h72c5cf5_0 conda-forge\n",
            "    libgcc-ng:       7.2.0-hdf63c60_3                                             --> 8.2.0-hdf63c60_1                \n",
            "    libstdcxx-ng:    7.2.0-hdf63c60_3                                             --> 8.2.0-hdf63c60_1                \n",
            "    openssl:         1.0.2o-h20670df_0                                            --> 1.1.1b-h14c3975_1    conda-forge\n",
            "    python:          3.6.5-hc3d631a_2                                             --> 3.6.7-h381d211_1004  conda-forge\n",
            "    sqlite:          3.23.1-he433501_0                                            --> 3.26.0-h67949de_1001 conda-forge\n",
            "    tk:              8.6.7-hc745277_3                                             --> 8.6.9-h84994c4_1001  conda-forge\n",
            "\n",
            "Preparing transaction: ...working... done\n",
            "Verifying transaction: ...working... done\n",
            "Executing transaction: ...working... done\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9oOCJ4NYMjY7",
        "colab_type": "text"
      },
      "source": [
        "# cuDF and cuML Examples #\n",
        "\n",
        "Now you can run code! \n",
        "\n",
        "What follows are basic examples where all processing takes place on the GPU."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V38dg-oUJtEO",
        "colab_type": "text"
      },
      "source": [
        "#[cuDF](https://github.com/rapidsai/cudf)#\n",
        "\n",
        "Load a dataset into a GPU memory resident DataFrame and perform a basic calculation.\n",
        "\n",
        "Everything from CSV parsing to calculating tip percentage and computing a grouped average is done on the GPU.\n",
        "\n",
        "_Note_: You must import nvstrings and nvcategory before cudf, else you'll get errors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EwaJSKuswsNi",
        "colab_type": "code",
        "outputId": "ba7cb330-1733-41d9-8c07-6e313c329160",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "source": [
        "import nvstrings, nvcategory, cudf\n",
        "import io, requests\n",
        "\n",
        "# download CSV file from GitHub\n",
        "url=\"https://github.com/plotly/datasets/raw/master/tips.csv\"\n",
        "content = requests.get(url).content.decode('utf-8')\n",
        "\n",
        "# read CSV from memory\n",
        "tips_df = cudf.read_csv(io.StringIO(content))\n",
        "tips_df['tip_percentage'] = tips_df['tip']/tips_df['total_bill']*100\n",
        "\n",
        "# display average tip by dining party size\n",
        "print(tips_df.groupby('size').tip_percentage.mean())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "size\n",
            "1     21.72920154872781\n",
            "2    16.571919173482893\n",
            "3    15.215685473711831\n",
            "4    14.594900639351332\n",
            "5    14.149548965142023\n",
            "6    15.622920072028379\n",
            "Name: tip_percentage, dtype: float64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ul3UZJdUJqlT",
        "colab_type": "text"
      },
      "source": [
        "#[cuML](https://github.com/rapidsai/cuml)#\n",
        "\n",
        "This snippet loads a \n",
        "\n",
        "As above, all calculations are performed on the GPU."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dCE8WhO3HpL_",
        "colab_type": "code",
        "outputId": "cb5a0ecf-7902-4c9a-8da3-d00356542f31",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "import cuml\n",
        "\n",
        "# Create and populate a GPU DataFrame\n",
        "df_float = cudf.DataFrame()\n",
        "df_float['0'] = [1.0, 2.0, 5.0]\n",
        "df_float['1'] = [4.0, 2.0, 1.0]\n",
        "df_float['2'] = [4.0, 2.0, 1.0]\n",
        "\n",
        "# Setup and fit clusters\n",
        "dbscan_float = cuml.DBSCAN(eps=1.0, min_samples=1)\n",
        "dbscan_float.fit(df_float)\n",
        "\n",
        "print(dbscan_float.labels_)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0    0\n",
            "1    1\n",
            "2    2\n",
            "dtype: int32\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dlsyk9m9NN2K",
        "colab_type": "text"
      },
      "source": [
        "# Next Steps #\n",
        "\n",
        "For an overview of how you can access and work with your own datasets in Colab, check out [this guide](https://towardsdatascience.com/3-ways-to-load-csv-files-into-colab-7c14fcbdcb92).\n",
        "\n",
        "For more RAPIDS examples, check out our RAPIDS notebooks repos:\n",
        "1. https://github.com/rapidsai/notebooks\n",
        "2. https://github.com/rapidsai/notebooks-extended"
      ]
    }
  ]
}