Prediksi_Penyakit_Stroke_Pada_Seseorang_Menggunakan_PySpark.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Prediksi Penyakit Stroke Pada Seseorang Menggunakan PySpark.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyPohsttnlGGuYXq6Sd6SXcW",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/alfianhid/Prediksi-Penyakit-Stroke-Pada-Seseorang-Menggunakan-PySpark/blob/main/Prediksi_Penyakit_Stroke_Pada_Seseorang_Menggunakan_PySpark.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mn2e5qcIiDrv"
      },
      "source": [
        "**Installing Dependencies & Initiating a New Spark Session**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AkgHv0jzgQp4",
        "outputId": "a96f7067-6821-4311-9245-b0f728d2f2f4"
      },
      "source": [
        "# Install PySpark\n",
        "!pip install pyspark"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting pyspark\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/89/db/e18cfd78e408de957821ec5ca56de1250645b05f8523d169803d8df35a64/pyspark-3.1.2.tar.gz (212.4MB)\n",
            "\u001b[K     |████████████████████████████████| 212.4MB 70kB/s \n",
            "\u001b[?25hCollecting py4j==0.10.9\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/9e/b6/6a4fb90cd235dc8e265a6a2067f2a2c99f0d91787f06aca4bcf7c23f3f80/py4j-0.10.9-py2.py3-none-any.whl (198kB)\n",
            "\u001b[K     |████████████████████████████████| 204kB 18.1MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: pyspark\n",
            "  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyspark: filename=pyspark-3.1.2-py2.py3-none-any.whl size=212880768 sha256=861e30a9542211a43e5eba1dd3c5f33efdb2a72692ee49410b70168a2c08eb01\n",
            "  Stored in directory: /root/.cache/pip/wheels/40/1b/2c/30f43be2627857ab80062bef1527c0128f7b4070b6b2d02139\n",
            "Successfully built pyspark\n",
            "Installing collected packages: py4j, pyspark\n",
            "Successfully installed py4j-0.10.9 pyspark-3.1.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d_v39sHtilkD"
      },
      "source": [
        "# Creating a SparkSession\n",
        "from pyspark.sql import SparkSession\n",
        "spark = SparkSession.builder.appName(\"Prediksi Penyakit Stroke Pada Seseorang Menggunakan PySpark\").getOrCreate()"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7e41NuXziva9"
      },
      "source": [
        "**Load & Explore dataset**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yM08WNuei1fY",
        "outputId": "a3a9d4c9-7b67-4b79-9258-e5b78b340959"
      },
      "source": [
        "# Loading the stroke dataset from Google Drive\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive/', force_remount=True)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive/\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8bF3upgHkfdQ"
      },
      "source": [
        "# Create a spark dataframe\n",
        "df = spark.read.csv('/content/drive/MyDrive/Colab Notebooks/datasets/healthcare-dataset-stroke-data.csv', header=True, inferSchema=True)"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N1uaAmNHlXbD",
        "outputId": "4e3e8bd6-d04b-4ad3-cb90-2a7d84e5e3f7"
      },
      "source": [
        "# Displaying the dataframe\n",
        "df.show()"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-----+------+----+------------+-------------+------------+-------------+--------------+-----------------+----+---------------+------+\n",
            "|   id|gender| age|hypertension|heart_disease|ever_married|    work_type|Residence_type|avg_glucose_level| bmi| smoking_status|stroke|\n",
            "+-----+------+----+------------+-------------+------------+-------------+--------------+-----------------+----+---------------+------+\n",
            "| 9046|  Male|67.0|           0|            1|         Yes|      Private|         Urban|           228.69|36.6|formerly smoked|     1|\n",
            "|51676|Female|61.0|           0|            0|         Yes|Self-employed|         Rural|           202.21| N/A|   never smoked|     1|\n",
            "|31112|  Male|80.0|           0|            1|         Yes|      Private|         Rural|           105.92|32.5|   never smoked|     1|\n",
            "|60182|Female|49.0|           0|            0|         Yes|      Private|         Urban|           171.23|34.4|         smokes|     1|\n",
            "| 1665|Female|79.0|           1|            0|         Yes|Self-employed|         Rural|           174.12|  24|   never smoked|     1|\n",
            "|56669|  Male|81.0|           0|            0|         Yes|      Private|         Urban|           186.21|  29|formerly smoked|     1|\n",
            "|53882|  Male|74.0|           1|            1|         Yes|      Private|         Rural|            70.09|27.4|   never smoked|     1|\n",
            "|10434|Female|69.0|           0|            0|          No|      Private|         Urban|            94.39|22.8|   never smoked|     1|\n",
            "|27419|Female|59.0|           0|            0|         Yes|      Private|         Rural|            76.15| N/A|        Unknown|     1|\n",
            "|60491|Female|78.0|           0|            0|         Yes|      Private|         Urban|            58.57|24.2|        Unknown|     1|\n",
            "|12109|Female|81.0|           1|            0|         Yes|      Private|         Rural|            80.43|29.7|   never smoked|     1|\n",
            "|12095|Female|61.0|           0|            1|         Yes|     Govt_job|         Rural|           120.46|36.8|         smokes|     1|\n",
            "|12175|Female|54.0|           0|            0|         Yes|      Private|         Urban|           104.51|27.3|         smokes|     1|\n",
            "| 8213|  Male|78.0|           0|            1|         Yes|      Private|         Urban|           219.84| N/A|        Unknown|     1|\n",
            "| 5317|Female|79.0|           0|            1|         Yes|      Private|         Urban|           214.09|28.2|   never smoked|     1|\n",
            "|58202|Female|50.0|           1|            0|         Yes|Self-employed|         Rural|           167.41|30.9|   never smoked|     1|\n",
            "|56112|  Male|64.0|           0|            1|         Yes|      Private|         Urban|           191.61|37.5|         smokes|     1|\n",
            "|34120|  Male|75.0|           1|            0|         Yes|      Private|         Urban|           221.29|25.8|         smokes|     1|\n",
            "|27458|Female|60.0|           0|            0|          No|      Private|         Urban|            89.22|37.8|   never smoked|     1|\n",
            "|25226|  Male|57.0|           0|            1|          No|     Govt_job|         Urban|           217.08| N/A|        Unknown|     1|\n",
            "+-----+------+----+------------+-------------+------------+-------------+--------------+-----------------+----+---------------+------+\n",
            "only showing top 20 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Vu4Ie3yrm2VS",
        "outputId": "0ff46275-c0d9-4e57-8181-38a12d1b5ada"
      },
      "source": [
        "# Printing the schema\n",
        "df.printSchema()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "root\n",
            " |-- id: integer (nullable = true)\n",
            " |-- gender: string (nullable = true)\n",
            " |-- age: double (nullable = true)\n",
            " |-- hypertension: integer (nullable = true)\n",
            " |-- heart_disease: integer (nullable = true)\n",
            " |-- ever_married: string (nullable = true)\n",
            " |-- work_type: string (nullable = true)\n",
            " |-- Residence_type: string (nullable = true)\n",
            " |-- avg_glucose_level: double (nullable = true)\n",
            " |-- bmi: string (nullable = true)\n",
            " |-- smoking_status: string (nullable = true)\n",
            " |-- stroke: integer (nullable = true)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ApUyq3Vqm4Mk",
        "outputId": "ef0309bd-86bd-48de-cf0e-8d3f4cf9f916"
      },
      "source": [
        "# Count the total of people with stroke and non-stroke class\n",
        "print((df.count(),len(df.columns)))\n",
        "df.groupBy('stroke').count().show()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(5100, 12)\n",
            "+------+-----+\n",
            "|stroke|count|\n",
            "+------+-----+\n",
            "|     1|  244|\n",
            "|     0| 4856|\n",
            "+------+-----+\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "99doIwXGnRoo",
        "outputId": "42e6e66f-9bd5-4769-c4ba-15e7eee730b3"
      },
      "source": [
        "# Displaying the summary of statistics\n",
        "df.describe().show()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-------+------------------+------+------------------+-------------------+-------------------+------------+---------+--------------+------------------+------------------+--------------+-------------------+\n",
            "|summary|                id|gender|               age|       hypertension|      heart_disease|ever_married|work_type|Residence_type| avg_glucose_level|               bmi|smoking_status|             stroke|\n",
            "+-------+------------------+------+------------------+-------------------+-------------------+------------+---------+--------------+------------------+------------------+--------------+-------------------+\n",
            "|  count|              5100|  5100|              5100|               5100|               5100|        5100|     5100|          5100|              5100|              5100|          5100|               5100|\n",
            "|   mean| 36520.33803921568|  null|43.223529411764716|0.09725490196078432|0.05411764705882353|        null|     null|          null|106.17687058823516|28.896040816326543|          null|0.04784313725490196|\n",
            "| stddev|21174.909841880577|  null|22.597164286369456|0.29633360315559665| 0.2262718858997944|        null|     null|          null| 45.30738331909153| 7.852847792430417|          null|0.21345516019705116|\n",
            "|    min|                67|Female|              0.08|                  0|                  0|          No| Govt_job|         Rural|             55.12|              10.3|       Unknown|                  0|\n",
            "|    max|             72940| Other|              82.0|                  1|                  1|         Yes| children|         Urban|            271.74|               N/A|        smokes|                  1|\n",
            "+-------+------------------+------+------------------+-------------------+-------------------+------------+---------+--------------+------------------+------------------+--------------+-------------------+\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8j2L5NEgnYJF"
      },
      "source": [
        "**Data Preparation**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EIeb3F8una7O",
        "outputId": "15c90df5-cb53-41f9-8ec4-43b4e6b57d58"
      },
      "source": [
        "# Checking for null values\n",
        "for col in df.columns:\n",
        "  print(col+\":\",df[df[col].isNull()].count())"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "id: 0\n",
            "gender: 0\n",
            "age: 0\n",
            "hypertension: 0\n",
            "heart_disease: 0\n",
            "ever_married: 0\n",
            "work_type: 0\n",
            "Residence_type: 0\n",
            "avg_glucose_level: 0\n",
            "bmi: 0\n",
            "smoking_status: 0\n",
            "stroke: 0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m22_rP5YnhLB"
      },
      "source": [
        "# Check for any unnecessary values\n",
        "def count_zeros():\n",
        "  columns_list = ['age', 'hypertension', 'heart_disease','avg_glucose_level','bmi']\n",
        "  for i in columns_list:\n",
        "    print(i+\":\",df[df[i]=='N/A'].count())"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4ByyabmZoAX7",
        "outputId": "ccb0e85c-92b4-4fbf-deae-e0a51ba1dd62"
      },
      "source": [
        "count_zeros()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "age: 0\n",
            "hypertension: 0\n",
            "heart_disease: 0\n",
            "avg_glucose_level: 0\n",
            "bmi: 200\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p7EAQAlAoCkS",
        "outputId": "641b9c09-b9c2-4b9c-cf61-10b901072844"
      },
      "source": [
        "# Calculate and replace the unnecessary values by the mean value of the respective column\n",
        "from pyspark.sql.functions import *\n",
        "for i in df.columns[9:10]:\n",
        "  data= df.agg({i:'mean'}).first()[0]\n",
        "  print(f'mean value for {i} is {int(data)}')\n",
        "  df=df.withColumn(i,when(df[i]=='N/A',int(data)).otherwise(df[i]))"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mean value for bmi is 28\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "numm69QtpCws",
        "outputId": "b26dd5aa-3d7a-4e33-d8b1-d5d598d12460"
      },
      "source": [
        "# Display the dataframe \n",
        "df.show()"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-----+------+----+------------+-------------+------------+-------------+--------------+-----------------+----+---------------+------+\n",
            "|   id|gender| age|hypertension|heart_disease|ever_married|    work_type|Residence_type|avg_glucose_level| bmi| smoking_status|stroke|\n",
            "+-----+------+----+------------+-------------+------------+-------------+--------------+-----------------+----+---------------+------+\n",
            "| 9046|  Male|67.0|           0|            1|         Yes|      Private|         Urban|           228.69|36.6|formerly smoked|     1|\n",
            "|51676|Female|61.0|           0|            0|         Yes|Self-employed|         Rural|           202.21|  28|   never smoked|     1|\n",
            "|31112|  Male|80.0|           0|            1|         Yes|      Private|         Rural|           105.92|32.5|   never smoked|     1|\n",
            "|60182|Female|49.0|           0|            0|         Yes|      Private|         Urban|           171.23|34.4|         smokes|     1|\n",
            "| 1665|Female|79.0|           1|            0|         Yes|Self-employed|         Rural|           174.12|  24|   never smoked|     1|\n",
            "|56669|  Male|81.0|           0|            0|         Yes|      Private|         Urban|           186.21|  29|formerly smoked|     1|\n",
            "|53882|  Male|74.0|           1|            1|         Yes|      Private|         Rural|            70.09|27.4|   never smoked|     1|\n",
            "|10434|Female|69.0|           0|            0|          No|      Private|         Urban|            94.39|22.8|   never smoked|     1|\n",
            "|27419|Female|59.0|           0|            0|         Yes|      Private|         Rural|            76.15|  28|        Unknown|     1|\n",
            "|60491|Female|78.0|           0|            0|         Yes|      Private|         Urban|            58.57|24.2|        Unknown|     1|\n",
            "|12109|Female|81.0|           1|            0|         Yes|      Private|         Rural|            80.43|29.7|   never smoked|     1|\n",
            "|12095|Female|61.0|           0|            1|         Yes|     Govt_job|         Rural|           120.46|36.8|         smokes|     1|\n",
            "|12175|Female|54.0|           0|            0|         Yes|      Private|         Urban|           104.51|27.3|         smokes|     1|\n",
            "| 8213|  Male|78.0|           0|            1|         Yes|      Private|         Urban|           219.84|  28|        Unknown|     1|\n",
            "| 5317|Female|79.0|           0|            1|         Yes|      Private|         Urban|           214.09|28.2|   never smoked|     1|\n",
            "|58202|Female|50.0|           1|            0|         Yes|Self-employed|         Rural|           167.41|30.9|   never smoked|     1|\n",
            "|56112|  Male|64.0|           0|            1|         Yes|      Private|         Urban|           191.61|37.5|         smokes|     1|\n",
            "|34120|  Male|75.0|           1|            0|         Yes|      Private|         Urban|           221.29|25.8|         smokes|     1|\n",
            "|27458|Female|60.0|           0|            0|          No|      Private|         Urban|            89.22|37.8|   never smoked|     1|\n",
            "|25226|  Male|57.0|           0|            1|          No|     Govt_job|         Urban|           217.08|  28|        Unknown|     1|\n",
            "+-----+------+----+------------+-------------+------------+-------------+--------------+-----------------+----+---------------+------+\n",
            "only showing top 20 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5BiOJR-B1dV8"
      },
      "source": [
        "**Performing Correlation Analysis & Feature Selection**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SMHINWDr9Ev8",
        "outputId": "aa5d79d4-5609-41a8-dc41-90c84c2c4dda"
      },
      "source": [
        "# Convert 'bmi' column from string to double\n",
        "from pyspark.sql.functions import col , column\n",
        "changed_df = df.withColumn(\"bmi\", col(\"bmi\").cast(\"double\"))\n",
        "changed_df.dtypes"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('id', 'int'),\n",
              " ('gender', 'string'),\n",
              " ('age', 'double'),\n",
              " ('hypertension', 'int'),\n",
              " ('heart_disease', 'int'),\n",
              " ('ever_married', 'string'),\n",
              " ('work_type', 'string'),\n",
              " ('Residence_type', 'string'),\n",
              " ('avg_glucose_level', 'double'),\n",
              " ('bmi', 'double'),\n",
              " ('smoking_status', 'string'),\n",
              " ('stroke', 'int')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lXZFu7q71inL",
        "outputId": "8fa06358-eb14-4e7f-91ca-e84582f6d563"
      },
      "source": [
        "# Find the correlation between the set of input & output variables\n",
        "columns_list = ['age', 'hypertension', 'heart_disease','avg_glucose_level','bmi']\n",
        "for i in columns_list:\n",
        "  print(f'Correlation to stroke for {i} is {df.stat.corr(\"stroke\",i)}')"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Correlation to stroke for age is 0.24487886380792556\n",
            "Correlation to stroke for hypertension is 0.1279557579344937\n",
            "Correlation to stroke for heart_disease is 0.13722522884862864\n",
            "Correlation to stroke for avg_glucose_level is 0.1360330038652827\n",
            "Correlation to stroke for bmi is 0.03603127327571626\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "upgptyq47B9h"
      },
      "source": [
        "# Feature selection\n",
        "from pyspark.ml.feature import VectorAssembler\n",
        "assembler= VectorAssembler(inputCols=['age', 'hypertension', 'heart_disease','avg_glucose_level','bmi'],outputCol='features')\n",
        "output_data= assembler.transform(df)"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cPzIA5IB_n1I",
        "outputId": "929e5a8c-70b6-40a9-fc25-6b820191e4c9"
      },
      "source": [
        "# Print the schema\n",
        "output_data.printSchema()"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "root\n",
            " |-- id: integer (nullable = true)\n",
            " |-- gender: string (nullable = true)\n",
            " |-- age: double (nullable = true)\n",
            " |-- hypertension: integer (nullable = true)\n",
            " |-- heart_disease: integer (nullable = true)\n",
            " |-- ever_married: string (nullable = true)\n",
            " |-- work_type: string (nullable = true)\n",
            " |-- Residence_type: string (nullable = true)\n",
            " |-- avg_glucose_level: double (nullable = true)\n",
            " |-- bmi: double (nullable = true)\n",
            " |-- smoking_status: string (nullable = true)\n",
            " |-- stroke: integer (nullable = true)\n",
            " |-- features: vector (nullable = true)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UacNC34f_so9",
        "outputId": "9948532b-2d38-4e87-98fb-a7ce175e9efc"
      },
      "source": [
        "# Display the dataframe\n",
        "output_data.show()"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-----+------+----+------------+-------------+------------+-------------+--------------+-----------------+----+---------------+------+--------------------+\n",
            "|   id|gender| age|hypertension|heart_disease|ever_married|    work_type|Residence_type|avg_glucose_level| bmi| smoking_status|stroke|            features|\n",
            "+-----+------+----+------------+-------------+------------+-------------+--------------+-----------------+----+---------------+------+--------------------+\n",
            "| 9046|  Male|67.0|           0|            1|         Yes|      Private|         Urban|           228.69|36.6|formerly smoked|     1|[67.0,0.0,1.0,228...|\n",
            "|51676|Female|61.0|           0|            0|         Yes|Self-employed|         Rural|           202.21|28.0|   never smoked|     1|[61.0,0.0,0.0,202...|\n",
            "|31112|  Male|80.0|           0|            1|         Yes|      Private|         Rural|           105.92|32.5|   never smoked|     1|[80.0,0.0,1.0,105...|\n",
            "|60182|Female|49.0|           0|            0|         Yes|      Private|         Urban|           171.23|34.4|         smokes|     1|[49.0,0.0,0.0,171...|\n",
            "| 1665|Female|79.0|           1|            0|         Yes|Self-employed|         Rural|           174.12|24.0|   never smoked|     1|[79.0,1.0,0.0,174...|\n",
            "|56669|  Male|81.0|           0|            0|         Yes|      Private|         Urban|           186.21|29.0|formerly smoked|     1|[81.0,0.0,0.0,186...|\n",
            "|53882|  Male|74.0|           1|            1|         Yes|      Private|         Rural|            70.09|27.4|   never smoked|     1|[74.0,1.0,1.0,70....|\n",
            "|10434|Female|69.0|           0|            0|          No|      Private|         Urban|            94.39|22.8|   never smoked|     1|[69.0,0.0,0.0,94....|\n",
            "|27419|Female|59.0|           0|            0|         Yes|      Private|         Rural|            76.15|28.0|        Unknown|     1|[59.0,0.0,0.0,76....|\n",
            "|60491|Female|78.0|           0|            0|         Yes|      Private|         Urban|            58.57|24.2|        Unknown|     1|[78.0,0.0,0.0,58....|\n",
            "|12109|Female|81.0|           1|            0|         Yes|      Private|         Rural|            80.43|29.7|   never smoked|     1|[81.0,1.0,0.0,80....|\n",
            "|12095|Female|61.0|           0|            1|         Yes|     Govt_job|         Rural|           120.46|36.8|         smokes|     1|[61.0,0.0,1.0,120...|\n",
            "|12175|Female|54.0|           0|            0|         Yes|      Private|         Urban|           104.51|27.3|         smokes|     1|[54.0,0.0,0.0,104...|\n",
            "| 8213|  Male|78.0|           0|            1|         Yes|      Private|         Urban|           219.84|28.0|        Unknown|     1|[78.0,0.0,1.0,219...|\n",
            "| 5317|Female|79.0|           0|            1|         Yes|      Private|         Urban|           214.09|28.2|   never smoked|     1|[79.0,0.0,1.0,214...|\n",
            "|58202|Female|50.0|           1|            0|         Yes|Self-employed|         Rural|           167.41|30.9|   never smoked|     1|[50.0,1.0,0.0,167...|\n",
            "|56112|  Male|64.0|           0|            1|         Yes|      Private|         Urban|           191.61|37.5|         smokes|     1|[64.0,0.0,1.0,191...|\n",
            "|34120|  Male|75.0|           1|            0|         Yes|      Private|         Urban|           221.29|25.8|         smokes|     1|[75.0,1.0,0.0,221...|\n",
            "|27458|Female|60.0|           0|            0|          No|      Private|         Urban|            89.22|37.8|   never smoked|     1|[60.0,0.0,0.0,89....|\n",
            "|25226|  Male|57.0|           0|            1|          No|     Govt_job|         Urban|           217.08|28.0|        Unknown|     1|[57.0,0.0,1.0,217...|\n",
            "+-----+------+----+------------+-------------+------------+-------------+--------------+-----------------+----+---------------+------+--------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LY57AdrI_ytc"
      },
      "source": [
        "**Split Dataset & Build the Model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-jrrV2eQ_0-8"
      },
      "source": [
        "# Create the final data\n",
        "from pyspark.ml.classification import LogisticRegression\n",
        "final_data = output_data.select('features','stroke')"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lMA-5JS6_8xT",
        "outputId": "d530a3e2-c3d4-4b1a-8ee7-5e7dc2e320fc"
      },
      "source": [
        "# Print schema of the final data\n",
        "final_data.printSchema()"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "root\n",
            " |-- features: vector (nullable = true)\n",
            " |-- stroke: integer (nullable = true)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0LTefWui__N4"
      },
      "source": [
        "# Split the dataset and build the model\n",
        "train, test = final_data.randomSplit([0.7,0.3])\n",
        "models = LogisticRegression(labelCol='stroke')\n",
        "model = models.fit(train)"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZZl70FVjAFK3",
        "outputId": "6ba88005-9179-4ee4-9113-e0044ee61076"
      },
      "source": [
        "# Summary of the model\n",
        "summary = model.summary\n",
        "summary.predictions.describe().show()"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-------+-------------------+----------+\n",
            "|summary|             stroke|prediction|\n",
            "+-------+-------------------+----------+\n",
            "|  count|               3528|      3528|\n",
            "|   mean|0.04195011337868481|       0.0|\n",
            "| stddev| 0.2005036069622285|       0.0|\n",
            "|    min|                0.0|       0.0|\n",
            "|    max|                1.0|       0.0|\n",
            "+-------+-------------------+----------+\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TquMOeQ6AN1S"
      },
      "source": [
        "**Evaluate and Save the Model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sopGZ0UEAQDh"
      },
      "source": [
        "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
        "predictions = model.evaluate(test)"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0vRJFIx_AU1K",
        "outputId": "2166d59e-24c7-4a5d-f517-fb018283a9fa"
      },
      "source": [
        "predictions.predictions.show(10)"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+--------------------+------+--------------------+--------------------+----------+\n",
            "|            features|stroke|       rawPrediction|         probability|prediction|\n",
            "+--------------------+------+--------------------+--------------------+----------+\n",
            "|[0.16,0.0,0.0,69....|     0|[7.83035802869214...|[0.99960267478563...|       0.0|\n",
            "|[0.16,0.0,0.0,109...|     0|[7.61723343465783...|[0.99950834102130...|       0.0|\n",
            "|[0.16,0.0,0.0,114...|     0|[7.55563853894242...|[0.99947712156257...|       0.0|\n",
            "|[0.32,0.0,0.0,55....|     0|[7.86035823882333...|[0.9996144130735,...|       0.0|\n",
            "|[0.32,0.0,0.0,73....|     0|[7.76664435379704...|[0.99957654694350...|       0.0|\n",
            "|[0.32,0.0,0.0,108...|     0|[7.55328674711002...|[0.99947589105945...|       0.0|\n",
            "|[0.32,0.0,0.0,127...|     0|[7.44291388900208...|[0.99941476629455...|       0.0|\n",
            "|[0.4,0.0,0.0,109....|     0|[7.59557899573624...|[0.99949758371969...|       0.0|\n",
            "|[0.56,0.0,0.0,57....|     0|[7.79003826250297...|[0.99958633414102...|       0.0|\n",
            "|[0.56,0.0,0.0,80....|     0|[7.69118386368193...|[0.99954337158604...|       0.0|\n",
            "+--------------------+------+--------------------+--------------------+----------+\n",
            "only showing top 10 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5ZC7BTB5AYGl",
        "outputId": "0acc8ee1-2c7d-4072-e1e9-3d25b8a5198d"
      },
      "source": [
        "evaluator = BinaryClassificationEvaluator(rawPredictionCol='rawPrediction', labelCol='stroke')\n",
        "evaluator.evaluate(model.transform(test))"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8276027551942203"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SWvZOY_TAiHB"
      },
      "source": [
        "# Save the model\n",
        "model.save(\"my_model\")"
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zPHxtmyeAoOI"
      },
      "source": [
        "# Load saved model back to the environment\n",
        "from pyspark.ml.classification import LogisticRegressionModel\n",
        "model = LogisticRegressionModel.load('my_model')"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IQQEASiYAqvY"
      },
      "source": [
        "**Prediction on New Data with the saved model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fgKl3tG3AtcM"
      },
      "source": [
        "# Create a new spark dataframe\n",
        "test_df = spark.read.csv(\"/content/drive/MyDrive/Colab Notebooks/datasets/new-healthcare-dataset-stroke-data.csv\", header= True, inferSchema=True)"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gfbduSbDA4BV",
        "outputId": "57c43875-05b6-49b5-ac7a-04ba6e2b46c7"
      },
      "source": [
        "# Print the schema\n",
        "test_df.printSchema()"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "root\n",
            " |-- id: integer (nullable = true)\n",
            " |-- gender: string (nullable = true)\n",
            " |-- age: integer (nullable = true)\n",
            " |-- hypertension: integer (nullable = true)\n",
            " |-- heart_disease: integer (nullable = true)\n",
            " |-- ever_married: string (nullable = true)\n",
            " |-- work_type: string (nullable = true)\n",
            " |-- Residence_type: string (nullable = true)\n",
            " |-- avg_glucose_level: double (nullable = true)\n",
            " |-- bmi: string (nullable = true)\n",
            " |-- smoking_status: string (nullable = true)\n",
            " |-- stroke: integer (nullable = true)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lI6V5usbBK3j",
        "outputId": "ab15645d-b4b8-41ac-9ab3-3c8b495ef621"
      },
      "source": [
        "# Convert 'bmi' column from string to double\n",
        "from pyspark.sql.functions import col , column\n",
        "test_df = test_df.withColumn(\"bmi\", col(\"bmi\").cast(\"double\"))\n",
        "test_df.dtypes"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('id', 'int'),\n",
              " ('gender', 'string'),\n",
              " ('age', 'int'),\n",
              " ('hypertension', 'int'),\n",
              " ('heart_disease', 'int'),\n",
              " ('ever_married', 'string'),\n",
              " ('work_type', 'string'),\n",
              " ('Residence_type', 'string'),\n",
              " ('avg_glucose_level', 'double'),\n",
              " ('bmi', 'double'),\n",
              " ('smoking_status', 'string'),\n",
              " ('stroke', 'int')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zuBUfOvpA671"
      },
      "source": [
        "# Create an additional feature, then merged the column \n",
        "test_data = assembler.transform(test_df)"
      ],
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gZwKbq5TBUf7",
        "outputId": "34da6462-caa5-46a4-82a3-50e1d9acaaeb"
      },
      "source": [
        "# Print the schema\n",
        "test_data.printSchema()"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "root\n",
            " |-- id: integer (nullable = true)\n",
            " |-- gender: string (nullable = true)\n",
            " |-- age: integer (nullable = true)\n",
            " |-- hypertension: integer (nullable = true)\n",
            " |-- heart_disease: integer (nullable = true)\n",
            " |-- ever_married: string (nullable = true)\n",
            " |-- work_type: string (nullable = true)\n",
            " |-- Residence_type: string (nullable = true)\n",
            " |-- avg_glucose_level: double (nullable = true)\n",
            " |-- bmi: double (nullable = true)\n",
            " |-- smoking_status: string (nullable = true)\n",
            " |-- stroke: integer (nullable = true)\n",
            " |-- features: vector (nullable = true)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GGYmTKjqBWTy",
        "outputId": "ecfe5e29-bf0c-4158-8ffc-cba9ffb8d517"
      },
      "source": [
        "# Use model to make predictions\n",
        "results = model.transform(test_data)\n",
        "results.printSchema()"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "root\n",
            " |-- id: integer (nullable = true)\n",
            " |-- gender: string (nullable = true)\n",
            " |-- age: integer (nullable = true)\n",
            " |-- hypertension: integer (nullable = true)\n",
            " |-- heart_disease: integer (nullable = true)\n",
            " |-- ever_married: string (nullable = true)\n",
            " |-- work_type: string (nullable = true)\n",
            " |-- Residence_type: string (nullable = true)\n",
            " |-- avg_glucose_level: double (nullable = true)\n",
            " |-- bmi: double (nullable = true)\n",
            " |-- smoking_status: string (nullable = true)\n",
            " |-- stroke: integer (nullable = true)\n",
            " |-- features: vector (nullable = true)\n",
            " |-- rawPrediction: vector (nullable = true)\n",
            " |-- probability: vector (nullable = true)\n",
            " |-- prediction: double (nullable = false)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0lWp51w4QHqg",
        "outputId": "7270e8ce-7578-4a6f-8805-535a47f1ba86"
      },
      "source": [
        "results.printSchema()"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "root\n",
            " |-- id: integer (nullable = true)\n",
            " |-- gender: string (nullable = true)\n",
            " |-- age: integer (nullable = true)\n",
            " |-- hypertension: integer (nullable = true)\n",
            " |-- heart_disease: integer (nullable = true)\n",
            " |-- ever_married: string (nullable = true)\n",
            " |-- work_type: string (nullable = true)\n",
            " |-- Residence_type: string (nullable = true)\n",
            " |-- avg_glucose_level: double (nullable = true)\n",
            " |-- bmi: double (nullable = true)\n",
            " |-- smoking_status: string (nullable = true)\n",
            " |-- stroke: integer (nullable = true)\n",
            " |-- features: vector (nullable = true)\n",
            " |-- rawPrediction: vector (nullable = true)\n",
            " |-- probability: vector (nullable = true)\n",
            " |-- prediction: double (nullable = false)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XLfLCKoPBaro",
        "outputId": "5c400a27-b7aa-4534-ba6a-dacc47e0c7d3"
      },
      "source": [
        "# Display the predictions\n",
        "results.show(2)"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-----+------+---+------------+-------------+------------+---------+--------------+-----------------+----+--------------+------+--------------------+--------------------+--------------------+----------+\n",
            "|   id|gender|age|hypertension|heart_disease|ever_married|work_type|Residence_type|avg_glucose_level| bmi|smoking_status|stroke|            features|       rawPrediction|         probability|prediction|\n",
            "+-----+------+---+------------+-------------+------------+---------+--------------+-----------------+----+--------------+------+--------------------+--------------------+--------------------+----------+\n",
            "|17739|  Male| 57|           0|            0|         Yes|  Private|         Rural|            84.96|36.7|       Unknown|     1|[57.0,0.0,0.0,84....|[3.38145191423093...|[0.96711980607462...|       0.0|\n",
            "|49669|Female| 14|           0|            0|          No| children|         Rural|            57.93|30.9|       Unknown|     1|[14.0,0.0,0.0,57....|[6.70597015655595...|[0.99877791027185...|       0.0|\n",
            "+-----+------+---+------------+-------------+------------+---------+--------------+-----------------+----+--------------+------+--------------------+--------------------+--------------------+----------+\n",
            "only showing top 2 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}