Personal_Loan_Campaign_Modelling_SALINAS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Personal Loan Campaign Modelling Project\n",
    "*by: Garey Salinas*\n",
    "\n",
    "\n",
    "## <span style=\"color:blue\">Description</span>\n",
    "### Background and Context\n",
    "AllLife Bank is a US bank that has a growing customer base. The majority of these customers are liability customers (depositors) with varying sizes of deposits. The number of customers who are also borrowers (asset customers) is quite small, and the bank is interested in expanding this base rapidly to bring in more loan business and in the process, earn more through the interest on loans. In particular, the management wants to explore ways of converting its liability customers to personal loan customers (while retaining them as depositors).\n",
    "\n",
    "A campaign that the bank ran last year for liability customers showed a healthy conversion rate of over 9% success. This has encouraged the retail marketing department to devise campaigns with better target marketing to increase the success ratio.\n",
    "\n",
    "You as a Data scientist at AllLife bank have to build a model that will help the marketing department to identify the potential customers who have a higher probability of purchasing the loan.\n",
    "\n",
    "\n",
    "### Objective\n",
    "\n",
    "1. To predict whether a liability customer will buy a personal loan or not.\n",
    "2. Which variables are most significant.\n",
    "3. Which segment of customers should be targeted more.\n",
    "\n",
    "### Data Dictionary\n",
    "LABELS | DESCRIPTION\n",
    "-------|:------------\n",
    "ID | Customer ID\n",
    "Age | Customer’s age in completed years\n",
    "Experience | #years of professional experience\n",
    "Income | Annual income of the customer (in thousand dollars)\n",
    "ZIP Code | Home Address ZIP code.\n",
    "Family | the Family size of the customer\n",
    "CCAvg | Average spending on credit cards per month (in thousand dollars)\n",
    "Education | Education Level. 1: Undergrad; 2: Graduate;3: Advanced/Professional\n",
    "Mortgage | Value of house mortgage if any. (in thousand dollars)\n",
    "Personal_Loan | Did this customer accept the personal loan offered in the last campaign?\n",
    "Securities_Account | Does the customer have securities account with the bank?\n",
    "CD_Account | Does the customer have a certificate of deposit (CD) account with the bank?\n",
    "Online | Do customers use internet banking facilities?\n",
    "CreditCard | Does the customer use a credit card issued by any other Bank (excluding All life Bank)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Import libraries and load dataset</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import scipy.stats as stats\n",
    "from sklearn import metrics, tree\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.metrics import (confusion_matrix, classification_report, \n",
    "                             accuracy_score, precision_score, recall_score, f1_score)\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")  # ignore warnings\n",
    "\n",
    "%matplotlib inline\n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"Loan_Modelling.csv\")\n",
    "df = data.copy()\n",
    "print(f\"There is {df.shape[0]} rows and {df.shape[1]} columns in this dataset.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Overview of Dataset</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.concat([df.head(10), df.tail(10)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Edit column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns = df.columns.str.lower()\n",
    "df.columns = df.columns.str.replace(\"creditcard\", \"credit_card\")\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observation**\n",
    "- All column names are lowercase\n",
    "- There are 5000 observations in this dataset.\n",
    "- All values are of a numerical type (int, float).\n",
    "- There are zero missing values in all columns. We will confirm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check for duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df.duplicated()].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Describe dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- `id` has 5000 unique values. We can drop this column.\n",
    "- We can change `family, education` to categorical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop(['id'], axis=1, inplace=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_features = ['family', 'education']\n",
    "\n",
    "for feature in cat_features:\n",
    "    df[feature] = pd.Categorical(df[feature])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe(include='all').T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- All columns have a count of 5000, meaning there are zero missing values in these columns.\n",
    "- There are 4 unique values in `family` and 3 unique values in the `education` column.\n",
    "- There are only 2 unique values in the `personal_loan, securities_account, cd_account, online and credit_card` columns.\n",
    "- `age` has a mean of 45 and a standard deviation of about 11.4. The min `age` is 23 and the max is 67. \n",
    "- `experience` has a mean of 20 and a standard deviation of 11.5. The min is -3 and the max is 43 years. We will inspect the negative value further.\n",
    "-`income` has a mean of 74K and a standard deviation of 46K. The values range from 8K to 224K.\n",
    "- `ccavg` has a mean of 1.93 and a standard deviation of 1.7. The values range from 0.0 to 10.0.\n",
    "- `mortgage` has a mean of 56.5K and a standard deviation of 101K. The standard deviation is greater than the mean. We will investigate further.\n",
    "- There are zero values in the `mortgage` column. We will inspect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.isnull().sum().sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.isnull().values.any() # If there are any null values in data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- Confirming dtype changed to categorical variables for the columns mentioned previously.\n",
    "- Confirming there are zero missing values. Not to be confused with values that are zero. We have alot of those in the `mortgage` column. Also, we will investigate the outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numerical_feature_df = df.select_dtypes(include=['int64','float64'])\n",
    "numerical_feature_df.skew()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- `income`, `ccavg` and `mortgage` are heavily skewed. We will investigate further."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Exploratory Data Analysis</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Univariate Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def histogram_boxplot(feature, figsize=(15, 7), bins=None):\n",
    "    \"\"\" \n",
    "    Boxplot and histogram combined\n",
    "    feature: 1-d feature array\n",
    "    figsize: size of fig (default (15,10))\n",
    "    bins: number of bins (default None / auto)\n",
    "    \"\"\"\n",
    "    f2, (ax_box2, ax_hist2) = plt.subplots(nrows = 2, # Number of rows of the subplot grid= 2\n",
    "                                           sharex = True, # x-axis will be shared among all subplots\n",
    "                                           gridspec_kw = {\"height_ratios\": (.25, .75)}, \n",
    "                                           figsize = figsize \n",
    "                                           ) # creating the 2 subplots\n",
    "    \n",
    "    sns.boxplot(feature, ax=ax_box2, showmeans=True, color='yellow') # boxplot will be created and a star will indicate the mean value of the column\n",
    "    sns.distplot(feature, kde=True, ax=ax_hist2, bins=bins) if bins else sns.distplot(feature, kde=True, ax=ax_hist2) # For histogram\n",
    "    ax_hist2.axvline(np.mean(feature), color='green', linestyle='--') # Add mean to the histogram\n",
    "    ax_hist2.axvline(np.median(feature), color='blue', linestyle='-');# Add median to the histogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_outliers(feature: str, data=df):\n",
    "    \"\"\" \n",
    "    Returns dataframe object of feature outliers.\n",
    "    feature: 1-d feature array\n",
    "    data: pandas dataframe (default is df)\n",
    "    \"\"\"\n",
    "    Q1 = data[feature].quantile(0.25)\n",
    "    Q3 = data[feature].quantile(0.75)\n",
    "    IQR = Q3 - Q1\n",
    "    #print(((df.Mileage < (Q1 - 1.5 * IQR)) | (df.Mileage > (Q3 + 1.5 * IQR))).sum())\n",
    "    return data[((data[feature] < (Q1 - 1.5 * IQR)) | (data[feature] > (Q3 + 1.5 * IQR)))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `age`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "histogram_boxplot(df.age)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- No outliers in the `age` column. The mean is near the median.\n",
    "- Average `age` is about 45 years old.\n",
    "- The `age` column distribution is uniform."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `income`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "histogram_boxplot(df.income)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- The average `income` is about 60K, with a median value of about 70K.\n",
    "- `income` column is right skewed and has many outliers to the upside."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `income` outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "outliers = create_outliers('income')\n",
    "outliers.sort_values(by='income', ascending=False).head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"There are {outliers.shape[0]} outliers.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `ccavg`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "histogram_boxplot(df.ccavg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- `ccavg` has  an average of about 1.5 and a median of about 2.\n",
    "- `ccavg` column is right skewed and has many outliers to the upside."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `ccavg` outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "outliers = create_outliers('ccavg')\n",
    "outliers.sort_values(by='ccavg', ascending=False).head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"There are {outliers.shape[0]} outliers.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `mortgage`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "histogram_boxplot(df.mortgage)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- `mortgage` has many values that aren't null but are equal to zero. We will dissect further.\n",
    "- `mortgage` column has many outliers to the upside."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `mortgage` outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "outliers = create_outliers('mortgage')\n",
    "outliers.sort_values(by='mortgage', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"There are {outliers.shape[0]} outliers in the outlier column.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check zero values in `mortgage` column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'There are {df[df.mortgage==0].shape[0]} rows where mortgage equals to ZERO!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check `zipcodes` frequency where `mortgage` equals zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 10))\n",
    "sns.countplot(y=df[df.mortgage==0]['zipcode'], \n",
    "              data=df, \n",
    "              order=df[df.mortgage==0]['zipcode'].value_counts().index[:40]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- The `zipcode` 94720 has the most frequent number of mortgages that equal zero with over 120 values.\n",
    "- The second highest number of zero values is 94305, and the third highest is 95616."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `experience`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "histogram_boxplot(df.experience)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- The `experience` column is uniform and has no outliers.\n",
    "- The average and median `experience` is about 20 years.\n",
    "- `experience` column is uniformly distributed. The mean is close to the median."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 10))\n",
    "sns.countplot(y=df.experience, \n",
    "              data=df, \n",
    "              order=df.experience.value_counts().index[:]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- 32 years is the greatest number of `experience` years observed with about 150 observations.\n",
    "- The plot shows negative values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"There are {df[df.experience<0].shape[0]} rows that have professional experience less than zero.\")\n",
    "df[df.experience<0].sort_values(by='experience', ascending=True).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Countplot for `experience` less than zero vs. `age`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 4))\n",
    "sns.countplot(y=df[df.experience<0]['age'], \n",
    "              data=df, \n",
    "              order=df[df.experience<0]['age'].value_counts().index[:]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- Most of the negative values are from the 25 year old `age` group with over 17.\n",
    "- This is a error in the data entry. You can't have negative years of `experience` so we will take the absolute value of the `experience`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Taking absolute values of the `experience` column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['abs_experience'] = np.abs(df.experience)\n",
    "df.sort_values(by='experience', ascending=True).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "histogram_boxplot(df.abs_experience)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- It didn't change the distribution that much."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 10))\n",
    "sns.countplot(y=df.abs_experience, \n",
    "              data=df, \n",
    "              order=df.abs_experience.value_counts().index[:]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- There are no more negative `experience` values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview on distributions of numerical columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lets plot histogram of all plots\n",
    "features = ['age', 'experience', 'income',\n",
    "            'ccavg', 'mortgage', 'zipcode',\n",
    "            'abs_experience']\n",
    "\n",
    "n_rows = math.ceil(len(features)/3)\n",
    "plt.figure(figsize=(15, n_rows*3.5))\n",
    "for i, feature in enumerate(list(features)):\n",
    "    plt.subplot(n_rows, 3, i+1)\n",
    "    plt.hist(df[feature])\n",
    "    plt.tight_layout()\n",
    "    plt.title(feature, fontsize=15);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview on the dispersion of numerical columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# outlier detection using boxplot\n",
    "plt.figure(figsize=(15, n_rows*4))\n",
    "for i, feature in enumerate(features):\n",
    "    plt.subplot(n_rows, 3, i+1)\n",
    "    plt.boxplot(df[feature], whis=1.5)\n",
    "    plt.tight_layout()\n",
    "    plt.title(feature, fontsize=15);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display value counts from categorical columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# looking at value counts for non-numeric features\n",
    "num_to_display = 10  # defining this up here so it's easy to change later if I want\n",
    "for colname in df.dtypes[df.dtypes=='category'].index:\n",
    "    val_counts = df[colname].value_counts(dropna=False)  # i want to see NA counts\n",
    "    print(f\"Column: {colname}\")\n",
    "    print(\"=\"*40)\n",
    "    print(val_counts[:num_to_display])\n",
    "    if len(val_counts) > num_to_display:\n",
    "        print(f\"Only displaying first {num_to_display} of {len(val_counts)} values.\")\n",
    "    print(\"\\n\") # just for more space between "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `zipcode`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 10))\n",
    "sns.countplot(y=\"zipcode\", data=df, order=df.zipcode.value_counts().index[0:50]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- Most of the values come from the `zipcode` 94720 with over 160."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def perc_on_bar(plot, feature):\n",
    "    \"\"\"\n",
    "    Shows the percentage on the top of bar in plot.\n",
    "    feature: categorical feature\n",
    "    The function won't work if a column is passed in hue parameter\n",
    "    \"\"\"\n",
    "    total = len(feature) # length of the column\n",
    "    for p in ax.patches:\n",
    "        # percentage = '{:.1f}%'.format(100 * p.get_height()/total) # percentage of each class of the category\n",
    "        percentage = 100 * p.get_height()/total\n",
    "        percentage_label = f\"{percentage:.1f}%\"\n",
    "        x = p.get_x() + p.get_width() / 2 - 0.05 # width of the plot\n",
    "        y = p.get_y() + p.get_height()           # hieght of the plot\n",
    "        ax.annotate(percentage_label, (x, y), size = 12) # annotate the percantage \n",
    "    plt.show() # show the plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `family`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 7))\n",
    "ax = sns.countplot(df.family, palette='mako')\n",
    "perc_on_bar(ax, df.family)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- The largest category of the `family` column is 1 with a percentage of 29.4%.\n",
    "- The second largest category of the `family` column is a size of 2, then 4. A size of 3 is the smallest portion in our dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `education`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 7))\n",
    "ax = sns.countplot(df.education, palette='mako')\n",
    "perc_on_bar(ax, df.education)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- The `education` column has 3 categories.\n",
    "- Category 1 (undergrad) hold the greatest proportion with 41.9%.\n",
    "- Category 3 holds the second highest with 30%.\n",
    "- Category 2 holds the third highest proportion with 28.1%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Oberservations on `personal_loan`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 7))\n",
    "ax = sns.countplot(df.personal_loan, palette='mako')\n",
    "perc_on_bar(ax, df.personal_loan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- Those that didn't accept a `personal_loan` from the last campaign make up the greatest percentage with 90.4%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `securities_account`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15,7))\n",
    "ax = sns.countplot(df.securities_account, palette='mako')\n",
    "perc_on_bar(ax, df.securities_account)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- Those customers without a `securities_account` make up the greatest proportion with 89.6%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `cd_account`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 7))\n",
    "ax = sns.countplot(df.cd_account, palette='mako')\n",
    "perc_on_bar(ax, df.cd_account)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- Those customers without a `cd_account` make up the greatest percentage with 94%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `online`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 7))\n",
    "ax = sns.countplot(df.online, palette='mako')\n",
    "perc_on_bar(ax, df.online)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- Those customers that use `online` banking facilities makes up the majority with 59.7%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations on `credit_card`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 7))\n",
    "ax = sns.countplot(df.credit_card, palette='mako')\n",
    "perc_on_bar(ax, df.credit_card)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- Those customers that don't use `credit_cards` issued by other banks makes up the majority with 70.6%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bivariate Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Function to plot stacked bar chart\n",
    "def stacked_plot(x, y):\n",
    "    \"\"\"\n",
    "    Shows stacked plot from x and y pandas data series\n",
    "    x: pandas data series\n",
    "    y: pandas data series\n",
    "    \"\"\"\n",
    "    info = pd.crosstab(x, y, margins=True)\n",
    "    info['% - 0'] = round(info[0]/info['All']*100, 2)\n",
    "    info['% - 1'] = round(info[1]/info['All']*100, 2)\n",
    "    print(info)\n",
    "    print('='*80)\n",
    "    visual = pd.crosstab(x, y, normalize='index')\n",
    "    visual.plot(kind='bar', stacked=True, figsize=(10,5));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_boxplots(cols: list, feature: str, show_fliers=True, data=df): #method call to show bloxplots\n",
    "    n_rows = math.ceil(len(cols)/2)\n",
    "    plt.figure(figsize=(15, n_rows*5))\n",
    "    for i, variable in enumerate(cols):\n",
    "        plt.subplot(n_rows, 2, i+1)\n",
    "        if show_fliers:\n",
    "            sns.boxplot(data[feature], data[variable], palette=\"mako\", showfliers=True) \n",
    "        else:\n",
    "            sns.boxplot(data[feature], data[variable], palette=\"mako\", showfliers=False)\n",
    "        plt.tight_layout()\n",
    "        plt.title(variable, fontsize=12)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlation and heatmap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12, 7))\n",
    "sns.heatmap(df.corr(), annot=True, cmap=\"coolwarm\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- `age` and `experience` are heavily positively correlated.\n",
    "- `ccavg` and `income` are positively correlated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(data=df[['age','income','zipcode','ccavg',\n",
    "                      'mortgage','abs_experience','personal_loan']], \n",
    "             hue='personal_loan');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- Plot show that income is higher among those customers with personal loans.\n",
    "- ccavg is higher among those customers with personal loans. we will investigate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols = ['age','income','ccavg','mortgage','abs_experience']\n",
    "show_boxplots(cols, 'personal_loan')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show without outliers in boxplots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_boxplots(cols, 'personal_loan', show_fliers=False);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- On average, those customers with higher incomes have personal loans.\n",
    "- On average, those customers with higher credit card usage have personal loans.\n",
    "- 75% of those customers with personal loans have a mortgage payments of 500K or less. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `personal_loan` vs `family`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacked_plot(df.family, df.personal_loan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations** \n",
    "- Those customers with a `family` of 4 have more `personal loans`. \n",
    "- A family of 3 have the second most personal loans followed by a family of 1 and 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `personal_loan` vs `education`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacked_plot(df.education, df.personal_loan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- Those customers with an education of '2' and '3' hold a greater percentage of personal loans that those customer with an education of '1'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `personal_loan` vs `secuities_account`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacked_plot(df.securities_account, df.personal_loan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- There is not much difference in securities account versus personal loans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `personal_loan` vs `cd_account`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacked_plot(df.cd_account, df.personal_loan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- Those customers with cd accounts. have a greater percentage of personal loans than those customer without a cd account. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `personal_loan` vs `online`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacked_plot(df.online, df.personal_loan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- There isnt much difference between customers who use online facilities and those who don't versus personal loans."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `personal_loan` vs `credit_card`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacked_plot(df.credit_card, df.personal_loan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- There isn't much difference between those who have credit cards from other banks versus personal loans."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `cd_account` vs `family`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacked_plot(df.family, df.cd_account)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- A family of 3 has the greatest percentage(8.12) of customers with cd accounts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `cd_account` vs `education`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacked_plot(df.education, df.cd_account)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- There isnt much of a difference between education categories."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `cd_account` vs `securities_account`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacked_plot(df.securities_account, df.cd_account)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- A greater percentage of those customers with security accounts also have cd accounts versus those customer that dont have security accounts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `cd_account` vs `online`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacked_plot(df.online, df.cd_account)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- Customers who use the online facilities have a greater percentage cd accounts than those customer who don't use online facilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `cd_account` vs `credit_card`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacked_plot(df.credit_card, df.cd_account)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- A greater percentage of those customers who have credit cards with other bank institutions have personal cd accounts than those customers who dont have credit cards from other institutions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let us check which of these differences are statistically significant.\n",
    "The Chi-Square test is a statistical method to determine if two categorical variables have a significant correlation between them.\n",
    "  \n",
    "**$H_0$:**  There is no association between the two variables.      \n",
    "**$H_a$:**  There is an association between two variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_significance(feature1: str, feature2: str, data=df):\n",
    "    \"\"\"\n",
    "    Checks the significance of feature1 agaisnt feature2\n",
    "    feature1: column name\n",
    "    feature2: column name\n",
    "    data: pandas dataframe object (defaults to df)\n",
    "    \"\"\"\n",
    "    crosstab = pd.crosstab(data[feature1], data[feature2])  # Contingency table of region and smoker attributes\n",
    "    chi, p_value, dof, expected =  stats.chi2_contingency(crosstab)\n",
    "    Ho = f\"{feature1} has no effect on {feature2}\"   # Stating the Null Hypothesis\n",
    "    Ha = f\"{feature1} has an effect on {feature2}\"   # Stating the Alternate Hypothesis\n",
    "    if p_value < 0.05:  # Setting our significance level at 5%\n",
    "        print(f'{Ha.upper()} as the p_value ({p_value.round(3)}) < 0.05')\n",
    "    else:\n",
    "        print(f'{Ho} as the p_value ({p_value.round(3)}) > 0.05')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_significance(features: list, data=df):\n",
    "    \"\"\"\n",
    "    Prints out the significance of all the list of features passed.\n",
    "    features: list of column names\n",
    "    data: pandas dataframe object (defaults to df)\n",
    "    \"\"\"\n",
    "    for feature in features:\n",
    "        print(\"=\"*30, feature, \"=\"*(50-len(feature)))\n",
    "        for col in list(data.columns):\n",
    "            if col != feature: check_significance(col , feature)\n",
    "                \n",
    "show_significance(['personal_loan', 'cd_account'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Key Observations - \n",
    "* `cd_account`, `family` and `education` seem to be strong indicators of customers received a personal loan.\n",
    "* `securities_account`, `online` and `credit_card` seem to be strong indicators of customers who have cd accounts.\n",
    "* Other factors appear to be not very good indicators of those customers that have cd accounts. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\"> Build Model, Train and Evaluate</span>\n",
    "1. Data preparation\n",
    "2. Partition the data into train and test set.\n",
    "3. Build a CART model on the train data.\n",
    "4. Tune the model and prune the tree, if required.\n",
    "5. Test the data on test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    df.drop(['experience'], axis=1, inplace=True)\n",
    "except KeyError:\n",
    "    print(f\"Column experience must already be dropped.\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_dummies = pd.get_dummies(df, columns=['education', 'family'], drop_first=True)\n",
    "df_dummies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_dummies.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Partition Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df_dummies.drop(['personal_loan'], axis=1)\n",
    "X.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df_dummies['personal_loan']\n",
    "y.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splitting data into training and test set:\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1)\n",
    "print(\"The shape of X_train: \", X_train.shape)\n",
    "print(\"The shape of X_test: \", X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build Initial Decision Tree Model\n",
    "* We will build our model using the DecisionTreeClassifier function. Using default 'gini' criteria to split. \n",
    "* If the frequency of class A is 10% and the frequency of class B is 90%, then class B will become the dominant class and the decision tree will become biased toward the dominant classes.\n",
    "\n",
    "* In this case, we can pass a dictionary {0:0.15,1:0.85} to the model to specify the weight of each class and the decision tree will give more weightage to class 1.\n",
    "\n",
    "* class_weight is a hyperparameter for the decision tree classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = DecisionTreeClassifier(criterion='gini', \n",
    "                               class_weight={0:0.15, 1:0.85}, \n",
    "                               random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Function to create confusion matrix\n",
    "def make_confusion_matrix(model,  y_actual, labels=[1, 0], xtest=X_test):\n",
    "    \"\"\"\n",
    "    model : classifier to predict values of X\n",
    "    y_actual : ground truth  \n",
    "    \"\"\"\n",
    "    y_predict = model.predict(xtest)\n",
    "    cm = metrics.confusion_matrix(y_actual, y_predict, labels=[0, 1])\n",
    "    df_cm = pd.DataFrame(cm, index=[\"Actual - No\",\"Actual - Yes\"], \n",
    "                         columns=['Predicted - No','Predicted - Yes'])\n",
    "    #print(df_cm)\n",
    "    #print(\"=\"*80)\n",
    "    \n",
    "    group_counts = [f\"{value:0.0f}\" for value in cm.flatten()]   \n",
    "    group_percentages = [f\"{value:.2%}\" for value in cm.flatten()/np.sum(cm)]\n",
    "    \n",
    "    labels = [f\"{gc}\\n{gp}\" for gc, gp in zip(group_counts, group_percentages)]  \n",
    "    labels = np.asarray(labels).reshape(2,2)\n",
    "    \n",
    "    plt.figure(figsize = (10, 7))\n",
    "    sns.heatmap(df_cm, annot=labels, fmt='')\n",
    "    plt.ylabel('True label', fontsize=14)\n",
    "    plt.xlabel('Predicted label', fontsize=14);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "make_confusion_matrix(model, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "- We only have ~10% of positive classes, so if our model marks each sample as negative, then also we'll get 90% accuracy, hence accuracy is not a good metric to evaluate here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##  Function to calculate recall score\n",
    "def get_recall_score(model):\n",
    "    '''\n",
    "    Prints the recall score from model\n",
    "    model : classifier to predict values of X\n",
    "    '''\n",
    "    pred_train = model.predict(X_train)\n",
    "    pred_test = model.predict(X_test)\n",
    "    print(\"Recall on training set : \", metrics.recall_score(y_train, pred_train))\n",
    "    print(\"Recall on test set : \", metrics.recall_score(y_test, pred_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recall score from baseline model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recall on train and test\n",
    "get_recall_score(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the decision tree from baseline model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_names = list(X.columns)\n",
    "print(feature_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 30))\n",
    "out = tree.plot_tree(model,\n",
    "                     feature_names=feature_names,\n",
    "                     filled=True,\n",
    "                     fontsize=9,\n",
    "                     node_ids=False,\n",
    "                     class_names=None,)\n",
    "\n",
    "#below code will add arrows to the decision tree split if they are missing\n",
    "for o in out:\n",
    "     arrow = o.arrow_patch\n",
    "     if arrow is not None:\n",
    "        arrow.set_edgecolor('black')\n",
    "        arrow.set_linewidth(1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Text report showing the rules of a decision tree -\n",
    "print(tree.export_text(model,feature_names=feature_names,show_weights=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature importance from baseline model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def importance_plot(model):\n",
    "    \"\"\"\n",
    "    Displays feature importance barplot\n",
    "    model: decision tree classifier\n",
    "    \"\"\"\n",
    "    importances = model.feature_importances_\n",
    "    indices = np.argsort(importances)\n",
    "    size = len(indices)//2 # to help scale the plot.\n",
    "    \n",
    "    plt.figure(figsize=(10, size))\n",
    "    plt.title(\"Feature Importances\", fontsize=14)\n",
    "    plt.barh(range(len(indices)), importances[indices], color='blue', align='center')\n",
    "    plt.yticks(range(len(indices)), [feature_names[i] for i in indices])\n",
    "    plt.xlabel(\"Relative Importance\", fontsize=12);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importance_plot(model=model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importance of features in the tree building ( The importance of a feature is computed as the \n",
    "#(normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance )\n",
    "pd.DataFrame(model.feature_importances_, \n",
    "                   columns=[\"Imp\"], \n",
    "                   index=X_train.columns).sort_values(by='Imp', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using GridSearch for hyperparameter tuning of our tree model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose the type of classifier. \n",
    "estimator = DecisionTreeClassifier(random_state=1, class_weight={0:.15,1:.85})\n",
    "\n",
    "# Grid of parameters to choose from\n",
    "parameters = {'max_depth': np.arange(1,10),\n",
    "              'criterion': ['entropy','gini'],\n",
    "              'splitter': ['best','random'],\n",
    "              'min_impurity_decrease': [0.000001,0.00001,0.0001],\n",
    "              'max_features': ['log2','sqrt']}\n",
    "\n",
    "# Type of scoring used to compare parameter combinations\n",
    "scorer = metrics.make_scorer(metrics.recall_score)\n",
    "\n",
    "# Run the grid search\n",
    "grid_obj = GridSearchCV(estimator, param_grid=parameters, scoring=scorer, cv=5)\n",
    "grid_obj = grid_obj.fit(X_train, y_train)\n",
    "\n",
    "# Set the clf to the best combination of parameters\n",
    "estimator = grid_obj.best_estimator_\n",
    "\n",
    "# Fit the best algorithm to the data. \n",
    "estimator.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion matrix using GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "make_confusion_matrix(estimator, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recall score using GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_recall_score(estimator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the decision tree from the best fit estimator using GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 10))\n",
    "out = tree.plot_tree(estimator,\n",
    "                     feature_names=feature_names,\n",
    "                     filled=True,\n",
    "                     fontsize=10,\n",
    "                     node_ids=True,\n",
    "                     class_names=None)\n",
    "for o in out:\n",
    "    arrow = o.arrow_patch\n",
    "    if arrow is not None:\n",
    "        arrow.set_edgecolor('black')\n",
    "        arrow.set_linewidth(1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Text report showing the rules of a decision tree -\n",
    "print(tree.export_text(estimator,\n",
    "                       feature_names=feature_names,\n",
    "                       show_weights=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature importance using GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importance of features in the tree building ( The importance of a feature is computed as the \n",
    "#(normalized) total reduction of the 'criterion' brought by that feature. It is also known as the Gini importance )\n",
    "pd.DataFrame(estimator.feature_importances_, \n",
    "             columns=[\"Imp\"],\n",
    "             index=X_train.columns).sort_values(by='Imp', ascending=False)\n",
    "#Here we will see that importance of features has increased"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importance_plot(model=estimator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cost Complexity Pruning\n",
    "The `DecisionTreeClassifier` provides parameters such as\n",
    "``min_samples_leaf`` and ``max_depth`` to prevent a tree from overfiting. Cost\n",
    "complexity pruning provides another option to control the size of a tree. In\n",
    "`DecisionTreeClassifier`, this pruning technique is parameterized by the\n",
    "cost complexity parameter, ``ccp_alpha``. Greater values of ``ccp_alpha``\n",
    "increase the number of nodes pruned. Here we only show the effect of\n",
    "``ccp_alpha`` on regularizing the trees and how to choose a ``ccp_alpha``\n",
    "based on validation scores.\n",
    "\n",
    "### Total impurity of leaves vs effective alphas of pruned tree\n",
    "\n",
    "Minimal cost complexity pruning recursively finds the node with the \"weakest\n",
    "link\". The weakest link is characterized by an effective alpha, where the\n",
    "nodes with the smallest effective alpha are pruned first. To get an idea of\n",
    "what values of ``ccp_alpha`` could be appropriate, scikit-learn provides\n",
    "`DecisionTreeClassifier.cost_complexity_pruning_path` that returns the\n",
    "effective alphas and the corresponding total leaf impurities at each step of\n",
    "the pruning process. As alpha increases, more of the tree is pruned, which\n",
    "increases the total impurity of its leaves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = DecisionTreeClassifier(random_state=1, class_weight = {0:0.15, 1:0.85})\n",
    "path = clf.cost_complexity_pruning_path(X_train, y_train)\n",
    "ccp_alphas, impurities = path.ccp_alphas, path.impurities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15, 7))\n",
    "ax.plot(ccp_alphas[:-1], impurities[:-1], marker='o', drawstyle=\"steps-post\")\n",
    "ax.set_xlabel(\"Effective alpha\")\n",
    "ax.set_ylabel(\"Total impurity of leaves\")\n",
    "ax.set_title(\"Total Impurity vs effective alpha for training set\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clfs = []\n",
    "for ccp_alpha in ccp_alphas:\n",
    "    clf = DecisionTreeClassifier(random_state=1, \n",
    "                                 ccp_alpha=ccp_alpha,\n",
    "                                 class_weight = {0:0.15,1:0.85})\n",
    "    clf.fit(X_train, y_train)\n",
    "    clfs.append(clf)\n",
    "    \n",
    "print(f\"Number of nodes in the last tree is: {clfs[-1].tree_.node_count} with ccp_alpha: {ccp_alphas[-1]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clfs = clfs[:-1]\n",
    "ccp_alphas = ccp_alphas[:-1]\n",
    "\n",
    "node_counts = [clf.tree_.node_count for clf in clfs]\n",
    "depth = [clf.tree_.max_depth for clf in clfs]\n",
    "fig, ax = plt.subplots(2, 1, figsize=(15, 10), sharex=True)\n",
    "ax[0].plot(ccp_alphas, node_counts, marker='o', drawstyle=\"steps-post\")\n",
    "ax[0].set_ylabel(\"Number of nodes\")\n",
    "ax[0].set_title(\"Number of nodes vs alpha\")\n",
    "ax[1].plot(ccp_alphas, depth, marker='o', drawstyle=\"steps-post\")\n",
    "ax[1].set_xlabel(\"alpha\")\n",
    "ax[1].set_ylabel(\"depth of tree\")\n",
    "ax[1].set_title(\"Depth vs alpha\")\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "recall_train = []\n",
    "for clf in clfs:\n",
    "    pred_train3 = clf.predict(X_train)\n",
    "    values_train = metrics.recall_score(y_train, pred_train3)\n",
    "    recall_train.append(values_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "recall_test = []\n",
    "for clf in clfs:\n",
    "    pred_test3 = clf.predict(X_test)\n",
    "    values_test = metrics.recall_score(y_test, pred_test3)\n",
    "    recall_test.append(values_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_scores = [clf.score(X_train, y_train) for clf in clfs]\n",
    "test_scores = [clf.score(X_test, y_test) for clf in clfs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15, 7))\n",
    "ax.set_xlabel(\"alpha\")\n",
    "ax.set_ylabel(\"Recall\")\n",
    "ax.set_title(\"Recall vs alpha for training and testing sets\")\n",
    "ax.plot(ccp_alphas, \n",
    "        recall_train, \n",
    "        marker='o',\n",
    "        label=\"train\",\n",
    "        drawstyle=\"steps-post\",)\n",
    "ax.plot(ccp_alphas,\n",
    "        recall_test, \n",
    "        marker='o',\n",
    "        label=\"test\",\n",
    "        drawstyle=\"steps-post\")\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating the model where we get highest train and test recall\n",
    "index_best_model = np.argmax(recall_test)\n",
    "best_model = clfs[index_best_model]\n",
    "print(best_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "make_confusion_matrix(best_model, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_recall_score(best_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 8))\n",
    "\n",
    "out = tree.plot_tree(best_model,\n",
    "                     feature_names=feature_names,\n",
    "                     filled=True,\n",
    "                     fontsize=12,\n",
    "                     node_ids=True,\n",
    "                     class_names=None)\n",
    "for o in out:\n",
    "    arrow = o.arrow_patch\n",
    "    if arrow is not None:\n",
    "        arrow.set_edgecolor('black')\n",
    "        arrow.set_linewidth(1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Text report showing the rules of a decision tree -\n",
    "print(tree.export_text(best_model, feature_names=feature_names, show_weights=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importance_plot(model=best_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_model2 = DecisionTreeClassifier(ccp_alpha=0.01, \n",
    "                                     class_weight={0: 0.15, 1: 0.85}, \n",
    "                                     random_state=1)\n",
    "best_model2.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "make_confusion_matrix(best_model2, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_recall_score(best_model2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 8))\n",
    "\n",
    "out = tree.plot_tree(best_model2,\n",
    "                     feature_names=feature_names,\n",
    "                     filled=True,\n",
    "                     fontsize=12,\n",
    "                     node_ids=True,\n",
    "                     class_names=None)\n",
    "for o in out:\n",
    "    arrow = o.arrow_patch\n",
    "    if arrow is not None:\n",
    "        arrow.set_edgecolor('black')\n",
    "        arrow.set_linewidth(1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(tree.export_text(best_model2, feature_names=feature_names, show_weights=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importance_plot(model=best_model2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparison_frame = pd.DataFrame({'Model':['Initial decision tree model','Decision treee with hyperparameter tuning',\n",
    "                                          'Decision tree with post-pruning'], \n",
    "                                 'Train_Recall':[1, 0.95, 0.99], \n",
    "                                 'Test_Recall':[0.91, 0.91, 0.98]}) \n",
    "comparison_frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Decision tree model with post pruning has given the best recall score on data.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Conclusion</span>\n",
    "\n",
    "- I analyzed the \"Potential Loan marketing data\" using different techniques and used a Decision Tree Classifier to build a predictive model. The predictive model helps predict whether a liability customer will buy a personal loan or not.\n",
    "- Income, education, family, and credit card usage are the most important features in predicting potential loan customers.\n",
    "- Those customers with separate securities and cd accounts are more likely to get a personal loan. Customers who use the bank's online facilities are more likely to get a personal loan versus those customers who don't use the online facilities.\n",
    "- We established the importance of hyper-parameters/pruning to reduce overfitting during the model selection process.\n",
    "\n",
    "## <span style=\"color:blue\">Recommendations</span>\n",
    "- From the decision tree model, income is the most important feature. If our customer's yearly income is less than 98.5K, there is a good chance the customer won't have a personal loan. \n",
    "- From the model, those customers with an income greater than 98.5 and with an education level greater than or equal to 3 (Advanced/Professional) were most likely to have a personal loan. Recommend to target customers that have incomes lower than 98K.\n",
    "- It was observed that those customers who use the online facilities were more likely to have personal loans. Make the site more user-friendly and encourage those customers who don't use the facilities to use the online facilities. Make the application process to get personal loans easy with a better user experience."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}