Cheat Sheet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f8d233e8",
   "metadata": {},
   "source": [
    "# Data Science Cheat Sheet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca4c11c9",
   "metadata": {},
   "source": [
    "Exploratory Data Analytics\n",
    "\n",
    "1. Descriptive Analytics\n",
    "2. Diagnostis Analytics\n",
    "3. Predictive Analytics\n",
    "4. Prescriptive Analytics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bced19b",
   "metadata": {},
   "source": [
    "Categorical Variables:\n",
    "    1. Nominal - Male/Female\n",
    "    2. Ordinal - Bachelor/Master\n",
    "    \n",
    "Numerical Variables:\n",
    "    1. Discrete - #of goals\n",
    "    2. Continuous - height, weight"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de43c225",
   "metadata": {},
   "source": [
    "Statistics Subfields:\n",
    "\n",
    "    1. Descriptive Statistics\n",
    "    2. Inference\n",
    "    3. Risk and Probability\n",
    "    4. Correlation and Relationships\n",
    "    5. Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44c27bc6",
   "metadata": {},
   "source": [
    "## Import Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e171dad",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "188e5462",
   "metadata": {},
   "outputs": [],
   "source": [
    "# df = pd.read_excel(\"file_name.xlsx\")\n",
    "# df = pd.read_stata(\"file_name.dta\")\n",
    "# df = pd.read_sas(\"file_name.sas7bdat\")\n",
    "# df = pd.read_hdf(\"file_name.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8df9a61b",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"file_name.csv\")\n",
    "df = data.copy()\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fb60779",
   "metadata": {},
   "source": [
    "## EDA - Exploratory Data Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6c4119d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a930cd28",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f19f9bd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Showing number of null values\n",
    "\n",
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a902dcde",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualizing null values\n",
    "import seaborn as sns\n",
    "plt.figure(figsize=(16, 8))\n",
    "sns.heatmap(df.isnull(),yticklabels=False,cbar=False,cmap='Dark2_r')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19e2bb7e",
   "metadata": {},
   "source": [
    "Show count of unique values for each column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2db3441",
   "metadata": {},
   "outputs": [],
   "source": [
    "for column in df.columns:\n",
    "    print(column,'\\t', df[column].nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f0bebb4",
   "metadata": {},
   "source": [
    "Checking Imbalance (for classification model target variable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c3dc7c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data.target.value_counts())\n",
    "\n",
    "sns.countplot(data.iloc[:,-1], palette=['green', 'red'])\n",
    "plt.title(\"[0] == Not Disease, [1] == Disease\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a8fa5dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Alternative for graph of imbalance for target\n",
    "#below instead of\"target\" I will replace my own output name in dataframe\n",
    "df.target.value_counts().plot(kind='bar', title='Count (target)')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01b46440",
   "metadata": {},
   "source": [
    "Correlation of columns with eacht other using appropriate visuals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a707c234",
   "metadata": {},
   "outputs": [],
   "source": [
    "# subdata = df[['age','trestbps','chol','thalach','oldpeak', 'target']]\n",
    "sns.pairplot(data = df, hue = 'target') #or choose data = subdata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8733cb91",
   "metadata": {},
   "source": [
    "Correlation of columns with each other (with numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed226c92",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.subplots(figsize=(15,10))\n",
    "sns.heatmap(df.corr(), annot=True, cmap=sns.diverging_palette(220, 20, as_cmap=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "382b82ca",
   "metadata": {},
   "source": [
    "## Data Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6912aa8e",
   "metadata": {},
   "source": [
    "### Change column names to lower and rename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f98fe054",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns = df.columns.str.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c109701",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.rename(columns = {'old_name':'new_name'}, inplace = True)\n",
    "\n",
    "#or\n",
    "\n",
    "df.columns = ['col1', 'col2', 'col3']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b125b7b",
   "metadata": {},
   "source": [
    "### Drop unimportant columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "059699ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop('column_name',inplace=True,axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2253450f",
   "metadata": {},
   "source": [
    "### Add new column "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "085fcb97",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['age'] = df['this_year'] - df['birth_year']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "523a894c",
   "metadata": {},
   "source": [
    "### Checking duplicate values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cce0b79",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df.duplicated()==True]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fa38270",
   "metadata": {},
   "source": [
    "#### Drop duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "daadd2af",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop_duplicates(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30bf2f67",
   "metadata": {},
   "source": [
    "### Impute Missing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79133f55",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.Column_Name.fillna(df.Column_Name.mode()[0], inplace= True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78fdbc47",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "df_num = df.select_dtypes('number')\n",
    "df_col = df.select_dtypes('object')\n",
    "\n",
    "imp_mean = SimpleImputer(missing_values = np.nan, strategy = 'mean')\n",
    "imp_mean_col = SimpleImputer(missing_values = np.nan, strategy = 'most_frequent')\n",
    "\n",
    "df[df_num.columns] = imp_mean.fit_transform(df_num)\n",
    "df[df_col.columns] = imp_mean_col.fit_transform(df_col)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc79df07",
   "metadata": {},
   "source": [
    "Impute missing according to class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9e23c7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#boxplot with age on y-axis and Passenger class on x-axis.\n",
    "#here we take approximate medians for each class\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt  \n",
    "plt.figure(figsize=(16, 8))\n",
    "sns.set_style('darkgrid')\n",
    "sns.boxplot(x='Pclass', y='Age', data=df, palette='Dark2').set_title('Age_distribition_acording_to_pclass')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63a2c3d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "#here in each if statement we put each median age according to each class\n",
    "def impute_age(cols):\n",
    "    Age = cols[0]\n",
    "    Pclass = cols[1]\n",
    "    \n",
    "    if pd.isnull(Age):\n",
    "        if Pclass == 1:\n",
    "            return 37\n",
    "        elif Pclass == 2:\n",
    "            return 29\n",
    "        else:\n",
    "            return 24\n",
    "    else:\n",
    "        return Age"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4d0cc6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'] = df[['Age','Pclass']].apply(impute_age,axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "868f2e8b",
   "metadata": {},
   "source": [
    "If missing value is written as something other than null. for ex: \" ?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4aa526eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "#it will return column names with count of them\n",
    "df.isin([\" ?\"]).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2089875e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['columnname'].replace(' ?', df.columnname.mode()[0], inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dc3a90d",
   "metadata": {},
   "source": [
    "### Eliminating spaces or (, ) like symbols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b76f2ef8",
   "metadata": {},
   "outputs": [],
   "source": [
    "df= list(map(lambda x: x.strip(), df.column_name))\n",
    "df= list(map(lambda x: x.strip('('), df.column_name))\n",
    "df= list(map(lambda x: x.strip(')'), df.column_name))\n",
    "df= list(map(lambda x: x.strip(\"'\"), df.column_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d72f284",
   "metadata": {},
   "source": [
    "### Group same variables with different names (China-Mainland China)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "441c4b14",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.country = list(map(lambda x: 'china' if x == \"mainland china\" else x, df.country))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d9e26a7",
   "metadata": {},
   "source": [
    "### Outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8e484db",
   "metadata": {},
   "outputs": [],
   "source": [
    "#check for numeric columns\n",
    "plt.boxplot(columnname) \n",
    "fig = plt.figure(figsize =(10, 7))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1096307a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#detecting outliers with Z scores for numeric columns\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "def detect_outlier(columnname):\n",
    "    \n",
    "    outliers=[]    \n",
    "    threshold=3\n",
    "    mean_1 = np.mean(columnname)\n",
    "    std_1 =np.std(columnname)\n",
    "    \n",
    "    \n",
    "    for y in columnname:\n",
    "        z_score= (y - mean_1)/std_1 \n",
    "        if np.abs(z_score) > threshold:\n",
    "            outliers.append(y)\n",
    "    return outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "082f5381",
   "metadata": {},
   "outputs": [],
   "source": [
    "#deleting outlier in a spesific column\n",
    "sample_outlier = detect_outlier(columnname)\n",
    "for i in sample_outliers:\n",
    "    columnname = np.delete(sample, np.where(sample==i))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e66578bc",
   "metadata": {},
   "source": [
    "## Feature Engineering "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01190fc1",
   "metadata": {},
   "source": [
    "### Merging two dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfa5e0ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "store = furniture.merge(office, how='inner', on='Order Date')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6b838ce",
   "metadata": {},
   "source": [
    "### Convert categorical variable into dummy/indicator variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6544b0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.get_dummies(data, prefix_sep='_', columns = ['Gender'], drop_first=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89bdcf1a",
   "metadata": {},
   "source": [
    "### Binarize Data (for ratio using threshold ex: age > 60 then 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "407e0682",
   "metadata": {},
   "outputs": [],
   "source": [
    "age = df.iloc[:, 1]\n",
    "from sklearn.preprocessing import Binarizer\n",
    "x = age.values.reshape(1, -1)\n",
    "\n",
    "# For age, let threshold be 35 \n",
    "binarizer = Binarizer(35)\n",
    "\n",
    "# Transformed feature \n",
    "Binarized_age = binarizer.fit_transform(x)\n",
    "df ['Bin.Age>35'] = Binarized_age[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a328b33b",
   "metadata": {},
   "source": [
    "### One - hot Encoding (for Categorical Nominal data ex: weekdays)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51690402",
   "metadata": {},
   "outputs": [],
   "source": [
    "#creates seperate columns for each row and gives 0 or 1 accordingly\n",
    "\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.compose import ColumnTransformer\n",
    "ct = ColumnTransformer(\n",
    "    [('one_hot_encoder', OneHotEncoder(categories='auto'), [0])],   # The column numbers to be transformed (here is [0] but can be [0, 1, 3])\n",
    "    remainder='passthrough'                                         # Leave the rest of the columns untouched\n",
    ")\n",
    "\n",
    "x = ct.fit_transform(x)\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acd7f7b9",
   "metadata": {},
   "source": [
    "### Label Encoder (for Ordinal data ex: education level)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5844097",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "label_encoder = LabelEncoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "195f1ae0",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2460d2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cat = [ 'Gender', 'Married', 'Dependents', 'Education', 'Self_Employed', 'Property_Area', 'Loan_Status']\n",
    "for col in df_cat:\n",
    "    df[col] = label_encoder.fit_transform(df[col])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ae924ab",
   "metadata": {},
   "source": [
    "## Scaling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e8fbd53",
   "metadata": {},
   "outputs": [],
   "source": [
    "# StandardScaler results in a distribution with a standard deviation equal to 1.\n",
    "# StandardScaler makes the mean of the distribution approximately 0.\n",
    "# Use StandardScaler if you want each feature to have zero-mean, unit standard-deviation.\n",
    "# If you want more normally distributed data, and are okay with transforming your data.\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "x = StandardScaler().fit_transform(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44cd5195",
   "metadata": {},
   "outputs": [],
   "source": [
    "#MinMaxScaler preserves the shape of the original distribution.\n",
    "#MinMaxScaler doesn’t reduce the importance of outliers.\n",
    "#Default range for the feature returned by MinMaxScaler is 0 to 1.\n",
    "\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "x = MinMaxScaler().fit_transform(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "355a4dad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# RobustScaler does not scale the data into a predetermined interval like MinMaxScaler.\n",
    "# Use RobustScaler if you want to reduce the effects of outliers, relative to MinMaxScaler.\n",
    "\n",
    "from sklearn.preprocessing import RobustScaler\n",
    "x = RobustScaler().fit_transform(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36b8ba3a",
   "metadata": {},
   "source": [
    "# Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67f46e11",
   "metadata": {},
   "source": [
    "- Supervised Learning:\n",
    "    - Regressions:\n",
    "        - Linear Regression\n",
    "        - Support Vector Regressor\n",
    "        - Decision Trees and Random Forests\n",
    "        \n",
    "    - Classifiers:\n",
    "        - Logistic Regression\n",
    "        - Naive Bayes\n",
    "        - Support Vector Machines\n",
    "        - KNN: K-Nearest Neighbors\n",
    "        - Decision Trees\n",
    "        - Random Forests\n",
    "        - Neural Networks/Deep Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab4451ef",
   "metadata": {},
   "source": [
    "- Unsupervised Learning:\n",
    "    - Clustering:\n",
    "        - K-Means"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b329a7fc",
   "metadata": {},
   "source": [
    "## Linear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d257a5eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80cc22e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.iloc[:,:-1] #.values.reshape(-1,1) - for single x linear regression\n",
    "#X= df.drop(\"Price\", axis = 1)\n",
    "\n",
    "y = df.iloc[:,-1] #.values.reshape(-1,1) - for single linear regression\n",
    "#y= df['Price']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77b3aea2",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.2, random_state = 123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2791748",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LinearRegression()\n",
    "model.fit(X_train, y_train) #training the algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec6a3f5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Printing model coefficient and intercept\n",
    "#y = kx+b ----> b:\n",
    "print('Intercept: ', model.intercept_)\n",
    "#y = kx+b ----> k:\n",
    "coeff_df = pd.DataFrame(model.coef_, X.columns, columns=['Coefficient'])  \n",
    "coeff_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0eda2bc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_test)\n",
    "#print('predicted response:', y_pred, sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "033bd272",
   "metadata": {},
   "source": [
    "Comparing Actual and Predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19c8d10b",
   "metadata": {},
   "outputs": [],
   "source": [
    "comparison = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred})\n",
    "comparison.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46a41eab",
   "metadata": {},
   "source": [
    "Visual Comparison of Actual and Predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81ff4953",
   "metadata": {},
   "outputs": [],
   "source": [
    "compared = Comparison.head(25)\n",
    "compared.plot(kind='bar',figsize=(16,10))\n",
    "plt.grid(which='major', linestyle='-', linewidth='0.5', color='green')\n",
    "plt.grid(which='minor', linestyle=':', linewidth='0.5', color='black')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a96d4d56",
   "metadata": {},
   "source": [
    "X_test, y_test Scatter plot on X_test, y_pred regression line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31d7c69f",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(X_test, y_test,  color='gray')\n",
    "plt.plot(X_test, y_pred, color='red', linewidth=2)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "103b54ec",
   "metadata": {},
   "source": [
    "Evaluation of Linear Regression Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "897b47ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "R2 = metrics.r2_score(y_test, y_pred) #coefficient of determination\n",
    "print('R^2:' , R2)\n",
    "n = X_test.shape[0] #sample size\n",
    "p = X_test.shape[1] #number of predictors\n",
    "print('Adjusted R^2 :' , 1-(1-R2)*(n-1)/(n-p-1))\n",
    "print('Mean Absolute Error:' , metrics.mean_absolute_error(y_test, y_pred)) \n",
    "print('Mean Squared Error:' , metrics.mean_squared_error(y_test, y_pred)) \n",
    "print('Root Mean Squared Error:' , np.sqrt(metrics.mean_squared_error(y_test, y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9d5ba30",
   "metadata": {},
   "source": [
    "### Assumptions for Linear Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bef9f393",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\"Linearity\" with numeric columns\n",
    "p = sns.pairplot(ad_data, x_vars=['input columns'], y_vars='output column', size=7, aspect=0.7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e102c19",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\"Mean Of Residuals\". Should be close to 0.\n",
    "residuals = y_train.values-y_pred\n",
    "mean_residuals = np.mean(residuals)\n",
    "print(\"Mean of Residuals {}\".format(mean_residuals))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3224a4d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "#the \"Homoscedasticity\". \n",
    "p = sns.scatterplot(y_pred,residuals)\n",
    "plt.xlabel('y_pred/predicted values')\n",
    "plt.ylabel('Residuals')\n",
    "plt.ylim(-10,10)\n",
    "plt.xlim(0,26)\n",
    "p = sns.lineplot([0,26],[0,0],color='blue')\n",
    "p = plt.title('Residuals vs fitted values plot for homoscedasticity check')\n",
    "\n",
    "#on the new line - p value should be more than 0.05\n",
    "import statsmodels.stats.api as sms\n",
    "from statsmodels.compat import lzip\n",
    "name = ['F statistic', 'p-value']\n",
    "test = sms.het_goldfeldquandt(residuals, X_train)\n",
    "lzip(name, test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58396c10",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\"Normality of Residuals\".\n",
    "p = sns.distplot(residuals,kde=True)\n",
    "p = plt.title('Normality of error terms/residuals')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c2c7910",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Checking \"Autocorrelation\".\n",
    "plt.figure(figsize=(10,5))\n",
    "p = sns.lineplot(y_pred,residuals,marker='o',color='blue')\n",
    "plt.xlabel('y_pred/predicted values')\n",
    "plt.ylabel('Residuals')\n",
    "plt.ylim(-10,10)\n",
    "plt.xlim(0,26)\n",
    "p = sns.lineplot([0,26],[0,0],color='red')\n",
    "p = plt.title('Residuals vs fitted values plot for autocorrelation check')\n",
    "\n",
    "#on the new line - p value should be less than 0.05\n",
    "from statsmodels.stats import diagnostic as diag\n",
    "min(diag.acorr_ljungbox(residuals , lags = 40)[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7bf6a7f",
   "metadata": {},
   "source": [
    "## Logistic Regression - Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7842c27e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78ea09ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.iloc[:,:-1] #.values.reshape(-1,1) - for single x linear regression\n",
    "#X= df.drop(\"Price\", axis = 1)\n",
    "\n",
    "y = df.iloc[:,-1] #.values.reshape(-1,1) - for single linear regression\n",
    "#y= df['Price']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e513677",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.2, random_state = 123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fba392eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LogisticRegression(max_iter = 100000)\n",
    "model.fit(X_train, y_train) #train the model\n",
    "y_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6f27332",
   "metadata": {},
   "source": [
    "Comparing Actual and Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "440809af",
   "metadata": {},
   "outputs": [],
   "source": [
    "comparison = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred})\n",
    "comparison.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1157b59",
   "metadata": {},
   "source": [
    "Evaluation of Logistic Regression Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d852d526",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(confusion_matrix(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7457f82",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6037a9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "roc_auc_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "915044d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "logit_roc_auc = roc_auc_score(y_test, model.predict(X_test))\n",
    "fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])\n",
    "plt.figure(figsize=(16,8))\n",
    "plt.plot(fpr, tpr, label='Logistic Regression (area = %0.2f)' % logit_roc_auc)\n",
    "plt.plot([0, 1], [0, 1],'r--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Survived or Not')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.savefig('Log_ROC')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50afed3f",
   "metadata": {},
   "source": [
    "## Naive Bayes - Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1947c32d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25a871ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.iloc[:,:-1] #.values.reshape(-1,1) - for single x linear regression\n",
    "#X= df.drop(\"Price\", axis = 1)\n",
    "\n",
    "y = df.iloc[:,-1] #.values.reshape(-1,1) - for single linear regression\n",
    "#y= df['Price']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab2a2cd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.2, random_state = 123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9ae4211",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = GaussianNB()\n",
    "model.fit(X_train, y_train)\n",
    "y_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba24a782",
   "metadata": {},
   "source": [
    "Comparing actual and prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f49612d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "comparison = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred})\n",
    "comparison.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14e87728",
   "metadata": {},
   "source": [
    "Evaluation of Naive Bayes Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5b4ffe3",
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_test, y_pred)\n",
    "print('Confusion Matrix for Naive Bayes:''\\n', cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6a5d3d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "cr = classification_report(y_test, y_pred)\n",
    "print('Classification Report for Naive Bayes:''\\n''\\n', cr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f1444d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "auc = roc_auc_score(y_test, y_pred)\n",
    "print('Auc Score for Naive Bayes:', auc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccc4bcac",
   "metadata": {},
   "outputs": [],
   "source": [
    "NB_roc_auc = roc_auc_score(y_test, model.predict(X_test))\n",
    "fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])\n",
    "plt.figure(figsize=(10,5))\n",
    "plt.plot(fpr, tpr, label='Naive Bayes (area = %0.2f)' % NB_roc_auc)\n",
    "plt.plot([0, 1], [0, 1],'r--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Survived or Not')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.savefig('NB_ROC')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c337d23",
   "metadata": {},
   "source": [
    "## SVM - Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67d87a2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import svm\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b6a2e68",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.iloc[:,:-1] #.values.reshape(-1,1) - for single x linear regression\n",
    "#X= df.drop(\"Price\", axis = 1)\n",
    "\n",
    "y = df.iloc[:,-1] #.values.reshape(-1,1) - for single linear regression\n",
    "#y= df['Price']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd59d52c",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.2, random_state = 123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3abd263",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "kernel takes low dimensional input space and transforms it to a higher dimensional space. So, converts “not separable” \n",
    "problem to “separable problem” by adding more dimension to it.\n",
    "'''\n",
    "\n",
    "#\"Linear\" kernel\n",
    "model = svm.SVC(kernel = 'linear', probability=True)\n",
    "\n",
    "#“Polynomial” kernel is better for curved or nonlinear input space.\n",
    "#model = svm.SVC(kernel = 'poly', degree=2)\n",
    "\n",
    "#“Radial basis function” kernel can map an input space in infinite dimensional space.\n",
    "#model = svm.SVC(kernel = 'rbf', C=1, gamma=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "269f81f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Train the model using the training sets\n",
    "model = model.fit(X_train, y_train)\n",
    "#Predict the response for test dataset\n",
    "y_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "543b744a",
   "metadata": {},
   "source": [
    "Comparing Prediction and Actual"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "985bff1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "comparison = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred})\n",
    "comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a337c22a",
   "metadata": {},
   "source": [
    "Evaluation of SVM Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "834546d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_test, y_pred)\n",
    "print('Confusion Matrix for SVM:''\\n', cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1dc0cf4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "cr = classification_report(y_test, y_pred)\n",
    "print(\"Classification Report for SVM:\"\"\\n\", cr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a358213",
   "metadata": {},
   "outputs": [],
   "source": [
    "auc = roc_auc_score(y_test, y_pred)\n",
    "print('Auc Score for svm:', auc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "601d31c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "SVM_roc_auc = roc_auc_score(y_test, model.predict(X_test))\n",
    "fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])\n",
    "plt.figure(figsize=(16,8))\n",
    "plt.plot(fpr, tpr, label='SVM (area = %02f)' % SVM_roc_auc)\n",
    "plt.plot([0, 1], [0, 1],'r--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Survived or Not')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.savefig('SVM_ROC')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "887360bc",
   "metadata": {},
   "source": [
    "## Decision Trees - Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afe01895",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier \n",
    "from sklearn.model_selection import train_test_split \n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62f595a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.iloc[:,:-1] #.values.reshape(-1,1) - for single x linear regression\n",
    "#X= df.drop(\"Price\", axis = 1)\n",
    "\n",
    "y = df.iloc[:,-1] #.values.reshape(-1,1) - for single linear regression\n",
    "#y= df['Price']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40c83153",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.2, random_state = 123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5fddadd",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = DecisionTreeClassifier()\n",
    "#model = DecisionTreeClassifier(criterion=\"entropy\")\n",
    "model = model.fit(X_train, y_train)\n",
    "y_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18504840",
   "metadata": {},
   "source": [
    "Evaluation of Decision Tree Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b69685e",
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_test, y_pred)\n",
    "print('Confusion Matrix for Decision Tree:''\\n', cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27bdaa77",
   "metadata": {},
   "outputs": [],
   "source": [
    "cr = classification_report(y_test, y_pred)\n",
    "print(\"Classification Report for Decision Tree:\"\"\\n\"\"\\n\", cr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c75b6e84",
   "metadata": {},
   "outputs": [],
   "source": [
    "auc = roc_auc_score(y_test, y_pred)\n",
    "print('Auc Score for Decision Tree:', auc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "554a86e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "DT_roc_auc = roc_auc_score(y_test, model.predict(X_test))\n",
    "fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])\n",
    "plt.figure(figsize = (16, 8))\n",
    "plt.plot(fpr, tpr, label='SVM (area = %02f)' % DT_roc_auc)\n",
    "plt.plot([0, 1], [0, 1],'r--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Survived or Not')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.savefig('Decision_Tree_ROC')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "083f25ce",
   "metadata": {},
   "source": [
    "## Random Forest - Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9416cfbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import train_test_split \n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5bbbbec",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.iloc[:,:-1] #.values.reshape(-1,1) - for single x linear regression\n",
    "#X= df.drop(\"Price\", axis = 1)\n",
    "\n",
    "y = df.iloc[:,-1] #.values.reshape(-1,1) - for single linear regression\n",
    "#y= df['Price']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbe8fc58",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.2, random_state = 123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e811c93",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = RandomForestClassifier(n_estimators = 10, random_state = None)\n",
    "model = model.fit(X_train, y_train)\n",
    "y_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae7ad4f8",
   "metadata": {},
   "source": [
    "Evaluation of Random Forest Classifcation Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd8212ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_test, y_pred)\n",
    "print('Confusion Matrix for SVM:''\\n', cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0a1c3f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "cr = classification_report(y_test, y_pred)\n",
    "print(\"Classification Report for SVM:\"\"\\n\"\"\\n\", cr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b91e5fc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "auc = roc_auc_score(y_test, y_pred)\n",
    "print('Auc Score for svm:', auc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2882cc9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "RF_roc_auc = roc_auc_score(y_test, model.predict(X_test))\n",
    "fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])\n",
    "plt.figure(figsize = (16, 8))\n",
    "plt.plot(fpr, tpr, label = 'RF (area = %02f)' % RF_roc_auc)\n",
    "plt.plot([0, 1], [0, 1],'r--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Survived or Not')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.savefig('RF_ROC')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9da3b5e4",
   "metadata": {},
   "source": [
    "## KNN - Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7856ef57",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import train_test_split \n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d447c1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.iloc[:,:-1] #.values.reshape(-1,1) - for single x linear regression\n",
    "#X= df.drop(\"Price\", axis = 1)\n",
    "\n",
    "y = df.iloc[:,-1] #.values.reshape(-1,1) - for single linear regression\n",
    "#y= df['Price']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0dc66327",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.2, random_state = 123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce8e3059",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = KNeighborsClassifier(n_neighbors=5)\n",
    "model = model.fit(X_train, y_train)\n",
    "y_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29128583",
   "metadata": {},
   "source": [
    "Evaluation of KNN Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0328d981",
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_test, y_pred)\n",
    "print('Confusion matrix ''\\n', cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60388ad9",
   "metadata": {},
   "outputs": [],
   "source": [
    "cr = classification_report(y_test, y_pred)\n",
    "print(\"Classification Report for KNN:\"\"\\n\"\"\\n\", cr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7889cc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "auc = roc_auc_score(y_test, y_pred)\n",
    "print('Auc Score for KNN:', auc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e99e9f03",
   "metadata": {},
   "outputs": [],
   "source": [
    "KNN_roc_auc = roc_auc_score(y_test, model.predict(X_test))\n",
    "fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])\n",
    "plt.figure(figsize = (16, 8))\n",
    "plt.plot(fpr, tpr, label = 'KNN (area = %02f)' % KNN_roc_auc)\n",
    "plt.plot([0, 1], [0, 1],'r--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Survived or Not')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.savefig('KNN_ROC')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4df9e3d6",
   "metadata": {},
   "source": [
    "## Cross Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3e05c6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "accuracies = cross_val_score(estimator = model, X = X_train, y = y_train, cv = 10)\n",
    "\n",
    "print('Mean accuracy CV:',accuracies.mean())\n",
    "print('Standard deviation of CV:',accuracies.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c258df8",
   "metadata": {},
   "source": [
    "## Ridge and Lasso"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "143412d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge\n",
    "ridge_l = [0,1,10,100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c19baec",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, l in enumerate(ridge_l):\n",
    "    ridge = Ridge(alpha=l)\n",
    "    ridge.fit(x, y)\n",
    "\n",
    "    ridge_coeff = ridge.coef_\n",
    "    ridge_intercept = ridge.intercept_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad5f8cf0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Lasso\n",
    "lasso_l = [0,0.5,1,2,4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5a25b03",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, l in enumerate(lasso_l):\n",
    "    lasso = Lasso(alpha=l)\n",
    "    lasso.fit(x, y)\n",
    "\n",
    "    lasso_coeff = lasso.coef_\n",
    "    lasso_intercept = lasso.intercept_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "001b5ff0",
   "metadata": {},
   "source": [
    "## Gradient Boosting Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8765f00d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "gbm = GradientBoostingClassifier(random_state = 123)\n",
    "\n",
    "gbm.fit(x_train, y_train)\n",
    "\n",
    "gbm_pred = gbm.predict(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea0364b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(y_test, gbm_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e64eae8",
   "metadata": {},
   "source": [
    "## XGBoost Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68135638",
   "metadata": {},
   "outputs": [],
   "source": [
    "#if you need for regression problem you just change XGBClassifier to XGBRegressor\n",
    "pip install xgboost\n",
    "\n",
    "import xgboost\n",
    "print(xgboost.__version__)\n",
    "\n",
    "from xgboost import XGBClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "315255ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "xgbt = XGBClassifier(max_depth = 2,\n",
    "             learning_rate = 0.2,\n",
    "             objective  = \"multi:softmax\",\n",
    "             num_class = 2,\n",
    "             booster = \"gbtree\",\n",
    "             n_estimarors = 10,\n",
    "             random_state = 123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c24ccb0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "xgbt.fit(X_train, y_train)\n",
    "\n",
    "xgbt_pred = xgbt.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "583078a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(y_test, xgbt_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7e04035",
   "metadata": {},
   "outputs": [],
   "source": [
    "xgbt.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1325231",
   "metadata": {},
   "outputs": [],
   "source": [
    "xgbt.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cb60667",
   "metadata": {},
   "source": [
    "# CatBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "405f8647",
   "metadata": {},
   "outputs": [],
   "source": [
    "from catboost import CatBoostRegressor\n",
    "\n",
    "model = CatBoostRegressor(iterations=2,\n",
    "                          learning_rate=1,\n",
    "                          depth=2)\n",
    "\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "preds = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9b3992e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "model.score(X_train, y_train)\n",
    "\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a62a5e5",
   "metadata": {},
   "source": [
    "# Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "218c7253",
   "metadata": {},
   "source": [
    "## Hand C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df50498a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "svc = SVC(random_state=1234, kernel=\"rbf\",C=1)\n",
    "svc.fit(X_train, y_train)\n",
    "y_pred_1 = svc.predict(X_test)\n",
    "\n",
    "from sklearn import metrics\n",
    "\n",
    "print(\"Accuracy:\",metrics.accuracy_score(y_test, y_pred_1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ccf59c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "svc = SVC(random_state=1234, kernel=\"poly\", C = 1 )\n",
    "svc.fit(X_train, y_train)\n",
    "y_pred_2 = svc.predict(X_test)\n",
    "\n",
    "print(\"Accuracy:\",metrics.accuracy_score(y_test, y_pred_1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9046dfd1",
   "metadata": {},
   "source": [
    "## GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adc3cebf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5254ca90",
   "metadata": {},
   "source": [
    "Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "336ee1ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "rfc_param = {'n_estimators':[10,15,20], \n",
    "            'min_samples_split':[8,16],\n",
    "            'min_samples_leaf':[1,2,3,4,5]\n",
    "            }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd173621",
   "metadata": {},
   "outputs": [],
   "source": [
    "rfc_grid = GridSearchCV(estimator=rfc, \n",
    "                        param_grid=rfc_param,\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_jobs=-1,\n",
    "                        return_train_score=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a063b86e",
   "metadata": {},
   "outputs": [],
   "source": [
    "rfc_grid_fit = rfc_grid.fit(x,y)\n",
    "\n",
    "cv_results_rfc = pd.DataFrame.from_dict(rfc_grid_fit.cv_results_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4759fad6",
   "metadata": {},
   "source": [
    "Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41aa2890",
   "metadata": {},
   "outputs": [],
   "source": [
    "lrc_param = {'C':[0.01, 0.1, 0.5, 1, 2, 5, 10], \n",
    "            'penalty':['l2'],\n",
    "            'solver':['liblinear','lbfgs', 'saga']\n",
    "            }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b685575",
   "metadata": {},
   "outputs": [],
   "source": [
    "lrc_grid = GridSearchCV(estimator=lrc, \n",
    "                        param_grid=lrc_param,\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_jobs=-1,\n",
    "                        return_train_score=True,\n",
    "                        verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d045b681",
   "metadata": {},
   "outputs": [],
   "source": [
    "lrc_grid_fit = lrc_grid.fit(x, y)\n",
    "\n",
    "cv_results_lrc = pd.DataFrame.from_dict(lrc_grid_fit.cv_results_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ec05d74",
   "metadata": {},
   "source": [
    "Support Vector Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e9414f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "svc_param = {'C':[0.01, 0.1, 0.5, 1, 2, 5, 10], \n",
    "            'kernel':['rbf', 'linear'],\n",
    "            'gamma':[0.1, 0.25, 0.5, 1, 5]\n",
    "            }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df376944",
   "metadata": {},
   "outputs": [],
   "source": [
    "svc_grid = GridSearchCV(estimator=svc, \n",
    "                        param_grid=svc_param,\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_jobs=-1,\n",
    "                        return_train_score=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4611f542",
   "metadata": {},
   "outputs": [],
   "source": [
    "svc_grid_fit = svc_grid.fit(x, y)\n",
    "\n",
    "cv_results_svc = pd.DataFrame.from_dict(svc_grid_fit.cv_results_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b4bcadb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the top ranked test score for all the three classifiers\n",
    "\n",
    "rfc_top_rank = cv_results_rfc[cv_results_rfc['rank_test_score'] == 1]\n",
    "lrc_top_rank = cv_results_lrc[cv_results_lrc['rank_test_score'] == 1]\n",
    "svc_top_rank = cv_results_svc[cv_results_svc['rank_test_score'] == 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b21f3e6",
   "metadata": {},
   "source": [
    "## RandomizedSearchCV "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40ae1f2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "rfc = RandomForestClassifier(random_state=1234)\n",
    "\n",
    "from sklearn.model_selection import RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7273b735",
   "metadata": {},
   "outputs": [],
   "source": [
    "rfc_param = {'n_estimators':[10,15,20], \n",
    "            'min_samples_split':[8,16],\n",
    "            'min_samples_leaf':[1,2,3,4,5]\n",
    "            }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad272c8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "rfc_rs = RandomizedSearchCV(estimator=rfc, \n",
    "                        param_distributions=rfc_param,\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_iter=10,\n",
    "                        return_train_score=True,\n",
    "                        random_state=1234)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e82bb69a",
   "metadata": {},
   "outputs": [],
   "source": [
    "rfc_rs_fit = rfc_rs.fit(x, y)\n",
    "\n",
    "cv_results_rfc_rs = pd.DataFrame.from_dict(rfc_rs_fit.cv_results_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6ebe032",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('\\n The best Parameters are : ')\n",
    "print(rfc_rs_fit.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbbd6041",
   "metadata": {},
   "source": [
    "## BayesSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15093808",
   "metadata": {},
   "outputs": [],
   "source": [
    "from skopt import BayesSearchCV\n",
    "from skopt.space import Real, Categorical, Integer\n",
    "\n",
    "opt = BayesSearchCV(\n",
    "    SVC(),\n",
    "    {\n",
    "        'C': Real(1e-6, 1e+6, prior='log-uniform'),\n",
    "        'gamma': Real(1e-6, 1e+1, prior='log-uniform'),\n",
    "        'degree': Integer(1,8),\n",
    "        'kernel': Categorical(['linear', 'poly', 'rbf']),\n",
    "    },\n",
    "    n_iter=32,\n",
    "    random_state=0\n",
    ")\n",
    "\n",
    "_ = opt.fit(x_train, y_train)\n",
    "\n",
    "\n",
    "print(opt.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62f5e416",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2cffbad",
   "metadata": {},
   "source": [
    "# Unsupervised Learning "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70028db5",
   "metadata": {},
   "source": [
    "## PCA "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bbf0ac4",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pd.read_csv('Wine.csv')\n",
    "X = dataset.iloc[:, 0:13].values\n",
    "y = dataset.iloc[:, 13].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b57226d",
   "metadata": {},
   "outputs": [],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "138dfc5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e68d1a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = StandardScaler()\n",
    "X = sc.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2dffbcaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4aaf0423",
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = PCA(n_components = 2)\n",
    "X_train = pca.fit_transform(X_train)\n",
    "X_test = pca.transform(X_test)\n",
    "explained_variance = pca.explained_variance_ratio_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcfa1a72",
   "metadata": {},
   "outputs": [],
   "source": [
    "explained_variance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70e252f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a85e287",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6aebf433",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fitting Logistic Regression to the Training set\n",
    "\n",
    "classifier = LogisticRegression(random_state = 0)\n",
    "classifier.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b247f994",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predicting the Test set results\n",
    "y_pred = classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb686f8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Making the Confusion Matrix\n",
    "\n",
    "cm = confusion_matrix(y_test, y_pred)\n",
    "cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70d85f00",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3372d2ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f50b4152",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualising the Training set results\n",
    "\n",
    "X_set, y_set = X_train, y_train\n",
    "#minimum və maximum dəyərlərin köməyi ilə x1 və x2 (PCA1 və (PCA2) oxlarının aralığı təyin olunur)\n",
    "X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),\n",
    "                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))\n",
    "plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),\n",
    "             alpha = 0.75, cmap = ListedColormap(('red', 'green', 'blue')))\n",
    "plt.xlim(X1.min(), X1.max())\n",
    "plt.ylim(X2.min(), X2.max())\n",
    "for i, j in enumerate(np.unique(y_set)):\n",
    "    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],\n",
    "                c = ListedColormap(('red', 'green', 'blue'))(i), label = j)\n",
    "plt.title('Logistic Regression (Training set)')\n",
    "plt.xlabel('PC1')\n",
    "plt.ylabel('PC2')\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56b7f1ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualising the Test set results\n",
    "from matplotlib.colors import ListedColormap\n",
    "X_set, y_set = X_test, y_test\n",
    "X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),\n",
    "                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))\n",
    "plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),\n",
    "             alpha = 0.75, cmap = ListedColormap(('red', 'green', 'blue')))\n",
    "plt.xlim(X1.min(), X1.max())\n",
    "plt.ylim(X2.min(), X2.max())\n",
    "for i, j in enumerate(np.unique(y_set)):\n",
    "    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],\n",
    "                c = ListedColormap(('red', 'green', 'blue'))(i), label = j)\n",
    "plt.title('Logistic Regression (Test set)')\n",
    "plt.xlabel('PC1')\n",
    "plt.ylabel('PC2')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8c44567",
   "metadata": {},
   "source": [
    "# Time Series Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff224c4f",
   "metadata": {},
   "source": [
    "## FBprophet "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d74718ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"/content/wiki_machine_learning.csv\", sep = \" \")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0b05925",
   "metadata": {},
   "outputs": [],
   "source": [
    "#minimum and maximum dates\n",
    "df['date'].min(), df['date'].max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a1d6d8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#removing columns we don't need and check if there is any missing value\n",
    "cols = ['lang', 'page', 'rank', 'month', 'title']\n",
    "df.drop(cols, axis=1, inplace=True)\n",
    "df = df.sort_values('date')\n",
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92995971",
   "metadata": {},
   "outputs": [],
   "source": [
    "#to fill null calues\n",
    "df.fillna(method=‘backfill’,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1499e066",
   "metadata": {},
   "outputs": [],
   "source": [
    "#groupby index date\n",
    "df = df.groupby('date')['count'].sum().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2446956",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.set_index('date')\n",
    "df.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da906fdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df['count']\n",
    "y.plot(figsize=(15, 6))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4a3ddf4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8bd5fd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2686296",
   "metadata": {},
   "outputs": [],
   "source": [
    "#craeating prophet model\n",
    "from fbprophet import Prophet\n",
    "df = df.rename(columns={'date': 'ds', 'count': 'y'})\n",
    "df_model = Prophet(interval_width=0.95)\n",
    "df_model.fit(df)\n",
    "df_forecast = df_model.make_future_dataframe(periods=30, freq='D')\n",
    "df_forecast = df_model.predict(df_forecast)\n",
    "\n",
    "#for month - 'month' or 'M', hour - 'H', day - 'D',"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7933808e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#to show the plot\n",
    "plt.figure(figsize=(300, 500))\n",
    "df_model.plot(df_forecast, xlabel = 'date', ylabel = 'count')\n",
    "plt.title('Count for Next 30');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae6ed027",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_model.plot_components(df_forecast);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2b37ee5",
   "metadata": {},
   "source": [
    "## Neural Prophet "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cc65f3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from neuralprophet import NeuralProphet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a50e0b06",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_excel(\"Superstore.xls\")\n",
    "furniture = df.loc[df['Category'] == 'Furniture']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73e0e404",
   "metadata": {},
   "outputs": [],
   "source": [
    "#removing columns we don't need and check if there is any missing value\n",
    "cols = ['Row ID', 'Order ID', 'Ship Date', 'Ship Mode', 'Customer ID', 'Customer Name', 'Segment', 'Country', 'City', 'State', 'Postal Code', 'Region', 'Product ID', 'Category', 'Sub-Category', 'Product Name', 'Quantity', 'Discount', 'Profit']\n",
    "furniture.drop(cols, axis=1, inplace=True)\n",
    "furniture = furniture.sort_values('Order Date')\n",
    "furniture.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f36bac6",
   "metadata": {},
   "outputs": [],
   "source": [
    "#groupby index date\n",
    "furniture = furniture.groupby('Order Date')['Sales'].sum().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3f366c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "furniture = furniture.set_index('Order Date')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02ae6ead",
   "metadata": {},
   "outputs": [],
   "source": [
    "#we will use the averages daily sales value for that month instead, and we are using the start of each month as the timestamp.\n",
    "y = furniture['Sales'].resample('MS').mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5118ed07",
   "metadata": {},
   "outputs": [],
   "source": [
    "#rename column names as neural prophet expect time-series as ds and value to predict as y\n",
    "y= y.reset_index()\n",
    "y.rename(columns={\"Order Date\": \"ds\", \"Sales\": \"y\"}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98ed136b",
   "metadata": {},
   "outputs": [],
   "source": [
    "y.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d118427",
   "metadata": {},
   "outputs": [],
   "source": [
    "y.plot(x='ds',y='y',figsize=(15, 6))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d233db36",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model = NeuralProphet() if you're using default variables below.\n",
    "model = NeuralProphet(\n",
    "    growth=\"linear\",  # Determine trend types: 'linear', 'discontinuous', 'off'\n",
    "    changepoints=None, # list of dates that may include change points (None -> automatic )\n",
    "    n_changepoints=5,\n",
    "    changepoints_range=0.8,\n",
    "    trend_reg=0,\n",
    "    trend_reg_threshold=False,\n",
    "    yearly_seasonality=\"auto\",\n",
    "    weekly_seasonality=\"auto\",\n",
    "    daily_seasonality=\"auto\",\n",
    "    seasonality_mode=\"additive\",\n",
    "    seasonality_reg=0,\n",
    "    n_forecasts=1,\n",
    "    n_lags=0,\n",
    "    num_hidden_layers=0,\n",
    "    d_hidden=None,     # Dimension of hidden layers of AR-Net\n",
    "    ar_sparsity=None,  # Sparcity in the AR coefficients\n",
    "    learning_rate=None,\n",
    "    epochs=40,\n",
    "    loss_func=\"Huber\",\n",
    "    normalize=\"auto\",  # Type of normalization ('minmax', 'standardize', 'soft', 'off')\n",
    "    impute_missing=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10e8d062",
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics = model.fit(y, validate_each_epoch=True, freq=\"MS\") \n",
    "future = model.make_future_dataframe(y, periods=10, n_historic_predictions=len(y)) \n",
    "forecast = model.predict(future)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05194f2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#plotting predictions\n",
    "fig, ax = plt.subplots(figsize=(14, 10)) \n",
    "model.plot(forecast, xlabel=\"Date\", ylabel=\"Sales\", ax=ax);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b37ea1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#plotting components of model\n",
    "model.plot_parameters()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "369d9bb5",
   "metadata": {},
   "source": [
    "## Pipeline / Pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ed81a59",
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "from google.colab import files\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.preprocessing import LabelEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f30683e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "data=pd.read_csv('/content/income_evaluation.csv')\n",
    "df=data.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9808ae0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a520f037",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.compose import ColumnTransformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f934be0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns=df.columns.str.replace(' ','')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec43d055",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop('education',axis=1,inplace=True)\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ae62e8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "x=df.drop(\"income\",axis=1)\n",
    "y=df['income']\n",
    "x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.2,random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da90d363",
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_columns=x.select_dtypes(include=\"number\").columns\n",
    "categorical_columns=x.select_dtypes(include=\"object\").columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7586697d",
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77f4e2d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "numerical_Pipeline = Pipeline(steps=[( 'scaler', StandardScaler() ) ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea331b5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_Pipeline = Pipeline(steps=[('encoder1', OneHotEncoder(handle_unknown='ignore'))])\n",
    "                                       #('encoder2', LabelEncoder() ) ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c44600a",
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('categorical', categorical_Pipeline, categorical_columns),\n",
    "        ('numerical', numerical_Pipeline, numeric_columns)\n",
    "    ])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b0a7cac",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Pipeline(steps=[\n",
    "    ('preprocessor', preprocessor)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be41b1ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe.fit(x_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b714b5a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = pipe.transform(x_train)\n",
    "x_test = pipe.transform(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f3a0333",
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost as xgb\n",
    "model=xgb.XGBClassifier(random_state=1,learning_rate=0.01)\n",
    "model=model.fit(x_train,y_train)\n",
    "y_pred=model.predict(x_test)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "725ff94d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score,confusion_matrix\n",
    "print(\"Confusion Matrix \\n\",confusion_matrix(y_test,y_pred))\n",
    "accuracy_score(y_test,y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b97bd86b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "document=\"myModel\"\n",
    "pickle.dump(model,open(document,\"wb\"))\n",
    "loaded_model=pickle.load(open(document,'rb'))\n",
    "y_loded_model_pred=loaded_model.predict(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acd92faa",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score,confusion_matrix\n",
    "print(\"Confusion Matrix \\n\\n\",confusion_matrix(y_test,y_loded_model_pred),\"\\n\")\n",
    "print(\"Accuary Score\" , '= %0.2f' % accuracy_score(y_test,y_loded_model_pred),\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91104099",
   "metadata": {},
   "source": [
    "# Advanced Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38a145d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d8c4194",
   "metadata": {},
   "source": [
    "Scatter Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89cbb30a",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(16,8))\n",
    "df.plot(x='MinTemp', y='MaxTemp', style='+')  \n",
    "plt.title('Scatter Plot of ')  \n",
    "plt.xlabel('X Label')  \n",
    "plt.ylabel('Y Label')  \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39dc17dc",
   "metadata": {},
   "source": [
    "Distribution Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6554f6a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(16,8))\n",
    "sns.histplot(df['numeric_column'].dropna(), kde = True, bins = 30).set_title('Distribition of numeric_column')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73ee5a09",
   "metadata": {},
   "source": [
    "Distribution of Target variable according to class (for classification)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74839c04",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(16,8))\n",
    "sns.set_style('darkgrid')\n",
    "sns.countplot(x='target', hue='class', data=df,palette='Dark2').set_title('Survived_by_Pclass')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69c27b88",
   "metadata": {},
   "source": [
    "Boxplot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91feb8e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#boxplot with age on y-axis and Passenger class on x-axis There will be as many boxplots as number of classes\n",
    "plt.figure(figsize=(16, 8))\n",
    "sns.set_style('darkgrid')\n",
    "sns.boxplot(x='Pclass', y='Age', data=df, palette='Dark2').set_title('Age_distribition_acording_to_pclass')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}