Part2_LendingClub_LogisticRegression_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3da88423",
   "metadata": {},
   "source": [
    "### Logistic regression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f4b7d4d",
   "metadata": {},
   "source": [
    "#### Logistic Regression - Base Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02f765ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr_model= LogisticRegression(random_state= 123) # Using default parameters\n",
    "lr_model.fit(train_x_fs, train_y) # Fit the model with the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6c983998",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----Logistic regression base model-----\n",
      "Accuracy: 0.6441996231773758\n",
      "Precision: 0.6429097063801604\n",
      "Recall: 0.6486978151510089\n",
      "F1 Score: 0.6457907915769633\n"
     ]
    }
   ],
   "source": [
    "model_performance(lr_model,'Logistic regression base model',test_x_fs,test_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 1.0,\n",
       " 'class_weight': None,\n",
       " 'dual': False,\n",
       " 'fit_intercept': True,\n",
       " 'intercept_scaling': 1,\n",
       " 'l1_ratio': None,\n",
       " 'max_iter': 100,\n",
       " 'multi_class': 'auto',\n",
       " 'n_jobs': None,\n",
       " 'penalty': 'l2',\n",
       " 'random_state': 123,\n",
       " 'solver': 'lbfgs',\n",
       " 'tol': 0.0001,\n",
       " 'verbose': 0,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72893e54",
   "metadata": {},
   "source": [
    "#### logistic regression - hyperparameter tuning using grid search method\n",
    "First use grid search to systematically go thru different combinations of parameters to determine the best combination that can give the best model performance. \n",
    "\n",
    "It is done by first defining the parameter grid.  For logistic regression, parameters such as C (inverse of regularization strength), penalty and solver will be included and tuned in this experiment.\n",
    "\n",
    "The GridSearchCV will cross validation (CV) to evaluate each combination and it is done by splitting the dataset into a number of subsets. Then, training and testing of the model will be performed using these varied combinations.\n",
    "\n",
    "The explaination for each parameters of Logreg can be found at skleanr documentation at\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2511e2e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the parameter grid for hyperparameter tuning of logistic regression model\n",
    "param_grid_lr = {\n",
    "    'penalty' : ['l1', 'l2', 'elasticnet', 'none'],\n",
    "    'C' : [0.01, 0.1, 1.0, 10, 100],\n",
    "    'solver' : ['lbfgs', 'liblinear', 'newton-cg', 'newton-cholesky', 'sag', 'saga']\n",
    "    } \n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "lr = LogisticRegression(random_state = 123)\n",
    "# Perform the grid search \n",
    "lr_grid_search = GridSearchCV(lr,\n",
    "                              param_grid=param_grid_lr,\n",
    "                              cv = 5,\n",
    "                              verbose = 0,\n",
    "                              n_jobs= -1,\n",
    "                              scoring='accuracy')\n",
    "\n",
    "# Fit the grid search to find the best parameters\n",
    "lr_grid_search.fit(train_x_fs, train_y)\n",
    "\n",
    "# Get and save the best param for lr\n",
    "best_lr_param = lr_grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9979205",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now fit the data into the model using best param from grid search\n",
    "lr_finetuned_model= LogisticRegression(**best_lr_param, random_state= 123) # Using default parameters\n",
    "lr_finetuned_model.fit(train_x_fs, train_y) # Fit the model with the data\n",
    "\n",
    "lr_finetuned_model = lr_grid_search.fit(train_x_fs,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "acb47195",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the result for future reference\n",
    "import pickle\n",
    "with open('/lr_finetuned_model.pkl', 'wb') as file:\n",
    "    pickle.dump(lr_finetuned_model, file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1cf6f537",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'C': 100, 'penalty': 'l2', 'solver': 'newton-cg'}\n"
     ]
    }
   ],
   "source": [
    "# Show the best combination of parameters for the best lr model \n",
    "lr_best_param = lr_finetuned_model.best_params_\n",
    "print(lr_best_param)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "fe28080b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----Logistic Regression Fine-tuned model-----\n",
      "Accuracy: 0.6439397455055085\n",
      "Precision: 0.6426416482707873\n",
      "Recall: 0.6484750607933768\n",
      "F1 Score: 0.6455451765205902\n"
     ]
    }
   ],
   "source": [
    "model_performance(lr_finetuned_model,'Logistic Regression Fine-tuned model',test_x_fs,test_y)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  },
  "toc-autonumbering": true,
  "toc-showcode": false,
  "toc-showmarkdowntxt": true,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}