source/notebooks/sagemaker_fraud_detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Credit card fraud detector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this solution we will build the core of a credit card fraud detection system using SageMaker. We will start by training an anomaly detection algorithm, then proceed to train two XGBoost models for supervised training. To deal with the highly unbalanced data common in fraud detection, our first model will use re-weighting of the data, and the second will use re-sampling, using the popular SMOTE technique for oversampling the rare fraud data.\n",
    "\n",
    "Our solution includes an example of making calls to a REST API to simulate a real deployment, using AWS Lambda to trigger both the anomaly detection and XGBoost model.\n",
    "\n",
    "You can select Run->Run All from the menu to run all cells in Studio (or Cell->Run All in a SageMaker Notebook Instance)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: When running this notebook on SageMaker Studio, you should make sure the 'SageMaker JumpStart Data Science 1.0' image/kernel is used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, './src/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Investigate and process the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by reading in the credit card fraud data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "from package import config\n",
    "\n",
    "instance_type = 'ml.m5.large'\n",
    "s3 = boto3.resource('s3', region_name=config.AWS_REGION)\n",
    "object = s3.Object(f\"{config.SOLUTIONS_S3_BUCKET}-{config.AWS_REGION}\",f\"{config.SOLUTION_NAME}/data/creditcardfraud.zip\")\n",
    "object.download_file(\"creditcardfraud.zip\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zipfile import ZipFile\n",
    "\n",
    "with ZipFile('creditcardfraud.zip', 'r') as zf:\n",
    "    zf.extractall()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv('creditcard.csv', delimiter=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a peek at our data (we only show a subset of the columns in the table):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data.columns)\n",
    "data[['Time', 'V1', 'V2', 'V27', 'V28', 'Amount', 'Class']].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset contains\n",
    "only numerical features, because the original features have been transformed using PCA, to protect user privacy. As a result,\n",
    "the dataset contains 28 PCA components, V1-V28, and two features that haven't been transformed, _Amount_ and _Time_.\n",
    "_Amount_ refers to the transaction amount, and _Time_ is the seconds elapsed between any transaction in the data\n",
    "and the first transaction.\n",
    "\n",
    "The class column corresponds to whether or not a transaction is fraudulent. We see that the majority of data is non-fraudulent with only $492$ ($0.173\\%$) of the data corresponding to fraudulent examples, out of the total of 284,807 examples in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nonfrauds, frauds = data.groupby('Class').size()\n",
    "print('Number of frauds: ', frauds)\n",
    "print('Number of non-frauds: ', nonfrauds)\n",
    "print('Percentage of fradulent data:', 100.*frauds/(frauds + nonfrauds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We already know that the columns $V_i$ have been normalized to have $0$ mean and unit standard deviation as the result of a PCA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_columns = data.columns[:-1]\n",
    "label_column = data.columns[-1]\n",
    "\n",
    "features = data[feature_columns].values.astype('float32')\n",
    "labels = (data[label_column].values).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will prepare our data for loading and training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will split our dataset into a train and test to evaluate the performance of our models. It's important to do so _before_ any techniques meant to alleviate the class imbalance are used. This ensures that we don't leak information from the test set into the train set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    features, labels, test_size=0.1, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: If you are bringing your own data to this solution and they include categorical data, that have strings as values, you'd need to one-hot encode these values first using for example sklearn's [OneHotEncoder](https://scikit-learn.org/stable/modules/preprocessing.html#preprocessing-categorical-features), as XGBoost only supports numerical data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unsupervised Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a fraud detection scenario, commonly we will have very few labeled examples, and it's possible that labeling fraud takes a very long time. We would like then to extract information from the unlabeled data we have at hand as well. _Anomaly detection_ is a form of unsupervised learning where we try to identify anomalous examples based solely on their feature characteristics. Random Cut Forest is a state-of-the-art anomaly detection algorithm that is both accurate and scalable. We will train such a model on our training data and evaluate its performance on our test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sagemaker\n",
    "from package import config\n",
    "\n",
    "session = sagemaker.Session()\n",
    "bucket = config.MODEL_DATA_S3_BUCKET\n",
    "prefix = 'fraud-classifier'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import RandomCutForest\n",
    "\n",
    "# specify general training job information\n",
    "rcf = RandomCutForest(role=config.SAGEMAKER_IAM_ROLE,\n",
    "                      instance_count=1,\n",
    "                      instance_type=instance_type,\n",
    "                      data_location='s3://{}/{}/'.format(bucket, prefix),\n",
    "                      output_path='s3://{}/{}/output'.format(bucket, prefix),\n",
    "                      base_job_name=\"{}-rcf\".format(config.SOLUTION_PREFIX),\n",
    "                      num_samples_per_tree=512,\n",
    "                      num_trees=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we are ready to fit the model. The below cell should take around 5 minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "rcf.fit(rcf.record_set(X_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Host Random Cut Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Once we have a trained model we can deploy it and get some predictions for our test set. SageMaker will spin up an instance for us and deploy the model, the whole process should take around 10 minutes, you will see progress being made with each `-` and an exclamation point when the process is finished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rcf_predictor = rcf.deploy(\n",
    "    model_name=\"{}-rcf\".format(config.SOLUTION_PREFIX),\n",
    "    endpoint_name=\"{}-rcf\".format(config.SOLUTION_PREFIX),\n",
    "    initial_instance_count=1,\n",
    "    instance_type=instance_type)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sagemaker.serializers import CSVSerializer\n",
    "from sagemaker.deserializers import JSONDeserializer\n",
    " \n",
    "rcf_predictor.content_type = 'text/csv'\n",
    "rcf_predictor.serializer = CSVSerializer()\n",
    "rcf_predictor.accept = 'application/json'\n",
    "rcf_predictor.deserializer = JSONDeserializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Random Cut Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "With the model deployed, let's see how it performs in terms of separating fraudulent from legitimate transactions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_rcf(current_predictor, data, rows=500):\n",
    "    split_array = np.array_split(data, int(data.shape[0] / float(rows) + 1))\n",
    "    predictions = []\n",
    "    for array in split_array:\n",
    "        array_preds = [s['score'] for s in current_predictor.predict(array)['scores']]\n",
    "        predictions.append(array_preds)\n",
    "\n",
    "    return np.concatenate([np.array(batch) for batch in predictions])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "positives = X_test[y_test == 1]\n",
    "positives_scores = predict_rcf(rcf_predictor, positives)\n",
    "\n",
    "negatives = X_test[y_test == 0]\n",
    "negatives_scores = predict_rcf(rcf_predictor, negatives)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "sns.set(color_codes=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8, 6))\n",
    "sns.histplot(positives_scores, label='fraud', bins=20, ax=ax)\n",
    "sns.histplot(negatives_scores, label='not-fraud', bins=20, ax=ax)\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The unsupervised model already can achieve some separation between the classes, with higher anomaly scores being correlated to fraud."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supervised Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have gathered an adequate amount of labeled training data, we can use a supervised learning algorithm that discovers relationships between the features and the dependent class.\n",
    "\n",
    "We will use Gradient Boosted Trees as our model, as they have a proven track record, are highly scalable and can deal with missing data, reducing the need to pre-process datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare Data and Upload to S3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "First we copy the data to an in-memory buffer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import io\n",
    "import sklearn\n",
    "from sklearn.datasets import dump_svmlight_file   \n",
    "\n",
    "buf = io.BytesIO()\n",
    "\n",
    "sklearn.datasets.dump_svmlight_file(X_train, y_train, buf)\n",
    "buf.seek(0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we upload the data to S3 using boto3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "key = 'fraud-dataset'\n",
    "subdir = 'base'\n",
    "boto3.resource('s3', region_name=config.AWS_REGION).Bucket(bucket).Object(os.path.join(prefix, 'train', subdir, key)).upload_fileobj(buf)\n",
    "\n",
    "s3_train_data = 's3://{}/{}/train/{}/{}'.format(bucket, prefix, subdir, key)\n",
    "print('Uploaded training data location: {}'.format(s3_train_data))\n",
    "\n",
    "output_location = 's3://{}/{}/output'.format(bucket, prefix)\n",
    "print('Training artifacts will be uploaded to: {}'.format(output_location))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We can now train using SageMaker's built-in XGBoost algorithm. To specify the XGBoost algorithm, we use a utility function to obtain its URI. A complete list of built-in algorithms is found here: https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "# Get the XGBoost image URI\n",
    "xgboost_image_uri = sagemaker.image_uris.retrieve(\n",
    "    framework=\"xgboost\",\n",
    "    region=boto3.Session().region_name,\n",
    "    version=\"0.90-2\",\n",
    "    py_version=\"py3\",\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "SageMaker abstracts training via Estimators. We can pass the classifier and parameters along with hyperparameters to the estimator, and fit the estimator to the data in S3. An important parameter here is `scale_pos_weight` which scales the weights of the positive vs. negative class examples. This is crucial to do in an imbalanced dataset like the one we are using here, otherwise the majority class would dominate the learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from math import sqrt\n",
    "\n",
    "# Because the data set is so highly skewed, we set the scale position weight conservatively,\n",
    "# as sqrt(num_nonfraud/num_fraud).\n",
    "# Other recommendations for the scale_pos_weight are setting it to (num_nonfraud/num_fraud).\n",
    "scale_pos_weight = sqrt(np.count_nonzero(y_train==0)/np.count_nonzero(y_train))\n",
    "hyperparams = {\n",
    "    \"max_depth\":5,\n",
    "    \"subsample\":0.8,\n",
    "    \"num_round\":100,\n",
    "    \"eta\":0.2,\n",
    "    \"gamma\":4,\n",
    "    \"min_child_weight\":6,\n",
    "    \"silent\":0,\n",
    "    \"objective\":'binary:logistic',\n",
    "    \"eval_metric\":'auc',\n",
    "    \"scale_pos_weight\": scale_pos_weight\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us explain the hyper-parameters used above. The one that's very relevant for learning from skewed data is `scale_pos_weight`. This is a ratio that weighs the examples of the positive class (fraud) against the negative class (legitimate). Commonly this is set to `(num_nonfraud/num_fraud)`, but our data is exteremely skewed so we will set it to `sqrt(num_nonfraud/num_fraud)`.  For the data in this example, this would be `sqrt(284,807/492)` which would give our fraud examples a weight of ~24.\n",
    "\n",
    "The rest of the hyper-parameters are as follows:\n",
    "\n",
    "* `max_depth`: This is the maximum depth of the trees that will be built for our ensemble. A max depth of 5 will give us trees with up to 32 leaves. Note that tree size grows exponentially when increasing this parameter (`num_leaves=2^max_depth`), so a max depth of 10 would give us trees with 1024 leaves, which are likely to overfit.\n",
    "* `subsample`: The subsample ratio that we use to select a subset of the complete data to train each tree in the ensemble. With a value of 0.8, each tree is trained on a random sample containing 80% of the complete data. This is used to prevent overfitting.\n",
    "* `num_round`: This is the size of the ensemble. We will for 100 \"rounds\", each training round adding a new tree to the ensemble.\n",
    "* `eta`: This is the step size shrinkage applied at each update. This value will shrink the weights of new features to prevent overfitting.\n",
    "* `gamma`: This is the minimum loss reduction to reach before splitting a leaf. Splitting a leaf can sometimes have a small benefit, and splitting such leaves can lead to overfitting. By setting `gamma` to values larger than zero, we ensure that there should be at least some non-negligible amount of accuracy gain before splitting a leaf.\n",
    "* `min_child_weight`: This parameter has a similar effect to gamma, setting it to higher values means we'll wait until enough gain will be possible before splitting a leaf.\n",
    "* `objective`: We are doing binary classification, so we use a logistic loss objective.\n",
    "* `eval_metric`: Having a good evaluation metric is crucial when dealing with imbalanced data (see discussion below). We use AUC here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.estimator import Estimator\n",
    "\n",
    "clf = Estimator(\n",
    "    image_uri=xgboost_image_uri,\n",
    "    role=config.SAGEMAKER_IAM_ROLE,\n",
    "    instance_count=1,\n",
    "    instance_type=instance_type,\n",
    "    hyperparameters=hyperparams,\n",
    "    output_path=output_location,\n",
    "    sagemaker_session=session,\n",
    "    base_job_name=\"{}-xgb\".format(config.SOLUTION_PREFIX)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now fit our supervised training model, the call to fit below should take around 5 minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "clf.fit({'train': s3_train_data})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Host Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we deploy the estimator to and endpoint. As before progress will be indicated by `-`, and the deployment should be done after 10 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sagemaker.serializers import CSVSerializer\n",
    "\n",
    "predictor = clf.deploy(initial_instance_count=1,\n",
    "                       model_name=\"{}-xgb\".format(config.SOLUTION_PREFIX),\n",
    "                       endpoint_name=\"{}-xgb\".format(config.SOLUTION_PREFIX),\n",
    "                       instance_type=instance_type,\n",
    "                       serializer=CSVSerializer(),\n",
    "                       deserializer=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have trained the model we can use it to make predictions for the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Because we have a large test set, we call predict on smaller batches\n",
    "def predict(current_predictor, data, rows=500):\n",
    "    split_array = np.array_split(data, int(data.shape[0] / float(rows) + 1))\n",
    "    predictions = ''\n",
    "    for array in split_array:\n",
    "        predictions = ','.join([predictions, current_predictor.predict(array).decode('utf-8')])\n",
    "\n",
    "    return np.fromstring(predictions[1:], sep=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_preds = predict(predictor, X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We will use a few measures from the scikit-learn package to evaluate the performance of our model. When dealing with an imbalanced dataset, we need to choose metrics that take into account the frequency of each class in the data.\n",
    "\n",
    "Two such metrics are the [balanced accuracy score](https://scikit-learn.org/stable/modules/model_evaluation.html#balanced-accuracy-score), and [Cohen's Kappa](https://scikit-learn.org/stable/modules/model_evaluation.html#cohen-s-kappa)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import balanced_accuracy_score, cohen_kappa_score\n",
    "\n",
    "# scikit-learn expects 0/1 predictions, so we threshold our raw predictions\n",
    "y_preds = np.where(raw_preds > 0.5, 1, 0)\n",
    "print(\"Balanced accuracy = {}\".format(balanced_accuracy_score(y_test, y_preds)))\n",
    "print(\"Cohen's Kappa = {}\".format(cohen_kappa_score(y_test, y_preds)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can already see that our model performs very well in terms of both metrics, Cohen's Kappa scores above 0.8 are generally very favorable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apart from single-value metrics, it's also useful to look at metrics that indicate performance per class. A confusion matrix, and per-class precision, recall and f1-score can also provide more information about the model's performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "def plot_confusion_matrix(y_true, y_predicted):\n",
    "\n",
    "    cm  = confusion_matrix(y_true, y_predicted)\n",
    "    # Get the per-class normalized value for each cell\n",
    "    cm_norm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "    \n",
    "    # We color each cell according to its normalized value, annotate with exact counts.\n",
    "    ax = sns.heatmap(cm_norm, annot=cm, fmt=\"d\")\n",
    "    ax.set(xticklabels=[\"non-fraud\", \"fraud\"], yticklabels=[\"non-fraud\", \"fraud\"])\n",
    "    ax.set_ylim([0,2])\n",
    "    plt.title('Confusion Matrix')\n",
    "    plt.ylabel('Real Classes')\n",
    "    plt.xlabel('Predicted Classes')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "plot_confusion_matrix(y_test, y_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print(classification_report(\n",
    "    y_test, y_preds, target_names=['non-fraud', 'fraud']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Keep sending test traffic to the endpoint via lambda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:**\n",
    "If you're running the solution within Studio, uncomment the next cell to set `force_traffic_generation` to True, in order to run a simulation that sends traffic to the REST API deployed with this solution. **Before you do this you will need to attach the execute-api:Invoke action to the role that’s assumed by this notebook.** Here’s what the permission statement block you should add looks like: \n",
    "\n",
    "```json\n",
    "{\n",
    "    \"Action\": [\n",
    "        \"execute-api:Invoke\"\n",
    "    ],\n",
    "    \"Resource\": [\n",
    "        \"arn:aws:execute-api:*:*:*/*/POST/*\"\n",
    "    ],\n",
    "   \"Effect\": \"Allow\"\n",
    "}\n",
    "```\n",
    "\n",
    "See the APIGateway IAM documentation page for more info: https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonapigateway.html on managing execute-api   permissions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "force_traffic_generation = False\n",
    "#force_traffic_generation = True # If in Studio, set to True after you've updated you permissions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now show how we could use both of these models in a production system, using HTTP requests to an AWS Lambda function that invokes both the unsupervised and the supervised SageMaker endpoints.\n",
    "\n",
    "We create a background thread that will constantly create HTTP requests to invoke the Lambda, using our test data as input. See the included `generate_endpoint_traffic.py` file to see how that is done.\n",
    "The output will be logged to an S3 bucket through Kinesis, and you can also observe it in the Lambda function's CloudWatch logs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install aws_requests_auth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from threading import Thread\n",
    "from package.generate_endpoint_traffic import generate_traffic\n",
    "\n",
    "if config.SAGEMAKER_MODE == \"NotebookInstance\" or force_traffic_generation:\n",
    "    thread = Thread(target = generate_traffic, args=[np.copy(X_test)])\n",
    "    thread.start()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After a few minutes you can monitor the function invocations in the following URL:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display\n",
    "from IPython.display import Markdown as md\n",
    "if config.SAGEMAKER_MODE == \"NotebookInstance\" or force_traffic_generation:\n",
    "    print(\"hello\")\n",
    "    display(md(f\"[Link to Lambda Monitoring](https://{config.AWS_REGION}.console.aws.amazon.com/lambda/home?region={config.AWS_REGION}#/functions/{config.SOLUTION_PREFIX}-event-processor?tab=monitoring)\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SMOTE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a baseline model using XGBoost, we can try to see if sampling techniques that are designed specifically for imbalanced problems can improve the performance of the model.\n",
    "\n",
    "For that purpose we will be using the [imbalanced-learn](https://imbalanced-learn.readthedocs.io/en/stable/index.html) package that works well with scikit-learn. We have pre-installed the package for this kernel, but if you need it for a different Jupyter kernel you can install it by running `pip install --upgrade imbalanced-learn` within the conda environment you need.\n",
    "\n",
    "We will be using [Sythetic Minority Over-sampling](https://arxiv.org/abs/1106.1813) (SMOTE), which oversamples the minority class by interpolating new data points between existing ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install imblearn\n",
    "\n",
    "from imblearn.over_sampling import SMOTE\n",
    "\n",
    "smote = SMOTE(random_state=42)\n",
    "X_smote, y_smote = smote.fit_resample(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We can see that SMOTE has now balanced the two classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "print(sorted(Counter(y_smote).items()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We note that this is a case of extreme oversampling of the the minority class, we went from ~0.17% to 50%. An alternative would be to use a smaller resampling ratio, such as having one minority cl\n",
    "ass sample for every `sqrt(non_fraud/fraud)` majority samples, or using more advanced resampling techniques. See the [comparison](https://imbalanced-learn.readthedocs.io/en/stable/auto_examples/over-sampling/plot_comparison_over_sampling.html#sphx-glr-auto-examples-over-sampling-plot-comparison-over-sampling-py) provided by imbalanced-learn for more over-sampling options."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "In our case we'll use the SMOTE dataset we just created and upload it to S3 for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "smote_buf = io.BytesIO()\n",
    "\n",
    "# Dump the SMOTE data into a buffer\n",
    "sklearn.datasets.dump_svmlight_file(X_smote, y_smote, smote_buf)\n",
    "smote_buf.seek(0);\n",
    "\n",
    "# Upload from the buffer to S3\n",
    "key = 'fraud-dataset-smote'\n",
    "subdir = 'smote'\n",
    "boto3.resource('s3', region_name=config.AWS_REGION).Bucket(bucket).Object(os.path.join(prefix, 'train', subdir, key)).upload_fileobj(smote_buf)\n",
    "\n",
    "s3_smote_train_data = 's3://{}/{}/train/{}/{}'.format(bucket, prefix, subdir, key)\n",
    "print('Uploaded training data location: {}'.format(s3_smote_train_data))\n",
    "\n",
    "smote_output_location = 's3://{}/{}/smote-output'.format(bucket, prefix)\n",
    "print('Training artifacts will be uploaded to: {}'.format(smote_output_location))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# No need to scale weights after SMOTE resampling, so we remove that parameter\n",
    "hyperparams.pop(\"scale_pos_weight\", None)\n",
    "smote_xgb = sagemaker.estimator.Estimator(xgboost_image_uri,\n",
    "                                        role=config.SAGEMAKER_IAM_ROLE,\n",
    "                                        hyperparameters=hyperparams,\n",
    "                                        instance_count=1, \n",
    "                                        instance_type=instance_type,\n",
    "                                        output_path=smote_output_location,\n",
    "                                        sagemaker_session=session,\n",
    "                                        base_job_name=\"{}-xgb-smote\".format(config.SOLUTION_PREFIX))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We are now ready to fit the model, which should take around 5 minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "smote_xgb.fit({'train': s3_smote_train_data})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After fitting the model we can check its performance to compare it against the base XGBoost model. The deployment will take around 10 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sagemaker.serializers import CSVSerializer\n",
    "from sagemaker.deserializers import CSVDeserializer\n",
    "\n",
    "smote_predictor = smote_xgb.deploy(initial_instance_count=1,\n",
    "                                   model_name=\"{}-xgb-smote\".format(config.SOLUTION_PREFIX),\n",
    "                                   endpoint_name=\"{}-xgb-smote\".format(config.SOLUTION_PREFIX),\n",
    "                                   instance_type=instance_type)\n",
    "\n",
    "# Specify input and output formats.\n",
    "smote_predictor.content_type = 'text/csv'\n",
    "csv_serializer = CSVSerializer()\n",
    "smote_predictor.serializer = csv_serializer\n",
    "\n",
    "# Set the deserializer to handle the response from the inference endpoint\n",
    "#csv_deserializer = CSVDeserializer()\n",
    "#smote_predictor.deserializer = csv_deserializer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "smote_raw_preds = predict(smote_predictor, X_test)\n",
    "smote_preds = np.where(smote_raw_preds > 0.5, 1, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "print(\"Balanced accuracy = {}\".format(balanced_accuracy_score(y_test, smote_preds)))\n",
    "print(\"Cohen's Kappa = {}\".format(cohen_kappa_score(y_test, smote_preds)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "plot_confusion_matrix(y_test, smote_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(\n",
    "    y_test, smote_preds, target_names=['non-fraud', 'fraud']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Due to the randomness of XGBoost your results may vary, but overall, you should see a large increase in non-fraud cases being classified as fraud (false positives). The reason this happens is because SMOTE has oversampled the fraud class so much that it's increased its overlap in feature space with the non-fraud cases.\n",
    "Since Cohen's Kappa gives more weight to false positives than balanced accuracy does, the metric drops significantly, as does the precision and F1 score for fraud cases. However, we can bring a balance between the metrics again by adjusting our classification threshold."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far we've been using 0.5 as the threshold between labeling a point as fraud or not. We can try different thresholds to see if they affect the result of the classification. To evaluate we'll use the balanced accuracy and Cohen's Kappa metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "for thres in np.linspace(0.1, 0.9, num=9):\n",
    "    smote_thres_preds = np.where(smote_raw_preds > thres, 1, 0)\n",
    "    print(\"Threshold: {:.1f}\".format(thres))\n",
    "    print(\"Balanced accuracy = {:.3f}\".format(balanced_accuracy_score(y_test, smote_thres_preds)))\n",
    "    print(\"Cohen's Kappa = {:.3f}\\n\".format(cohen_kappa_score(y_test, smote_thres_preds)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that Cohen's Kappa keeps increasing along with the threshold, without a significant loss in balanced accuracy. This adds a useful knob to our model: We can keep a low threshold if we care more about not missing any fraudulent cases, or we can increase the threshold to try to minimize the number of false positives."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "\n",
    "We will leave the unsupervised and base XGBoost endpoints running at the end of this notebook so we can handle incoming event streams using the Lambda function. The solution will automatically clean up the endpoints when deleted, however, don't forget to ensure the prediction endpoints are deleted when you're done. You can do that at the Amazon SageMaker console in the Endpoints page. Or you can run `predictor_name.delete_endpoint()` here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment to clean up endpoints\n",
    "rcf_predictor.delete_model()\n",
    "rcf_predictor.delete_endpoint()\n",
    "predictor.delete_model()\n",
    "predictor.delete_endpoint()\n",
    "smote_predictor.delete_model()\n",
    "smote_predictor.delete_endpoint()\n",
    "sm_client = boto3.client('sagemaker', region_name=config.AWS_REGION)\n",
    "waiter = sm_client.get_waiter('endpoint_deleted')\n",
    "waiter.wait(EndpointName=\"{}-xgb-smote\".format(config.SOLUTION_PREFIX))\n",
    "waiter.wait(EndpointName=\"{}-xgb\".format(config.SOLUTION_PREFIX))\n",
    "waiter.wait(EndpointName=\"{}-rcf\".format(config.SOLUTION_PREFIX))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Data Acknowledgements\n",
    "\n",
    "The dataset used to demonstrated the fraud detection solution has been collected and analysed during a research collaboration of Worldline and the Machine Learning Group (http://mlg.ulb.ac.be) of ULB (Université Libre de Bruxelles) on big data mining and fraud detection. More details on current and past projects on related topics are available on https://www.researchgate.net/project/Fraud-detection-5 and the page of the [DefeatFraud](https://mlg.ulb.ac.be/wordpress/portfolio_page/defeatfraud-assessment-and-validation-of-deep-feature-engineering-and-learning-solutions-for-fraud-detection/) project\n",
    "We cite the following works:\n",
    "* Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015\n",
    "* Dal Pozzolo, Andrea; Caelen, Olivier; Le Borgne, Yann-Ael; Waterschoot, Serge; Bontempi, Gianluca. Learned lessons in credit card fraud detection from a practitioner perspective, Expert systems with applications,41,10,4915-4928,2014, Pergamon\n",
    "* Dal Pozzolo, Andrea; Boracchi, Giacomo; Caelen, Olivier; Alippi, Cesare; Bontempi, Gianluca. Credit card fraud detection: a realistic modeling and a novel learning strategy, IEEE transactions on neural networks and learning systems,29,8,3784-3797,2018,IEEE\n",
    "* Dal Pozzolo, Andrea Adaptive Machine learning for credit card fraud detection ULB MLG PhD thesis (supervised by G. Bontempi)\n",
    "* Carcillo, Fabrizio; Dal Pozzolo, Andrea; Le Borgne, Yann-Aël; Caelen, Olivier; Mazzer, Yannis; Bontempi, Gianluca. Scarff: a scalable framework for streaming credit card fraud detection with Spark, Information fusion,41, 182-194,2018,Elsevier\n",
    "* Carcillo, Fabrizio; Le Borgne, Yann-Aël; Caelen, Olivier; Bontempi, Gianluca. Streaming active learning strategies for real-life credit card fraud detection: assessment and visualization, International Journal of Data Science and Analytics, 5,4,285-300,2018,Springer International Publishing"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}