Shopping Recommender/association-rules-mining-market-basket-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "450fd31f-55ff-435d-b420-8a8cd33ea7c9",
    "_uuid": "4c788bfac2c397099c4eaa99656a7a83036b3d45"
   },
   "source": [
    "# Association Rules Mining"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Association rule mining is commonly used to identify relationships between two products.\n",
    "\n",
    "It translates joint probabilities and frequencies found in historical orders into metrics called support, confidence and lift.\n",
    "\n",
    "#### 1. use python generator to generates a list of items for each order and create item pairs for that order, one pair at a time, then count the pairs frequency "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "7cf99a98-f1f2-4666-a39e-aa923c3a0c07",
    "_uuid": "aa1d4de4449ca66d8d315d8df6d20fc12c4f82e7"
   },
   "source": [
    "a. Get all possible item pairs for a given order \n",
    "       eg:  order 1:  apple, egg, milk   -->  item pairs: {apple, egg}, {apple, milk}, {egg, milk}\n",
    "            order 2:  egg, milk          -->  item pairs: {egg, milk}\n",
    "            \n",
    "b. Count the number of times each item pair appears\n",
    "       eg: {apple, egg}: 1\n",
    "           {apple, milk}: 1\n",
    "           {egg, milk}: 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "e7552ed1-fcfe-422b-a76c-f1bf91a2e064",
    "_uuid": "2bfe59ecf1564e278d73bee93e222372f0c01dd3"
   },
   "source": [
    "#### 2. Apriori Algorithm to setup the minumum occurence threshold, and count the qualified pairs. \n",
    "\n",
    "It does so using a \"bottom up\" approach, first identifying individual items that satisfy a minimum occurence threshold. It then extends the item set, adding one item at a time and checking if the resulting item set still satisfies the specified threshold.  The algorithm stops when there are no more items to add that meet the minimum occurrence requirement.  Here's an example of apriori in action, assuming a minimum occurence threshold of 3:\n",
    "\n",
    "\n",
    "    order 1: apple, egg, milk  \n",
    "    order 2: carrot, milk  \n",
    "    order 3: apple, egg, carrot\n",
    "    order 4: apple, egg\n",
    "    order 5: apple, carrot\n",
    "\n",
    "    \n",
    "    Iteration 1:  Count the number of times each item occurs   \n",
    "    item set      occurrence count    \n",
    "    {apple}              4   \n",
    "    {egg}                3   \n",
    "    {milk}               2   \n",
    "    {carrot}             2   \n",
    "\n",
    "    {milk} and {carrot} are eliminated because they do not meet the minimum occurrence threshold.\n",
    "\n",
    "\n",
    "    Iteration 2: Build item sets of size 2 using the remaining items from Iteration 1 \n",
    "                 (ie: apple, egg)  \n",
    "    item set           occurence count  \n",
    "    {apple, egg}             3  \n",
    "\n",
    "    Only {apple, egg} remains and the algorithm stops since there are no more items to add."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "9814b186-7467-4fbe-b8e8-34adf970cfdc",
    "_uuid": "516cfe4664132e59fbbcd1d4742643ab58f21b29"
   },
   "source": [
    "### Association Rules Mining\n",
    "For item sets of size 2, the association rules we will generate will be of the form {A} -> {B}.  One common application of these rules is in the domain of recommender systems, where customers who purchased item A are recommended item B.\n",
    "\n",
    "Here are 3 key metrics to consider when evaluating association rules:\n",
    "\n",
    "1. <b>support</b>  \n",
    "    This is an indication of how frequently the itemset appears in the dataset. In the example above, there are 5 orders in total \n",
    "    and {apple,egg} occurs in 3 of them, so: \n",
    "       \n",
    "                    support{apple,egg} = 3/5 or 60%\n",
    "    \n",
    "2. <b>confidence</b>  \n",
    "    confidence measures the how often that item B is purchased, given that item A was purchased.\n",
    "                    \n",
    "    In our example:  \n",
    "       \n",
    "                    confidence{apple->egg} = support{apple,egg} / support{apple}\n",
    "                                           = (3/5) / (4/5)\n",
    "                                           = 0.75 or 75%\n",
    "\n",
    "    A confidence value of 0.75 implies that out of all orders that contain apple, 75% of them also contain egg. Note that the confidence measure is directional: if we look in the opposite direction (ie: egg->apple): \n",
    "       \n",
    "                    confidence{egg->apple} = support{apple,egg} / support{egg}\n",
    "                                           = (3/5) / (3/5)\n",
    "                                           = 1 or 100%  \n",
    "                                           \n",
    "    Here we see that all of the orders that contain egg also contain apple.  it it mean a relationship between each other or simply by chance?\n",
    "    \n",
    "3. <b>lift</b>  \n",
    "    lift indicates whether there is a relationship between A and B, or whether the two items are occuring together in the same orders simply by chance (ie: at random).  \n",
    "    In our example, we compute lift as follows:\n",
    "    \n",
    "         lift{apple,egg} = lift{egg,apple} = support{apple,egg} / (support{apple} * support{egg})\n",
    "                         = (3/5) / (4/5 * 3/5) \n",
    "                         = 1.25    \n",
    "                         \n",
    "    In summary, lift can take on the following values:\n",
    "    \n",
    "        * lift = 1 implies no relationship between A and B. \n",
    "          (ie: A and B occur together only by chance)\n",
    "      \n",
    "        * lift > 1 implies that there is a positive relationship between A and B.\n",
    "          (ie:  A and B occur together more often than random)\n",
    "    \n",
    "        * lift < 1 implies that there is a negative relationship between A and B.\n",
    "          (ie:  A and B occur together less often than random)\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "6187d6e8-3a64-4802-b5a5-c8dcf828a0b3",
    "_uuid": "0e487b731f49e05e7f81490e36894ade9fdd3f50"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import sys\n",
    "from itertools import combinations, groupby\n",
    "from collections import Counter\n",
    "from IPython.display import display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "ebb38b54-3250-4636-a53e-6c422f4456bf",
    "_uuid": "009e16e2fcdf0810f77627d1c6439ac5bcc17377"
   },
   "outputs": [],
   "source": [
    "# Function that returns the size of an object in MB\n",
    "def size(obj):\n",
    "    return \"{0:.2f} MB\".format(sys.getsizeof(obj) / (1000 * 1000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "ddefe399-1b17-415b-bad2-b4dfcc9c734f",
    "_uuid": "d92f5eb045da7e74b902fda4ba430b247100bd6f"
   },
   "source": [
    "### Part 1:  Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "611c8c1a-69df-4e82-8e7f-509ed8687af3",
    "_uuid": "98ca92d66cc1aa6d545d9e414971c718b0801e70"
   },
   "source": [
    "#### A. Load order  data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import data\n",
    "products = pd.read_csv('products.csv')\n",
    "aisles = pd.read_csv('aisles.csv')\n",
    "departments = pd.read_csv('departments.csv')\n",
    "clusters = pd.read_csv('customer_group_Kmeans.csv')\n",
    "order_detils = pd.read_csv('orders.csv')\n",
    "order_products_prior = pd.read_csv('order_products__prior.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>order_id</th>\n",
       "      <th>user_id</th>\n",
       "      <th>eval_set</th>\n",
       "      <th>order_number</th>\n",
       "      <th>order_dow</th>\n",
       "      <th>order_hour_of_day</th>\n",
       "      <th>days_since_prior_order</th>\n",
       "      <th>customer type</th>\n",
       "      <th>product_id</th>\n",
       "      <th>product_name</th>\n",
       "      <th>aisle_id</th>\n",
       "      <th>department_id</th>\n",
       "      <th>aisle</th>\n",
       "      <th>department</th>\n",
       "      <th>add_to_cart_order</th>\n",
       "      <th>reordered</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>prior</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>12427</td>\n",
       "      <td>Original Beef Jerky</td>\n",
       "      <td>23</td>\n",
       "      <td>19</td>\n",
       "      <td>popcorn jerky</td>\n",
       "      <td>snacks</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>prior</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>26088</td>\n",
       "      <td>Aged White Cheddar Popcorn</td>\n",
       "      <td>23</td>\n",
       "      <td>19</td>\n",
       "      <td>popcorn jerky</td>\n",
       "      <td>snacks</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>prior</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>196</td>\n",
       "      <td>Soda</td>\n",
       "      <td>77</td>\n",
       "      <td>7</td>\n",
       "      <td>soft drinks</td>\n",
       "      <td>beverages</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>prior</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>14084</td>\n",
       "      <td>Organic Unsweetened Vanilla Almond Milk</td>\n",
       "      <td>91</td>\n",
       "      <td>16</td>\n",
       "      <td>soy lactosefree</td>\n",
       "      <td>dairy eggs</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>prior</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>26405</td>\n",
       "      <td>XL Pick-A-Size Paper Towel Rolls</td>\n",
       "      <td>54</td>\n",
       "      <td>17</td>\n",
       "      <td>paper goods</td>\n",
       "      <td>household</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   order_id  user_id eval_set  order_number  order_dow  order_hour_of_day  \\\n",
       "0   2539329        1    prior             1          2                  8   \n",
       "1   2539329        1    prior             1          2                  8   \n",
       "2   2539329        1    prior             1          2                  8   \n",
       "3   2539329        1    prior             1          2                  8   \n",
       "4   2539329        1    prior             1          2                  8   \n",
       "\n",
       "   days_since_prior_order  customer type  product_id  \\\n",
       "0                     NaN              0       12427   \n",
       "1                     NaN              0       26088   \n",
       "2                     NaN              0         196   \n",
       "3                     NaN              0       14084   \n",
       "4                     NaN              0       26405   \n",
       "\n",
       "                              product_name  aisle_id  department_id  \\\n",
       "0                      Original Beef Jerky        23             19   \n",
       "1               Aged White Cheddar Popcorn        23             19   \n",
       "2                                     Soda        77              7   \n",
       "3  Organic Unsweetened Vanilla Almond Milk        91             16   \n",
       "4         XL Pick-A-Size Paper Towel Rolls        54             17   \n",
       "\n",
       "             aisle  department  add_to_cart_order  reordered  \n",
       "0    popcorn jerky      snacks                  3          0  \n",
       "1    popcorn jerky      snacks                  4          0  \n",
       "2      soft drinks   beverages                  1          0  \n",
       "3  soy lactosefree  dairy eggs                  2          0  \n",
       "4      paper goods   household                  5          0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# combine datasets\n",
    "user_merged = pd.merge(order_detils, clusters, on='user_id')\n",
    "\n",
    "product_aisle = pd.merge(products, aisles, on='aisle_id')\n",
    "product_department = pd.merge(product_aisle, departments, on='department_id')\n",
    "\n",
    "order_products = pd.merge(product_department, order_products_prior, on='product_id')\n",
    "\n",
    "orders_all= pd.merge(user_merged, order_products, on='order_id')\n",
    "orders_all.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>order_id</th>\n",
       "      <th>user_id</th>\n",
       "      <th>customer type</th>\n",
       "      <th>product_id</th>\n",
       "      <th>aisle_id</th>\n",
       "      <th>department_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>12427</td>\n",
       "      <td>23</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>26088</td>\n",
       "      <td>23</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>196</td>\n",
       "      <td>77</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>14084</td>\n",
       "      <td>91</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>26405</td>\n",
       "      <td>54</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32434484</td>\n",
       "      <td>2977660</td>\n",
       "      <td>206209</td>\n",
       "      <td>0</td>\n",
       "      <td>9405</td>\n",
       "      <td>91</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32434485</td>\n",
       "      <td>2977660</td>\n",
       "      <td>206209</td>\n",
       "      <td>0</td>\n",
       "      <td>16168</td>\n",
       "      <td>86</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32434486</td>\n",
       "      <td>2977660</td>\n",
       "      <td>206209</td>\n",
       "      <td>0</td>\n",
       "      <td>14197</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32434487</td>\n",
       "      <td>2977660</td>\n",
       "      <td>206209</td>\n",
       "      <td>0</td>\n",
       "      <td>39216</td>\n",
       "      <td>121</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32434488</td>\n",
       "      <td>2977660</td>\n",
       "      <td>206209</td>\n",
       "      <td>0</td>\n",
       "      <td>24852</td>\n",
       "      <td>24</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>32434489 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          order_id  user_id  customer type  product_id  aisle_id  \\\n",
       "0          2539329        1              0       12427        23   \n",
       "1          2539329        1              0       26088        23   \n",
       "2          2539329        1              0         196        77   \n",
       "3          2539329        1              0       14084        91   \n",
       "4          2539329        1              0       26405        54   \n",
       "...            ...      ...            ...         ...       ...   \n",
       "32434484   2977660   206209              0        9405        91   \n",
       "32434485   2977660   206209              0       16168        86   \n",
       "32434486   2977660   206209              0       14197         9   \n",
       "32434487   2977660   206209              0       39216       121   \n",
       "32434488   2977660   206209              0       24852        24   \n",
       "\n",
       "          department_id  \n",
       "0                    19  \n",
       "1                    19  \n",
       "2                     7  \n",
       "3                    16  \n",
       "4                    17  \n",
       "...                 ...  \n",
       "32434484             16  \n",
       "32434485             16  \n",
       "32434486              9  \n",
       "32434487             14  \n",
       "32434488              4  \n",
       "\n",
       "[32434489 rows x 6 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# setup datasets\n",
    "orders = orders_all[[\"order_id\", \"user_id\", \"customer type\", \"product_id\", \"aisle_id\", \"department_id\"]]\n",
    "orders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# orders.to_csv('merged_order_cluster.csv', encoding='utf-8')\n",
    "orders.to_csv('merged_order_cluster_all.csv', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read files\n",
    "orders = pd.read_csv('merged_order_cluster.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>order_id</th>\n",
       "      <th>user_id</th>\n",
       "      <th>customer type</th>\n",
       "      <th>product_id</th>\n",
       "      <th>aisle_id</th>\n",
       "      <th>department_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>3972198</td>\n",
       "      <td>3421083</td>\n",
       "      <td>25247</td>\n",
       "      <td>0</td>\n",
       "      <td>5020</td>\n",
       "      <td>3</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3972202</td>\n",
       "      <td>3421083</td>\n",
       "      <td>25247</td>\n",
       "      <td>0</td>\n",
       "      <td>39678</td>\n",
       "      <td>74</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3972204</td>\n",
       "      <td>3421083</td>\n",
       "      <td>25247</td>\n",
       "      <td>0</td>\n",
       "      <td>21162</td>\n",
       "      <td>92</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3972205</td>\n",
       "      <td>3421083</td>\n",
       "      <td>25247</td>\n",
       "      <td>0</td>\n",
       "      <td>35211</td>\n",
       "      <td>92</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3972206</td>\n",
       "      <td>3421083</td>\n",
       "      <td>25247</td>\n",
       "      <td>0</td>\n",
       "      <td>45309</td>\n",
       "      <td>92</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31824592</td>\n",
       "      <td>2</td>\n",
       "      <td>202279</td>\n",
       "      <td>0</td>\n",
       "      <td>17794</td>\n",
       "      <td>83</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31824591</td>\n",
       "      <td>2</td>\n",
       "      <td>202279</td>\n",
       "      <td>0</td>\n",
       "      <td>43668</td>\n",
       "      <td>123</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31824590</td>\n",
       "      <td>2</td>\n",
       "      <td>202279</td>\n",
       "      <td>0</td>\n",
       "      <td>33120</td>\n",
       "      <td>86</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31824588</td>\n",
       "      <td>2</td>\n",
       "      <td>202279</td>\n",
       "      <td>0</td>\n",
       "      <td>45918</td>\n",
       "      <td>19</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31824587</td>\n",
       "      <td>2</td>\n",
       "      <td>202279</td>\n",
       "      <td>0</td>\n",
       "      <td>1819</td>\n",
       "      <td>88</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>29084126 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          order_id  user_id  customer type  product_id  aisle_id  \\\n",
       "3972198    3421083    25247              0        5020         3   \n",
       "3972202    3421083    25247              0       39678        74   \n",
       "3972204    3421083    25247              0       21162        92   \n",
       "3972205    3421083    25247              0       35211        92   \n",
       "3972206    3421083    25247              0       45309        92   \n",
       "...            ...      ...            ...         ...       ...   \n",
       "31824592         2   202279              0       17794        83   \n",
       "31824591         2   202279              0       43668       123   \n",
       "31824590         2   202279              0       33120        86   \n",
       "31824588         2   202279              0       45918        19   \n",
       "31824587         2   202279              0        1819        88   \n",
       "\n",
       "          department_id  \n",
       "3972198              19  \n",
       "3972202              17  \n",
       "3972204              18  \n",
       "3972205              18  \n",
       "3972206              18  \n",
       "...                 ...  \n",
       "31824592              4  \n",
       "31824591              4  \n",
       "31824590             16  \n",
       "31824588             13  \n",
       "31824587             13  \n",
       "\n",
       "[29084126 rows x 6 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select the customer clusters: 0, 1, 2 ,3 , 4  \n",
    "orders = orders[orders[\"customer type\"] == 0] \n",
    "orders.sort_values(\"order_id\", ascending = False) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "_cell_guid": "17379a10-59fc-40ad-8dbb-13c27889094e",
    "_uuid": "7f3836b73419e5f8957f9d42500c40ee8f457f0f",
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "orders -- dimensions: (29084126, 6);   size: 1628.71 MB\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>order_id</th>\n",
       "      <th>user_id</th>\n",
       "      <th>customer type</th>\n",
       "      <th>product_id</th>\n",
       "      <th>aisle_id</th>\n",
       "      <th>department_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>12427</td>\n",
       "      <td>23</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>26088</td>\n",
       "      <td>23</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>196</td>\n",
       "      <td>77</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>14084</td>\n",
       "      <td>91</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2539329</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>26405</td>\n",
       "      <td>54</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   order_id  user_id  customer type  product_id  aisle_id  department_id\n",
       "0   2539329        1              0       12427        23             19\n",
       "1   2539329        1              0       26088        23             19\n",
       "2   2539329        1              0         196        77              7\n",
       "3   2539329        1              0       14084        91             16\n",
       "4   2539329        1              0       26405        54             17"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# orders = pd.read_csv('order_products__prior.csv')\n",
    "print('orders -- dimensions: {0};   size: {1}'.format(orders.shape, size(orders)))\n",
    "display(orders.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "92ac9b55-e7b7-4cfe-8d70-4ab601226a88",
    "_uuid": "94fdf6593d6d766c862f9d03e1172949ef9c6e8b"
   },
   "source": [
    "#### B. Convert order data into format expected by the association rules function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "_cell_guid": "e9736090-913e-4963-b381-7b24c6a70d00",
    "_uuid": "072c8a04a5f9488573f5082c01df7e809868dd15",
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "order_id\n",
       "2539329    12427\n",
       "2539329    26088\n",
       "2539329      196\n",
       "2539329    14084\n",
       "2539329    26405\n",
       "2398795    12427\n",
       "2398795    26088\n",
       "2398795    10258\n",
       "2398795      196\n",
       "2398795    13032\n",
       "Name: item_id, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert from DataFrame to a Series, with order_id as index and item_id as value\n",
    "orders = orders.set_index('order_id')['product_id'].rename('item_id')\n",
    "display(orders.head(10))\n",
    "type(orders)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "e3f20905-3b6a-4ff5-bdc2-ae8b92c069e5",
    "_uuid": "5cd89dbd29af1eb5d5d3cca36eb8e11b26431164"
   },
   "source": [
    "#### C. Display summary statistics for order data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "_cell_guid": "37d2f26c-6398-4544-8ac1-0afa21d066a3",
    "_uuid": "0950199479ff1e56263f66e4b171ff45379e03b8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dimensions: (29084126,);   size: 465.35 MB;   unique_orders: 2938109;   unique_items: 49652\n"
     ]
    }
   ],
   "source": [
    "print('dimensions: {0};   size: {1};   unique_orders: {2};   unique_items: {3}'\n",
    "      .format(orders.shape, size(orders), len(orders.index.unique()), len(orders.value_counts())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "afa98af1-9a11-43ce-8e12-014c6e724075",
    "_uuid": "697a072213997ed4e7c8affbfc0b1ea736113aac"
   },
   "source": [
    "### Part 2: Association Rules Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "7efcec16-16a8-407f-bd9a-6d53c491c01a",
    "_uuid": "1f5345c3bff3b50a35fec45ffa81838e24d134d3"
   },
   "source": [
    "#### A. Helper functions to the main association rules function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "_cell_guid": "95729dab-8f56-4ec4-8d12-15ed38eff18d",
    "_uuid": "55ba9d736129af7561f275d3dc29d8a3b2dfd30e",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Returns frequency counts for items and item pairs\n",
    "def freq(iterable):\n",
    "    if type(iterable) == pd.core.series.Series:\n",
    "        return iterable.value_counts().rename(\"freq\")\n",
    "    else: \n",
    "        return pd.Series(Counter(iterable)).rename(\"freq\")\n",
    "\n",
    "    \n",
    "# Returns number of unique orders\n",
    "def order_count(order_item):\n",
    "    return len(set(order_item.index))\n",
    "\n",
    "\n",
    "# Returns generator that yields item pairs, one at a time\n",
    "def get_item_pairs(order_item):\n",
    "    order_item = order_item.reset_index().values\n",
    "    for order_id, order_object in groupby(order_item, lambda x: x[0]):\n",
    "        item_list = [item[1] for item in order_object]\n",
    "              \n",
    "        for item_pair in combinations(item_list, 2):\n",
    "            yield item_pair\n",
    "            \n",
    "\n",
    "# Returns frequency and support associated with item\n",
    "def merge_item_stats(item_pairs, item_stats):\n",
    "    return (item_pairs\n",
    "                .merge(item_stats.rename(columns={'freq': 'freqA', 'support': 'supportA'}), left_on='item_A', right_index=True)\n",
    "                .merge(item_stats.rename(columns={'freq': 'freqB', 'support': 'supportB'}), left_on='item_B', right_index=True))\n",
    "\n",
    "\n",
    "# Returns name associated with item\n",
    "def merge_item_name(rules, item_name):\n",
    "    columns = ['itemA','itemB','freqAB','supportAB','freqA','supportA','freqB','supportB', \n",
    "               'confidenceAtoB','confidenceBtoA','lift']\n",
    "    rules = (rules\n",
    "                .merge(item_name.rename(columns={'item_name': 'itemA'}), left_on='item_A', right_on='item_id')\n",
    "                .merge(item_name.rename(columns={'item_name': 'itemB'}), left_on='item_B', right_on='item_id'))\n",
    "    return rules[columns]               "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>get_item_pairs()</i> generates a list of items for each order and produces item pairs for that order, one pair at a time.  The first item pair is passed to Counter which keeps track of the number of times an item pair occurs.  The next item pair is taken, and again, passed to Counter.  This process continues until there are no more item pairs left.  With this approach, we end up not using much memory as item pairs are discarded after the count is updated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "17b82fe5-464d-422b-8b4e-550551b54e58",
    "_uuid": "fadb8d35353ee366b37d22f051f6c5d670704b96"
   },
   "source": [
    "#### B. Association rules function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "_cell_guid": "e94b8466-b7db-4feb-b27a-b65e37626bac",
    "_uuid": "ebde898bf77e55b5ef8643d89ce961c1aa8ec08a"
   },
   "outputs": [],
   "source": [
    "def association_rules(order_item, min_support):\n",
    "\n",
    "    print(\"Starting order_item: {:22d}\".format(len(order_item)))\n",
    "\n",
    "\n",
    "    # Calculate item frequency and support\n",
    "    item_stats             = freq(order_item).to_frame(\"freq\")\n",
    "    item_stats['support']  = item_stats['freq'] / order_count(order_item) * 100\n",
    "    \n",
    "\n",
    "\n",
    "    # Filter from order_item items below min support \n",
    "    qualifying_items       = item_stats[item_stats['support'] >= min_support].index\n",
    "    order_item             = order_item[order_item.isin(qualifying_items)]\n",
    "\n",
    "    print(\"(qualifying_items)Items with support >= {}: {:15d}\".format(min_support, len(qualifying_items)))\n",
    "    print(\"Items above min support, Remaining order_item: {:21d}\".format(len(order_item)))\n",
    "\n",
    "\n",
    "    # Filter from order_item orders with less than 2 items\n",
    "    order_size             = freq(order_item.index)\n",
    "    qualifying_orders      = order_size[order_size >= 2].index\n",
    "    order_item             = order_item[order_item.index.isin(qualifying_orders)]\n",
    "\n",
    "    print(\"(qualifying_orders)Remaining orders with 2+ items: {:11d}\".format(len(qualifying_orders)))\n",
    "    print(\"orders more than 2 items, Remaining order_item: {:21d}\".format(len(order_item)))\n",
    "\n",
    "\n",
    "    # Recalculate item frequency and support\n",
    "    item_stats             = freq(order_item).to_frame(\"freq\")\n",
    "    item_stats['support']  = item_stats['freq'] / order_count(order_item) * 100\n",
    "\n",
    "\n",
    "    # Get item pairs generator\n",
    "    item_pair_gen          = get_item_pairs(order_item)\n",
    "\n",
    "\n",
    "    # Calculate item pair frequency and support\n",
    "    item_pairs              = freq(item_pair_gen).to_frame(\"freqAB\")\n",
    "    item_pairs['supportAB'] = item_pairs['freqAB'] / len(qualifying_orders) * 100\n",
    "\n",
    "    print(\"use paires generator, Item pairs: {:31d}\".format(len(item_pairs)))\n",
    "\n",
    "\n",
    "    # Filter from item_pairs those below min support\n",
    "    item_pairs              = item_pairs[item_pairs['supportAB'] >= min_support]\n",
    "\n",
    "    print(\"Item pairs with support >= {}: {:10d}\\n\".format(min_support, len(item_pairs)))\n",
    "\n",
    "\n",
    "    # Create table of association rules and compute relevant metrics\n",
    "    item_pairs = item_pairs.reset_index().rename(columns={'level_0': 'item_A', 'level_1': 'item_B'})\n",
    "    item_pairs = merge_item_stats(item_pairs, item_stats)\n",
    "    \n",
    "    item_pairs['confidenceAtoB'] = item_pairs['supportAB'] / item_pairs['supportA']\n",
    "    item_pairs['confidenceBtoA'] = item_pairs['supportAB'] / item_pairs['supportB']\n",
    "    item_pairs['lift']           = item_pairs['supportAB'] / (item_pairs['supportA'] * item_pairs['supportB'])\n",
    "    \n",
    "    \n",
    "    # Return association rules sorted by lift in descending order\n",
    "    return item_pairs.sort_values('lift', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "e608b37c-1872-4632-b0ec-307b3ff19a0c",
    "_uuid": "9cfdef69fb7abc9927cbab401b1a6cab14b494f9"
   },
   "source": [
    "### Part 3:  Association Rules Mining"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "_cell_guid": "09be74c2-1a24-48ff-bbcf-ca701b0abe4e",
    "_uuid": "bc848efd05034abd042467e15388d8a179e6b8a8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting order_item:               29084126\n",
      "(qualifying_items)Items with support >= 0.01:           10887\n",
      "Items above min support, Remaining order_item:              26728412\n",
      "(qualifying_orders)Remaining orders with 2+ items:     2747746\n",
      "orders more than 2 items, Remaining order_item:              26557711\n",
      "use paires generator, Item pairs:                        19529742\n",
      "Item pairs with support >= 0.01:      58186\n",
      "\n",
      "Wall time: 3min 27s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rules = association_rules(orders, 0.01)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results Without clustering \n",
    "Starting order_item:               32434489\n",
    "\n",
    "Items with support >= 0.01:           10906\n",
    "\n",
    "Items above min support, Remaining order_item:              29843570\n",
    "\n",
    "Remaining orders with 2+ items:     3013325\n",
    "\n",
    "orders more than 2 items, Remaining order_item:              29662716\n",
    "\n",
    "use paires generator, Item pairs:                        30622410\n",
    "\n",
    "Item pairs with support >= 0.01:      48751\n",
    "\n",
    "\n",
    "Wall time: 4min 20s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results With clustering = 0\n",
    "\n",
    "Starting order_item:               29084126\n",
    "\n",
    "**(qualifying_items)Items with support >= 0.01:           10887**\n",
    "\n",
    "Items above min support, Remaining order_item:              26728412\n",
    "\n",
    "(qualifying_orders)Remaining orders with 2+ items:     2747746\n",
    "\n",
    "orders more than 2 items, Remaining order_item:              26557711\n",
    "\n",
    "use paires generator, Item pairs:                        28786889\n",
    "\n",
    "**Item pairs with support >= 0.01:      46023**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results With clustering = 1\n",
    "\n",
    "Starting order_item:                 560066\n",
    "\n",
    "**(qualifying_items)Items with support >= 0.01:            9508**\n",
    "\n",
    "Items above min support, Remaining order_item:                548294\n",
    "\n",
    "(qualifying_orders)Remaining orders with 2+ items:       33776\n",
    "\n",
    "orders more than 2 items, Remaining order_item:                547989\n",
    "\n",
    "use paires generator, Item pairs:                         2330965\n",
    "\n",
    "**Item pairs with support >= 0.01:     340757**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results With clustering = 2\n",
    "\n",
    "Starting order_item:                2172599\n",
    "\n",
    "(qualifying_items)Items with support >= 0.01:           10377\n",
    "\n",
    "Items above min support, Remaining order_item:               2086915\n",
    "\n",
    "(qualifying_orders)Remaining orders with 2+ items:      153312\n",
    "\n",
    "orders more than 2 items, Remaining order_item:               2083974\n",
    "\n",
    "use paires generator, Item pairs:                         6029162\n",
    "\n",
    "**Item pairs with support >= 0.01:     143392**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results With clustering = 3\n",
    "\n",
    "Starting order_item:                 286014\n",
    "\n",
    "(qualifying_items)Items with support >= 0.01:            8281\n",
    "\n",
    "Items above min support, Remaining order_item:                274323\n",
    "\n",
    "(qualifying_orders)Remaining orders with 2+ items:       31546\n",
    "\n",
    "orders more than 2 items, Remaining order_item:                272554\n",
    "\n",
    "use paires generator, Item pairs:                         1035147\n",
    "\n",
    "**Item pairs with support >= 0.01:      76287**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results With clustering = 4\n",
    "\n",
    "Starting order_item:                 331684\n",
    "\n",
    "(qualifying_items)Items with support >= 0.01:            1786\n",
    "\n",
    "Items above min support, Remaining order_item:                327019\n",
    "\n",
    "(qualifying_orders)Remaining orders with 2+ items:       48592\n",
    "\n",
    "orders more than 2 items, Remaining order_item:                322861\n",
    "\n",
    "use paires generator, Item pairs:                          228222\n",
    "\n",
    "**Item pairs with support >= 0.01:      53491**\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "_cell_guid": "af1e9b6b-6191-43aa-a0b8-40a3abf49073",
    "_uuid": "b94f6796164b5915d923b71117e9322e7a4afa41"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>itemA</th>\n",
       "      <th>itemB</th>\n",
       "      <th>freqAB</th>\n",
       "      <th>supportAB</th>\n",
       "      <th>freqA</th>\n",
       "      <th>supportA</th>\n",
       "      <th>freqB</th>\n",
       "      <th>supportB</th>\n",
       "      <th>confidenceAtoB</th>\n",
       "      <th>confidenceBtoA</th>\n",
       "      <th>lift</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Dairy Free Greek Yogurt Strawberry</td>\n",
       "      <td>Dairy Free Greek Yogurt Blueberry</td>\n",
       "      <td>293</td>\n",
       "      <td>0.010663</td>\n",
       "      <td>578</td>\n",
       "      <td>0.021035</td>\n",
       "      <td>531</td>\n",
       "      <td>0.019325</td>\n",
       "      <td>0.506920</td>\n",
       "      <td>0.551789</td>\n",
       "      <td>26.231423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>Grassfed Whole Milk Strawberry Yogurt</td>\n",
       "      <td>Organic Strawberry Grassfed Whole Milk Yogurt</td>\n",
       "      <td>276</td>\n",
       "      <td>0.010045</td>\n",
       "      <td>561</td>\n",
       "      <td>0.020417</td>\n",
       "      <td>592</td>\n",
       "      <td>0.021545</td>\n",
       "      <td>0.491979</td>\n",
       "      <td>0.466216</td>\n",
       "      <td>22.835004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Organic Cashew Nondairy Blueberry Yogurt</td>\n",
       "      <td>Organic Nondairy Strawberry Cashew Yogurt</td>\n",
       "      <td>363</td>\n",
       "      <td>0.013211</td>\n",
       "      <td>576</td>\n",
       "      <td>0.020963</td>\n",
       "      <td>851</td>\n",
       "      <td>0.030971</td>\n",
       "      <td>0.630208</td>\n",
       "      <td>0.426557</td>\n",
       "      <td>20.348442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>Organic Strawberry Chia Lowfat 2% Cottage Cheese</td>\n",
       "      <td>Organic Cottage Cheese Blueberry Acai Chia</td>\n",
       "      <td>489</td>\n",
       "      <td>0.017796</td>\n",
       "      <td>1089</td>\n",
       "      <td>0.039632</td>\n",
       "      <td>766</td>\n",
       "      <td>0.027877</td>\n",
       "      <td>0.449036</td>\n",
       "      <td>0.638381</td>\n",
       "      <td>16.107524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>Hair Shampoos</td>\n",
       "      <td>Moroccan Argan Oil + Argan Stem Cell Triple Mo...</td>\n",
       "      <td>302</td>\n",
       "      <td>0.010991</td>\n",
       "      <td>634</td>\n",
       "      <td>0.023073</td>\n",
       "      <td>843</td>\n",
       "      <td>0.030680</td>\n",
       "      <td>0.476341</td>\n",
       "      <td>0.358244</td>\n",
       "      <td>15.526254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8314</td>\n",
       "      <td>Small Hass Avocado</td>\n",
       "      <td>Organic Hass Avocado</td>\n",
       "      <td>376</td>\n",
       "      <td>0.013684</td>\n",
       "      <td>44260</td>\n",
       "      <td>1.610775</td>\n",
       "      <td>186186</td>\n",
       "      <td>6.775954</td>\n",
       "      <td>0.008495</td>\n",
       "      <td>0.002019</td>\n",
       "      <td>0.001254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26549</td>\n",
       "      <td>Organic Cucumber</td>\n",
       "      <td>Cucumber Kirby</td>\n",
       "      <td>290</td>\n",
       "      <td>0.010554</td>\n",
       "      <td>72064</td>\n",
       "      <td>2.622659</td>\n",
       "      <td>90338</td>\n",
       "      <td>3.287713</td>\n",
       "      <td>0.004024</td>\n",
       "      <td>0.003210</td>\n",
       "      <td>0.001224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5314</td>\n",
       "      <td>Strawberries</td>\n",
       "      <td>Organic Strawberries</td>\n",
       "      <td>1093</td>\n",
       "      <td>0.039778</td>\n",
       "      <td>121251</td>\n",
       "      <td>4.412744</td>\n",
       "      <td>228619</td>\n",
       "      <td>8.320238</td>\n",
       "      <td>0.009014</td>\n",
       "      <td>0.004781</td>\n",
       "      <td>0.001083</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11273</td>\n",
       "      <td>Organic Hass Avocado</td>\n",
       "      <td>Organic Avocado</td>\n",
       "      <td>813</td>\n",
       "      <td>0.029588</td>\n",
       "      <td>186186</td>\n",
       "      <td>6.775954</td>\n",
       "      <td>160047</td>\n",
       "      <td>5.824665</td>\n",
       "      <td>0.004367</td>\n",
       "      <td>0.005080</td>\n",
       "      <td>0.000750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1273</td>\n",
       "      <td>Bag of Organic Bananas</td>\n",
       "      <td>Banana</td>\n",
       "      <td>1009</td>\n",
       "      <td>0.036721</td>\n",
       "      <td>326609</td>\n",
       "      <td>11.886433</td>\n",
       "      <td>428131</td>\n",
       "      <td>15.581171</td>\n",
       "      <td>0.003089</td>\n",
       "      <td>0.002357</td>\n",
       "      <td>0.000198</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>58186 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  itemA  \\\n",
       "0                    Dairy Free Greek Yogurt Strawberry   \n",
       "1                 Grassfed Whole Milk Strawberry Yogurt   \n",
       "2              Organic Cashew Nondairy Blueberry Yogurt   \n",
       "4      Organic Strawberry Chia Lowfat 2% Cottage Cheese   \n",
       "5                                         Hair Shampoos   \n",
       "...                                                 ...   \n",
       "8314                                 Small Hass Avocado   \n",
       "26549                                  Organic Cucumber   \n",
       "5314                                       Strawberries   \n",
       "11273                              Organic Hass Avocado   \n",
       "1273                             Bag of Organic Bananas   \n",
       "\n",
       "                                                   itemB  freqAB  supportAB  \\\n",
       "0                      Dairy Free Greek Yogurt Blueberry     293   0.010663   \n",
       "1          Organic Strawberry Grassfed Whole Milk Yogurt     276   0.010045   \n",
       "2              Organic Nondairy Strawberry Cashew Yogurt     363   0.013211   \n",
       "4             Organic Cottage Cheese Blueberry Acai Chia     489   0.017796   \n",
       "5      Moroccan Argan Oil + Argan Stem Cell Triple Mo...     302   0.010991   \n",
       "...                                                  ...     ...        ...   \n",
       "8314                                Organic Hass Avocado     376   0.013684   \n",
       "26549                                     Cucumber Kirby     290   0.010554   \n",
       "5314                                Organic Strawberries    1093   0.039778   \n",
       "11273                                    Organic Avocado     813   0.029588   \n",
       "1273                                              Banana    1009   0.036721   \n",
       "\n",
       "        freqA   supportA   freqB   supportB  confidenceAtoB  confidenceBtoA  \\\n",
       "0         578   0.021035     531   0.019325        0.506920        0.551789   \n",
       "1         561   0.020417     592   0.021545        0.491979        0.466216   \n",
       "2         576   0.020963     851   0.030971        0.630208        0.426557   \n",
       "4        1089   0.039632     766   0.027877        0.449036        0.638381   \n",
       "5         634   0.023073     843   0.030680        0.476341        0.358244   \n",
       "...       ...        ...     ...        ...             ...             ...   \n",
       "8314    44260   1.610775  186186   6.775954        0.008495        0.002019   \n",
       "26549   72064   2.622659   90338   3.287713        0.004024        0.003210   \n",
       "5314   121251   4.412744  228619   8.320238        0.009014        0.004781   \n",
       "11273  186186   6.775954  160047   5.824665        0.004367        0.005080   \n",
       "1273   326609  11.886433  428131  15.581171        0.003089        0.002357   \n",
       "\n",
       "            lift  \n",
       "0      26.231423  \n",
       "1      22.835004  \n",
       "2      20.348442  \n",
       "4      16.107524  \n",
       "5      15.526254  \n",
       "...          ...  \n",
       "8314    0.001254  \n",
       "26549   0.001224  \n",
       "5314    0.001083  \n",
       "11273   0.000750  \n",
       "1273    0.000198  \n",
       "\n",
       "[58186 rows x 11 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Replace item ID with item name and display association rules\n",
    "item_name   = pd.read_csv('products.csv')\n",
    "item_name   = item_name.rename(columns={'product_id':'item_id', 'product_name':'item_name'})\n",
    "rules_final = merge_item_name(rules, item_name).sort_values('lift', ascending=False)\n",
    "display(rules_final)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "rules_final.to_csv('Association_Rules_Mining_cluster_0.csv', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>itemA</th>\n",
       "      <th>itemB</th>\n",
       "      <th>freqAB</th>\n",
       "      <th>supportAB</th>\n",
       "      <th>freqA</th>\n",
       "      <th>supportA</th>\n",
       "      <th>freqB</th>\n",
       "      <th>supportB</th>\n",
       "      <th>confidenceAtoB</th>\n",
       "      <th>confidenceBtoA</th>\n",
       "      <th>lift</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Dairy Free Greek Yogurt Strawberry</td>\n",
       "      <td>Dairy Free Greek Yogurt Blueberry</td>\n",
       "      <td>293</td>\n",
       "      <td>0.010663</td>\n",
       "      <td>578</td>\n",
       "      <td>0.021035</td>\n",
       "      <td>531</td>\n",
       "      <td>0.019325</td>\n",
       "      <td>0.506920</td>\n",
       "      <td>0.551789</td>\n",
       "      <td>26.231423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>Grassfed Whole Milk Strawberry Yogurt</td>\n",
       "      <td>Organic Strawberry Grassfed Whole Milk Yogurt</td>\n",
       "      <td>276</td>\n",
       "      <td>0.010045</td>\n",
       "      <td>561</td>\n",
       "      <td>0.020417</td>\n",
       "      <td>592</td>\n",
       "      <td>0.021545</td>\n",
       "      <td>0.491979</td>\n",
       "      <td>0.466216</td>\n",
       "      <td>22.835004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Organic Cashew Nondairy Blueberry Yogurt</td>\n",
       "      <td>Organic Nondairy Strawberry Cashew Yogurt</td>\n",
       "      <td>363</td>\n",
       "      <td>0.013211</td>\n",
       "      <td>576</td>\n",
       "      <td>0.020963</td>\n",
       "      <td>851</td>\n",
       "      <td>0.030971</td>\n",
       "      <td>0.630208</td>\n",
       "      <td>0.426557</td>\n",
       "      <td>20.348442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>Organic Strawberry Chia Lowfat 2% Cottage Cheese</td>\n",
       "      <td>Organic Cottage Cheese Blueberry Acai Chia</td>\n",
       "      <td>489</td>\n",
       "      <td>0.017796</td>\n",
       "      <td>1089</td>\n",
       "      <td>0.039632</td>\n",
       "      <td>766</td>\n",
       "      <td>0.027877</td>\n",
       "      <td>0.449036</td>\n",
       "      <td>0.638381</td>\n",
       "      <td>16.107524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>Hair Shampoos</td>\n",
       "      <td>Moroccan Argan Oil + Argan Stem Cell Triple Mo...</td>\n",
       "      <td>302</td>\n",
       "      <td>0.010991</td>\n",
       "      <td>634</td>\n",
       "      <td>0.023073</td>\n",
       "      <td>843</td>\n",
       "      <td>0.030680</td>\n",
       "      <td>0.476341</td>\n",
       "      <td>0.358244</td>\n",
       "      <td>15.526254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32938</td>\n",
       "      <td>Appley Ever After Organic Juice Drink</td>\n",
       "      <td>Goodness Grapeness Organic Juice Drink</td>\n",
       "      <td>439</td>\n",
       "      <td>0.015977</td>\n",
       "      <td>2949</td>\n",
       "      <td>0.107324</td>\n",
       "      <td>4048</td>\n",
       "      <td>0.147321</td>\n",
       "      <td>0.148864</td>\n",
       "      <td>0.108449</td>\n",
       "      <td>1.010476</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32941</td>\n",
       "      <td>Organic Bagged Mini Milk Peanut Butter</td>\n",
       "      <td>Organic Bagged Mini Dark Peanut Butter</td>\n",
       "      <td>405</td>\n",
       "      <td>0.014739</td>\n",
       "      <td>2568</td>\n",
       "      <td>0.093458</td>\n",
       "      <td>4290</td>\n",
       "      <td>0.156128</td>\n",
       "      <td>0.157710</td>\n",
       "      <td>0.094406</td>\n",
       "      <td>1.010135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32942</td>\n",
       "      <td>Blue Machine Boosted 100% Juice Smoothie</td>\n",
       "      <td>Naked Green Machine Boosted Juice Smoothie</td>\n",
       "      <td>401</td>\n",
       "      <td>0.014594</td>\n",
       "      <td>2158</td>\n",
       "      <td>0.078537</td>\n",
       "      <td>5077</td>\n",
       "      <td>0.184770</td>\n",
       "      <td>0.185820</td>\n",
       "      <td>0.078984</td>\n",
       "      <td>1.005686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14936</td>\n",
       "      <td>Low-Fat Mango on the Bottom Greek Yogurt</td>\n",
       "      <td>Strawberry on the Bottom Nonfat Greek Yogurt</td>\n",
       "      <td>856</td>\n",
       "      <td>0.031153</td>\n",
       "      <td>3280</td>\n",
       "      <td>0.119371</td>\n",
       "      <td>7140</td>\n",
       "      <td>0.259849</td>\n",
       "      <td>0.260976</td>\n",
       "      <td>0.119888</td>\n",
       "      <td>1.004334</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29666</td>\n",
       "      <td>Fat Free Strawberry Yogurt</td>\n",
       "      <td>Fat Free Blueberry Yogurt</td>\n",
       "      <td>2255</td>\n",
       "      <td>0.082067</td>\n",
       "      <td>9194</td>\n",
       "      <td>0.334602</td>\n",
       "      <td>6731</td>\n",
       "      <td>0.244964</td>\n",
       "      <td>0.245269</td>\n",
       "      <td>0.335017</td>\n",
       "      <td>1.001242</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>599 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  itemA  \\\n",
       "0                    Dairy Free Greek Yogurt Strawberry   \n",
       "1                 Grassfed Whole Milk Strawberry Yogurt   \n",
       "2              Organic Cashew Nondairy Blueberry Yogurt   \n",
       "4      Organic Strawberry Chia Lowfat 2% Cottage Cheese   \n",
       "5                                         Hair Shampoos   \n",
       "...                                                 ...   \n",
       "32938             Appley Ever After Organic Juice Drink   \n",
       "32941            Organic Bagged Mini Milk Peanut Butter   \n",
       "32942          Blue Machine Boosted 100% Juice Smoothie   \n",
       "14936          Low-Fat Mango on the Bottom Greek Yogurt   \n",
       "29666                        Fat Free Strawberry Yogurt   \n",
       "\n",
       "                                                   itemB  freqAB  supportAB  \\\n",
       "0                      Dairy Free Greek Yogurt Blueberry     293   0.010663   \n",
       "1          Organic Strawberry Grassfed Whole Milk Yogurt     276   0.010045   \n",
       "2              Organic Nondairy Strawberry Cashew Yogurt     363   0.013211   \n",
       "4             Organic Cottage Cheese Blueberry Acai Chia     489   0.017796   \n",
       "5      Moroccan Argan Oil + Argan Stem Cell Triple Mo...     302   0.010991   \n",
       "...                                                  ...     ...        ...   \n",
       "32938             Goodness Grapeness Organic Juice Drink     439   0.015977   \n",
       "32941             Organic Bagged Mini Dark Peanut Butter     405   0.014739   \n",
       "32942         Naked Green Machine Boosted Juice Smoothie     401   0.014594   \n",
       "14936       Strawberry on the Bottom Nonfat Greek Yogurt     856   0.031153   \n",
       "29666                          Fat Free Blueberry Yogurt    2255   0.082067   \n",
       "\n",
       "       freqA  supportA  freqB  supportB  confidenceAtoB  confidenceBtoA  \\\n",
       "0        578  0.021035    531  0.019325        0.506920        0.551789   \n",
       "1        561  0.020417    592  0.021545        0.491979        0.466216   \n",
       "2        576  0.020963    851  0.030971        0.630208        0.426557   \n",
       "4       1089  0.039632    766  0.027877        0.449036        0.638381   \n",
       "5        634  0.023073    843  0.030680        0.476341        0.358244   \n",
       "...      ...       ...    ...       ...             ...             ...   \n",
       "32938   2949  0.107324   4048  0.147321        0.148864        0.108449   \n",
       "32941   2568  0.093458   4290  0.156128        0.157710        0.094406   \n",
       "32942   2158  0.078537   5077  0.184770        0.185820        0.078984   \n",
       "14936   3280  0.119371   7140  0.259849        0.260976        0.119888   \n",
       "29666   9194  0.334602   6731  0.244964        0.245269        0.335017   \n",
       "\n",
       "            lift  \n",
       "0      26.231423  \n",
       "1      22.835004  \n",
       "2      20.348442  \n",
       "4      16.107524  \n",
       "5      15.526254  \n",
       "...          ...  \n",
       "32938   1.010476  \n",
       "32941   1.010135  \n",
       "32942   1.005686  \n",
       "14936   1.004334  \n",
       "29666   1.001242  \n",
       "\n",
       "[599 rows x 11 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lift_over_1 = rules_final[rules_final['lift'] > 1]\n",
    "lift_over_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lift_over_1.to_csv('lift_over_1_cluster_0.csv', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Personalized Recommender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Soda                                       10\n",
       "Original Beef Jerky                        10\n",
       "Pistachios                                  9\n",
       "Organic String Cheese                       8\n",
       "Zero Calorie Cola                           3\n",
       "Cinnamon Toast Crunch                       3\n",
       "Organic Half & Half                         2\n",
       "XL Pick-A-Size Paper Towel Rolls            2\n",
       "Aged White Cheddar Popcorn                  2\n",
       "Bag of Organic Bananas                      2\n",
       "Organic Unsweetened Almond Milk             1\n",
       "Organic Unsweetened Vanilla Almond Milk     1\n",
       "Creamy Almond Butter                        1\n",
       "Milk Chocolate Almonds                      1\n",
       "Bartlett Pears                              1\n",
       "Organic Fuji Apples                         1\n",
       "0% Greek Strained Yogurt                    1\n",
       "Honeycrisp Apples                           1\n",
       "Name: product_name, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# example User 1 Shopping history\n",
    "user_1 = orders_all[orders_all['user_id'] == 1]\n",
    "user_1[\"product_name\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>itemA</th>\n",
       "      <th>itemB</th>\n",
       "      <th>freqAB</th>\n",
       "      <th>supportAB</th>\n",
       "      <th>freqA</th>\n",
       "      <th>supportA</th>\n",
       "      <th>freqB</th>\n",
       "      <th>supportB</th>\n",
       "      <th>confidenceAtoB</th>\n",
       "      <th>confidenceBtoA</th>\n",
       "      <th>lift</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>27733</td>\n",
       "      <td>Original Beef Jerky</td>\n",
       "      <td>Soda</td>\n",
       "      <td>759</td>\n",
       "      <td>0.027623</td>\n",
       "      <td>5170</td>\n",
       "      <td>0.188154</td>\n",
       "      <td>29206</td>\n",
       "      <td>1.062908</td>\n",
       "      <td>0.146809</td>\n",
       "      <td>0.025988</td>\n",
       "      <td>0.138120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3084</td>\n",
       "      <td>Original Beef Jerky</td>\n",
       "      <td>Bag of Organic Bananas</td>\n",
       "      <td>753</td>\n",
       "      <td>0.027404</td>\n",
       "      <td>5170</td>\n",
       "      <td>0.188154</td>\n",
       "      <td>326609</td>\n",
       "      <td>11.886433</td>\n",
       "      <td>0.145648</td>\n",
       "      <td>0.002306</td>\n",
       "      <td>0.012253</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27615</td>\n",
       "      <td>Original Beef Jerky</td>\n",
       "      <td>Trail Mix</td>\n",
       "      <td>631</td>\n",
       "      <td>0.022964</td>\n",
       "      <td>5170</td>\n",
       "      <td>0.188154</td>\n",
       "      <td>9195</td>\n",
       "      <td>0.334638</td>\n",
       "      <td>0.122050</td>\n",
       "      <td>0.068624</td>\n",
       "      <td>0.364723</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27660</td>\n",
       "      <td>Original Beef Jerky</td>\n",
       "      <td>Clementines</td>\n",
       "      <td>601</td>\n",
       "      <td>0.021872</td>\n",
       "      <td>5170</td>\n",
       "      <td>0.188154</td>\n",
       "      <td>19749</td>\n",
       "      <td>0.718735</td>\n",
       "      <td>0.116248</td>\n",
       "      <td>0.030432</td>\n",
       "      <td>0.161739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27633</td>\n",
       "      <td>Original Beef Jerky</td>\n",
       "      <td>0% Greek Strained Yogurt</td>\n",
       "      <td>532</td>\n",
       "      <td>0.019361</td>\n",
       "      <td>5170</td>\n",
       "      <td>0.188154</td>\n",
       "      <td>9612</td>\n",
       "      <td>0.349814</td>\n",
       "      <td>0.102901</td>\n",
       "      <td>0.055347</td>\n",
       "      <td>0.294160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29787</td>\n",
       "      <td>Original Beef Jerky</td>\n",
       "      <td>Extra Fancy Unsalted Mixed Nuts</td>\n",
       "      <td>474</td>\n",
       "      <td>0.017251</td>\n",
       "      <td>5170</td>\n",
       "      <td>0.188154</td>\n",
       "      <td>7291</td>\n",
       "      <td>0.265345</td>\n",
       "      <td>0.091683</td>\n",
       "      <td>0.065012</td>\n",
       "      <td>0.345523</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27596</td>\n",
       "      <td>Original Beef Jerky</td>\n",
       "      <td>Milk Chocolate Almonds</td>\n",
       "      <td>317</td>\n",
       "      <td>0.011537</td>\n",
       "      <td>5170</td>\n",
       "      <td>0.188154</td>\n",
       "      <td>3781</td>\n",
       "      <td>0.137604</td>\n",
       "      <td>0.061315</td>\n",
       "      <td>0.083840</td>\n",
       "      <td>0.445593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>34539</td>\n",
       "      <td>Original Beef Jerky</td>\n",
       "      <td>Sparkling Mineral Water</td>\n",
       "      <td>313</td>\n",
       "      <td>0.011391</td>\n",
       "      <td>5170</td>\n",
       "      <td>0.188154</td>\n",
       "      <td>14413</td>\n",
       "      <td>0.524539</td>\n",
       "      <td>0.060542</td>\n",
       "      <td>0.021717</td>\n",
       "      <td>0.115419</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>375</td>\n",
       "      <td>Original Beef Jerky</td>\n",
       "      <td>Banana</td>\n",
       "      <td>290</td>\n",
       "      <td>0.010554</td>\n",
       "      <td>5170</td>\n",
       "      <td>0.188154</td>\n",
       "      <td>428131</td>\n",
       "      <td>15.581171</td>\n",
       "      <td>0.056093</td>\n",
       "      <td>0.000677</td>\n",
       "      <td>0.003600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     itemA                            itemB  freqAB  \\\n",
       "27733  Original Beef Jerky                             Soda     759   \n",
       "3084   Original Beef Jerky           Bag of Organic Bananas     753   \n",
       "27615  Original Beef Jerky                        Trail Mix     631   \n",
       "27660  Original Beef Jerky                      Clementines     601   \n",
       "27633  Original Beef Jerky         0% Greek Strained Yogurt     532   \n",
       "29787  Original Beef Jerky  Extra Fancy Unsalted Mixed Nuts     474   \n",
       "27596  Original Beef Jerky           Milk Chocolate Almonds     317   \n",
       "34539  Original Beef Jerky          Sparkling Mineral Water     313   \n",
       "375    Original Beef Jerky                           Banana     290   \n",
       "\n",
       "       supportAB  freqA  supportA   freqB   supportB  confidenceAtoB  \\\n",
       "27733   0.027623   5170  0.188154   29206   1.062908        0.146809   \n",
       "3084    0.027404   5170  0.188154  326609  11.886433        0.145648   \n",
       "27615   0.022964   5170  0.188154    9195   0.334638        0.122050   \n",
       "27660   0.021872   5170  0.188154   19749   0.718735        0.116248   \n",
       "27633   0.019361   5170  0.188154    9612   0.349814        0.102901   \n",
       "29787   0.017251   5170  0.188154    7291   0.265345        0.091683   \n",
       "27596   0.011537   5170  0.188154    3781   0.137604        0.061315   \n",
       "34539   0.011391   5170  0.188154   14413   0.524539        0.060542   \n",
       "375     0.010554   5170  0.188154  428131  15.581171        0.056093   \n",
       "\n",
       "       confidenceBtoA      lift  \n",
       "27733        0.025988  0.138120  \n",
       "3084         0.002306  0.012253  \n",
       "27615        0.068624  0.364723  \n",
       "27660        0.030432  0.161739  \n",
       "27633        0.055347  0.294160  \n",
       "29787        0.065012  0.345523  \n",
       "27596        0.083840  0.445593  \n",
       "34539        0.021717  0.115419  \n",
       "375          0.000677  0.003600  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Our Shopping Recommandation\n",
    "results = rules_final[rules_final['itemA'] == \"Original Beef Jerky\"]\n",
    "results.sort_values('supportAB', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Association Rules Mining with Ailes and Departments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Test With Aisle and Departments\n",
    "\n",
    "# Convert from DataFrame to a Series, with order_id as index and item_id as value\n",
    "orders_aisle = orders.set_index('order_id')['aisle_id'].rename('item_id')\n",
    "\n",
    "\n",
    "orders_department = orders.set_index('order_id')['department_id'].rename('item_id')\n",
    "\n",
    "print('dimensions: {0};   size: {1};   unique_orders: {2};   unique_items: {3}'\n",
    "      .format(orders_aisle.shape, size(orders_aisle), len(orders_aisle.index.unique()), len(orders_aisle.value_counts())))\n",
    "\n",
    "print('dimensions: {0};   size: {1};   unique_orders: {2};   unique_items: {3}'\n",
    "      .format(orders_department.shape, size(orders_department), len(orders_department.index.unique()), len(orders_department.value_counts())))\n",
    "\n",
    "%%time\n",
    "rules = association_rules(orders_aisle, 0.01)  \n",
    "\n",
    "%%time\n",
    "rules = association_rules(orders_department, 0.01)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "cae6d63e-8a21-42cb-8f66-18c4fed5bcf4",
    "_uuid": "4a4692d044fb1977315f38165d47955a21cd054f"
   },
   "source": [
    "### Part 4:  Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "344d7ef3-953c-48f3-8c64-8f2cb5413341",
    "_uuid": "1fb820fbf9612419e59483bab6d37fb0bccd21f4"
   },
   "source": [
    "In this document we explored 2 different approaches for building product recommendations.\n",
    "\n",
    "Global recommendations give us insight into the most popular product pairings.\n",
    "\n",
    "Association rules provide us with insight into how often product itemsets are purchased together, as well as the influence that an itemset has on purchasing related itemsets. \n",
    "\n",
    "\n",
    "#### Why are recommenders important to companies like Instacart? \n",
    "\n",
    "#### It allows users to discover more products that they could truly be interested in, but would never find on their own. \n",
    "\n",
    "#### This increases sales and improves the customer experience. \n",
    "\n",
    "#### As customers begin to trust the recommendations, they are more likely to discover additional items and increase basket size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}