AI_PHASE4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Market Basket Analysis Project\n",
    "\n",
    "## Overview\n",
    "Market basket analysis is a technique used mostly by retailers to identify which products clients purchase together most frequently. This involves analyzing point of sale (POS) transaction data to identify the correlations between different items according to their co-occurrence in the data.\n",
    "\n",
    "Using this information, retailers are better able to understand customer purchasing patterns, especially which items customers are likely to purchase together. A well-known example of market basket analysis in action is on the Amazon website. Upon selecting a product, a viewer is taken to the product page which not only includes details about the selected product but also displays a section called “Frequently bought together”.\n",
    "\n",
    "\n",
    "## Dataset Information\n",
    "The dataset is stored in the file `Assignment-1_Data.csv` located at `C:\\Users\\mades\\Downloads\\Assignment-1_Data.csv`. It contains information related to market transactions.\n",
    "\n",
    "## Loading the Dataset\n",
    "Let's start by loading the dataset into a DataFrame using pandas.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-24T07:48:58.552368Z",
     "iopub.status.busy": "2023-10-24T07:48:58.551854Z",
     "iopub.status.idle": "2023-10-24T07:50:31.642228Z",
     "shell.execute_reply": "2023-10-24T07:50:31.641038Z",
     "shell.execute_reply.started": "2023-10-24T07:48:58.552335Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BillNo</th>\n",
       "      <th>Itemname</th>\n",
       "      <th>Quantity</th>\n",
       "      <th>Date</th>\n",
       "      <th>Price</th>\n",
       "      <th>CustomerID</th>\n",
       "      <th>Country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>536365</td>\n",
       "      <td>WHITE HANGING HEART T-LIGHT HOLDER</td>\n",
       "      <td>6</td>\n",
       "      <td>2010-12-01 08:26:00</td>\n",
       "      <td>2.55</td>\n",
       "      <td>17850.0</td>\n",
       "      <td>United Kingdom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>536365</td>\n",
       "      <td>WHITE METAL LANTERN</td>\n",
       "      <td>6</td>\n",
       "      <td>2010-12-01 08:26:00</td>\n",
       "      <td>3.39</td>\n",
       "      <td>17850.0</td>\n",
       "      <td>United Kingdom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>536365</td>\n",
       "      <td>CREAM CUPID HEARTS COAT HANGER</td>\n",
       "      <td>8</td>\n",
       "      <td>2010-12-01 08:26:00</td>\n",
       "      <td>2.75</td>\n",
       "      <td>17850.0</td>\n",
       "      <td>United Kingdom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>536365</td>\n",
       "      <td>KNITTED UNION FLAG HOT WATER BOTTLE</td>\n",
       "      <td>6</td>\n",
       "      <td>2010-12-01 08:26:00</td>\n",
       "      <td>3.39</td>\n",
       "      <td>17850.0</td>\n",
       "      <td>United Kingdom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>536365</td>\n",
       "      <td>RED WOOLLY HOTTIE WHITE HEART.</td>\n",
       "      <td>6</td>\n",
       "      <td>2010-12-01 08:26:00</td>\n",
       "      <td>3.39</td>\n",
       "      <td>17850.0</td>\n",
       "      <td>United Kingdom</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   BillNo                             Itemname  Quantity                Date  \\\n",
       "0  536365   WHITE HANGING HEART T-LIGHT HOLDER         6 2010-12-01 08:26:00   \n",
       "1  536365                  WHITE METAL LANTERN         6 2010-12-01 08:26:00   \n",
       "2  536365       CREAM CUPID HEARTS COAT HANGER         8 2010-12-01 08:26:00   \n",
       "3  536365  KNITTED UNION FLAG HOT WATER BOTTLE         6 2010-12-01 08:26:00   \n",
       "4  536365       RED WOOLLY HOTTIE WHITE HEART.         6 2010-12-01 08:26:00   \n",
       "\n",
       "   Price  CustomerID         Country  \n",
       "0   2.55     17850.0  United Kingdom  \n",
       "1   3.39     17850.0  United Kingdom  \n",
       "2   2.75     17850.0  United Kingdom  \n",
       "3   3.39     17850.0  United Kingdom  \n",
       "4   3.39     17850.0  United Kingdom  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from mlxtend.frequent_patterns import apriori\n",
    "from mlxtend.frequent_patterns import association_rules\n",
    "df=pd.read_excel(\"C:/Users/mades/Downloads/Assignment-1_Data.xlsx\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Initial Exploration**\n",
    "\n",
    "We'll perform an initial exploration of the dataset to understand its structure and characteristics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-24T07:50:31.644332Z",
     "iopub.status.busy": "2023-10-24T07:50:31.643864Z",
     "iopub.status.idle": "2023-10-24T07:50:31.987421Z",
     "shell.execute_reply": "2023-10-24T07:50:31.985753Z",
     "shell.execute_reply.started": "2023-10-24T07:50:31.644302Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows and columns: (522064, 7)\n",
      "\n",
      "Data Types and Missing Values:\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 522064 entries, 0 to 522063\n",
      "Data columns (total 7 columns):\n",
      " #   Column      Non-Null Count   Dtype         \n",
      "---  ------      --------------   -----         \n",
      " 0   BillNo      522064 non-null  object        \n",
      " 1   Itemname    520609 non-null  object        \n",
      " 2   Quantity    522064 non-null  int64         \n",
      " 3   Date        522064 non-null  datetime64[ns]\n",
      " 4   Price       522064 non-null  float64       \n",
      " 5   CustomerID  388023 non-null  float64       \n",
      " 6   Country     522064 non-null  object        \n",
      "dtypes: datetime64[ns](1), float64(2), int64(1), object(3)\n",
      "memory usage: 27.9+ MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# Display basic information about the dataset\n",
    "print(\"Number of rows and columns:\", df.shape)\n",
    "print(\"\\nData Types and Missing Values:\")\n",
    "print(df.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing\n",
    "\n",
    "We'll preprocess the data to ensure it's ready for analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-24T07:50:31.991526Z",
     "iopub.status.busy": "2023-10-24T07:50:31.99068Z",
     "iopub.status.idle": "2023-10-24T07:50:32.371143Z",
     "shell.execute_reply": "2023-10-24T07:50:32.370008Z",
     "shell.execute_reply.started": "2023-10-24T07:50:31.991493Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing Values:\n",
      "BillNo             0\n",
      "Itemname        1455\n",
      "Quantity           0\n",
      "Date               0\n",
      "Price              0\n",
      "CustomerID    134041\n",
      "Country            0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#Check Missing Values\n",
    "print(\"Missing Values:\")\n",
    "print(df.isnull().sum())\n",
    "\n",
    "#Drop Rows with Missing Values\n",
    "df.dropna(inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-24T07:50:32.373698Z",
     "iopub.status.busy": "2023-10-24T07:50:32.373214Z",
     "iopub.status.idle": "2023-10-24T07:50:34.029292Z",
     "shell.execute_reply": "2023-10-24T07:50:34.028267Z",
     "shell.execute_reply.started": "2023-10-24T07:50:32.373643Z"
    }
   },
   "outputs": [],
   "source": [
    "# Convert dataframe into transaction data\n",
    "transaction_data = df.groupby(['BillNo', 'Date'])['Itemname'].apply(lambda x: ', '.join(x)).reset_index()\n",
    "\n",
    "#Drop Unnecessary Columns\n",
    "columns_to_drop = ['BillNo', 'Date']\n",
    "transaction_data.drop(columns=columns_to_drop, inplace=True)\n",
    "\n",
    "# Save the transaction data to a CSV file\n",
    "transaction_data_path = \"C:/Users/mades/Downloads/Assignment-1_Data.csv\"\n",
    "transaction_data.to_csv(transaction_data_path, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-24T07:50:34.031288Z",
     "iopub.status.busy": "2023-10-24T07:50:34.030829Z",
     "iopub.status.idle": "2023-10-24T07:50:34.043407Z",
     "shell.execute_reply": "2023-10-24T07:50:34.042242Z",
     "shell.execute_reply.started": "2023-10-24T07:50:34.031192Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Transaction Data for Association Rule Mining:\n",
      "                                            Itemname\n",
      "0  WHITE HANGING HEART T-LIGHT HOLDER, WHITE META...\n",
      "1  HAND WARMER UNION JACK, HAND WARMER RED POLKA DOT\n",
      "2  ASSORTED COLOUR BIRD ORNAMENT, POPPY'S PLAYHOU...\n",
      "3  JAM MAKING SET WITH JARS, RED COAT RACK PARIS ...\n",
      "4                           BATH BUILDING BLOCK WORD\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(18192, 1)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display the first few rows of the transaction data\n",
    "print(\"\\nTransaction Data for Association Rule Mining:\")\n",
    "print(transaction_data.head())\n",
    "transaction_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Formatting the transaction data in a suitable format for analysis\n",
    "\n",
    "Developing the preprocessed data into analysis. Split the 'Itemname' column in `transaction_data` into individual items using `str.split(', ', expand=True)`.Concatenate the original DataFrame (`transaction_data`) with the items DataFrame (`items_df`) using `pd.concat`.Drop the original 'Itemname' column since individual items are now in separate columns.Display the resulting DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-24T07:50:34.045353Z",
     "iopub.status.busy": "2023-10-24T07:50:34.045021Z",
     "iopub.status.idle": "2023-10-24T07:50:37.106596Z",
     "shell.execute_reply": "2023-10-24T07:50:37.105366Z",
     "shell.execute_reply.started": "2023-10-24T07:50:34.045324Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                  0                            1    \\\n",
      "0  WHITE HANGING HEART T-LIGHT HOLDER          WHITE METAL LANTERN   \n",
      "1              HAND WARMER UNION JACK    HAND WARMER RED POLKA DOT   \n",
      "2       ASSORTED COLOUR BIRD ORNAMENT    POPPY'S PLAYHOUSE BEDROOM   \n",
      "3            JAM MAKING SET WITH JARS  RED COAT RACK PARIS FASHION   \n",
      "4            BATH BUILDING BLOCK WORD                         None   \n",
      "\n",
      "                              2                                    3    \\\n",
      "0  CREAM CUPID HEARTS COAT HANGER  KNITTED UNION FLAG HOT WATER BOTTLE   \n",
      "1                            None                                 None   \n",
      "2       POPPY'S PLAYHOUSE KITCHEN    FELTCRAFT PRINCESS CHARLOTTE DOLL   \n",
      "3  YELLOW COAT RACK PARIS FASHION         BLUE COAT RACK PARIS FASHION   \n",
      "4                            None                                 None   \n",
      "\n",
      "                              4                                   5    \\\n",
      "0  RED WOOLLY HOTTIE WHITE HEART.        SET 7 BABUSHKA NESTING BOXES   \n",
      "1                            None                                None   \n",
      "2          IVORY KNITTED MUG COSY  BOX OF 6 ASSORTED COLOUR TEASPOONS   \n",
      "3                            None                                None   \n",
      "4                            None                                None   \n",
      "\n",
      "                                 6                               7    \\\n",
      "0  GLASS STAR FROSTED T-LIGHT HOLDER                            None   \n",
      "1                               None                            None   \n",
      "2       BOX OF VINTAGE JIGSAW BLOCKS  BOX OF VINTAGE ALPHABET BLOCKS   \n",
      "3                               None                            None   \n",
      "4                               None                            None   \n",
      "\n",
      "                        8                         9    ...   534   535   536  \\\n",
      "0                      None                      None  ...  None  None  None   \n",
      "1                      None                      None  ...  None  None  None   \n",
      "2  HOME BUILDING BLOCK WORD  LOVE BUILDING BLOCK WORD  ...  None  None  None   \n",
      "3                      None                      None  ...  None  None  None   \n",
      "4                      None                      None  ...  None  None  None   \n",
      "\n",
      "    537   538   539   540   541   542   543  \n",
      "0  None  None  None  None  None  None  None  \n",
      "1  None  None  None  None  None  None  None  \n",
      "2  None  None  None  None  None  None  None  \n",
      "3  None  None  None  None  None  None  None  \n",
      "4  None  None  None  None  None  None  None  \n",
      "\n",
      "[5 rows x 544 columns]\n"
     ]
    }
   ],
   "source": [
    "# Split the 'Itemname' column into individual items\n",
    "items_df = transaction_data['Itemname'].str.split(', ', expand=True)\n",
    "\n",
    "# Concatenate the original DataFrame with the new items DataFrame\n",
    "transaction_data = pd.concat([transaction_data, items_df], axis=1)\n",
    "\n",
    "# Drop the original 'Itemname' column\n",
    "transaction_data = transaction_data.drop('Itemname', axis=1)\n",
    "\n",
    "# Display the resulting DataFrame\n",
    "print(transaction_data.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Association Rules - Data Mining\n",
    "## Converting Items to Boolean Columns\n",
    "\n",
    "To prepare the data for association rule mining, we convert the items in the `transaction_data` DataFrame into boolean columns using one-hot encoding. This is achieved through the `pd.get_dummies` function, which creates a new DataFrame (`df_encoded`) with boolean columns representing the presence or absence of each item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-24T07:50:37.108722Z",
     "iopub.status.busy": "2023-10-24T07:50:37.108249Z",
     "iopub.status.idle": "2023-10-24T07:51:44.234485Z",
     "shell.execute_reply": "2023-10-24T07:51:44.233304Z",
     "shell.execute_reply.started": "2023-10-24T07:50:37.10868Z"
    }
   },
   "outputs": [],
   "source": [
    "# Convert items to boolean columns\n",
    "df_encoded = pd.get_dummies(transaction_data, prefix='', prefix_sep='').groupby(level=0, axis=1).max()\n",
    "\n",
    "# Save the transaction data to a CSV file\n",
    "df_encoded.to_csv('transaction_data_encoded.csv', index=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Association Rule Mining\n",
    "We apply the Apriori algorithm to perform association rule mining on the encoded transaction data. The `min_support` parameter is set to 0.007 to filter out infrequent itemsets. The resulting frequent itemsets are then used to generate association rules based on a minimum confidence threshold of 0.5.Finally, we print the generated association rules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-10-24T07:51:44.236729Z",
     "iopub.status.busy": "2023-10-24T07:51:44.236253Z",
     "iopub.status.idle": "2023-10-24T07:52:39.329961Z",
     "shell.execute_reply": "2023-10-24T07:52:39.328654Z",
     "shell.execute_reply.started": "2023-10-24T07:51:44.236687Z"
    }
   },
   "outputs": [],
   "source": [
    "# Load transaction data into a DataFrame\n",
    "df_encoded = pd.read_csv('transaction_data_encoded.csv')\n",
    "\n",
    "from mlxtend.frequent_patterns import apriori, association_rules\n",
    "\n",
    "# Association Rule Mining\n",
    "frequent_itemsets = apriori(df_encoded, min_support=0.007, use_colnames=True)\n",
    "rules = association_rules(frequent_itemsets, metric=\"confidence\", min_threshold=0.5)\n",
    "\n",
    "# Display information of the rules\n",
    "print(\"Association Rules:\")\n",
    "print(rules.head())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}