K-means Clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# K-means Clustering Algorithm\n",
    "\n",
    "Thie notebook shows an implementation of the K-means Clustering algorithm. This is an unsupervised learning algorithm used to find meaningful groups and segments within a dataset.\n",
    "\n",
    "## Dataset\n",
    "\n",
    "The dataset that we will be working with refers to the clients of a wholesale distributer.\n",
    "\n",
    "- The Fresh, Milk, Grocery, Frozen, Detergents_Paper, and Deilicassen attributes are the annual spend that clients made on different product categories;\n",
    "- The Channel category refers to the marketing channel that the customers came from - Hotel/Restaurant/Cafe, or Retail;\n",
    "- The Region category refers to the region of Portugal that the clients came from - Lisbon, Oporto, or Other Region."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Channel</th>\n",
       "      <th>Region</th>\n",
       "      <th>Fresh</th>\n",
       "      <th>Milk</th>\n",
       "      <th>Grocery</th>\n",
       "      <th>Frozen</th>\n",
       "      <th>Detergents_Paper</th>\n",
       "      <th>Delicassen</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Retail</td>\n",
       "      <td>Other</td>\n",
       "      <td>12669</td>\n",
       "      <td>9656</td>\n",
       "      <td>7561</td>\n",
       "      <td>214</td>\n",
       "      <td>2674</td>\n",
       "      <td>1338</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Retail</td>\n",
       "      <td>Other</td>\n",
       "      <td>7057</td>\n",
       "      <td>9810</td>\n",
       "      <td>9568</td>\n",
       "      <td>1762</td>\n",
       "      <td>3293</td>\n",
       "      <td>1776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Retail</td>\n",
       "      <td>Other</td>\n",
       "      <td>6353</td>\n",
       "      <td>8808</td>\n",
       "      <td>7684</td>\n",
       "      <td>2405</td>\n",
       "      <td>3516</td>\n",
       "      <td>7844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Hotel/Restaurant/Cafe</td>\n",
       "      <td>Other</td>\n",
       "      <td>13265</td>\n",
       "      <td>1196</td>\n",
       "      <td>4221</td>\n",
       "      <td>6404</td>\n",
       "      <td>507</td>\n",
       "      <td>1788</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Retail</td>\n",
       "      <td>Other</td>\n",
       "      <td>22615</td>\n",
       "      <td>5410</td>\n",
       "      <td>7198</td>\n",
       "      <td>3915</td>\n",
       "      <td>1777</td>\n",
       "      <td>5185</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Channel Region  Fresh  Milk  Grocery  Frozen  \\\n",
       "0                 Retail  Other  12669  9656     7561     214   \n",
       "1                 Retail  Other   7057  9810     9568    1762   \n",
       "2                 Retail  Other   6353  8808     7684    2405   \n",
       "3  Hotel/Restaurant/Cafe  Other  13265  1196     4221    6404   \n",
       "4                 Retail  Other  22615  5410     7198    3915   \n",
       "\n",
       "   Detergents_Paper  Delicassen  \n",
       "0              2674        1338  \n",
       "1              3293        1776  \n",
       "2              3516        7844  \n",
       "3               507        1788  \n",
       "4              1777        5185  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv(\"data/wholesale-customer-data.csv\")\n",
    "\n",
    "# Relabel the categorical columns\n",
    "data.Region = data.Region.astype(\"category\")\n",
    "data.Region.cat.categories = [\"Lisbon\", \"Oporto\", \"Other\"]\n",
    "data.Channel = data.Channel.astype(\"category\")\n",
    "data.Channel.cat.categories = [\"Hotel/Restaurant/Cafe\", \"Retail\"]\n",
    "\n",
    "data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preprocessing\n",
    "\n",
    "The dataset is already in a pretty good state for us to use. We need to:\n",
    "    \n",
    "- encode the two categorical columns; and\n",
    "- scale the numerical columns.\n",
    "\n",
    "#### Encoding Categorical Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fresh</th>\n",
       "      <th>Milk</th>\n",
       "      <th>Grocery</th>\n",
       "      <th>Frozen</th>\n",
       "      <th>Detergents_Paper</th>\n",
       "      <th>Delicassen</th>\n",
       "      <th>Region_Oporto</th>\n",
       "      <th>Region_Other</th>\n",
       "      <th>Channel_Retail</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12669</td>\n",
       "      <td>9656</td>\n",
       "      <td>7561</td>\n",
       "      <td>214</td>\n",
       "      <td>2674</td>\n",
       "      <td>1338</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7057</td>\n",
       "      <td>9810</td>\n",
       "      <td>9568</td>\n",
       "      <td>1762</td>\n",
       "      <td>3293</td>\n",
       "      <td>1776</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6353</td>\n",
       "      <td>8808</td>\n",
       "      <td>7684</td>\n",
       "      <td>2405</td>\n",
       "      <td>3516</td>\n",
       "      <td>7844</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13265</td>\n",
       "      <td>1196</td>\n",
       "      <td>4221</td>\n",
       "      <td>6404</td>\n",
       "      <td>507</td>\n",
       "      <td>1788</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>22615</td>\n",
       "      <td>5410</td>\n",
       "      <td>7198</td>\n",
       "      <td>3915</td>\n",
       "      <td>1777</td>\n",
       "      <td>5185</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Fresh  Milk  Grocery  Frozen  Detergents_Paper  Delicassen  Region_Oporto  \\\n",
       "0  12669  9656     7561     214              2674        1338              0   \n",
       "1   7057  9810     9568    1762              3293        1776              0   \n",
       "2   6353  8808     7684    2405              3516        7844              0   \n",
       "3  13265  1196     4221    6404               507        1788              0   \n",
       "4  22615  5410     7198    3915              1777        5185              0   \n",
       "\n",
       "   Region_Other  Channel_Retail  \n",
       "0             1               1  \n",
       "1             1               1  \n",
       "2             1               1  \n",
       "3             1               0  \n",
       "4             1               1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# encode the Region column\n",
    "data = pd.concat(\n",
    "    [\n",
    "        data,\n",
    "        pd.get_dummies(data.Region, drop_first=True, prefix=\"Region\")\n",
    "    ],\n",
    "    axis=1\n",
    ").drop([\"Region\"], axis=1)\n",
    "\n",
    "# encode the Channel column\n",
    "data = pd.concat(\n",
    "    [\n",
    "        data,\n",
    "        pd.get_dummies(data.Channel, drop_first=True, prefix=\"Channel\")\n",
    "    ],\n",
    "    axis=1\n",
    ").drop([\"Channel\"], axis=1)\n",
    "\n",
    "data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Feature Scaling\n",
    "\n",
    "We will standardise the numerical data as this is recommended for unsupervised learning algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fresh</th>\n",
       "      <th>Milk</th>\n",
       "      <th>Grocery</th>\n",
       "      <th>Frozen</th>\n",
       "      <th>Detergents_Paper</th>\n",
       "      <th>Delicassen</th>\n",
       "      <th>Region_Oporto</th>\n",
       "      <th>Region_Other</th>\n",
       "      <th>Channel_Retail</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.052933</td>\n",
       "      <td>0.523568</td>\n",
       "      <td>-0.041115</td>\n",
       "      <td>-0.589367</td>\n",
       "      <td>-0.043569</td>\n",
       "      <td>-0.066339</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.391302</td>\n",
       "      <td>0.544458</td>\n",
       "      <td>0.170318</td>\n",
       "      <td>-0.270136</td>\n",
       "      <td>0.086407</td>\n",
       "      <td>0.089151</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.447029</td>\n",
       "      <td>0.408538</td>\n",
       "      <td>-0.028157</td>\n",
       "      <td>-0.137536</td>\n",
       "      <td>0.133232</td>\n",
       "      <td>2.243293</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.100111</td>\n",
       "      <td>-0.624020</td>\n",
       "      <td>-0.392977</td>\n",
       "      <td>0.687144</td>\n",
       "      <td>-0.498588</td>\n",
       "      <td>0.093411</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.840239</td>\n",
       "      <td>-0.052396</td>\n",
       "      <td>-0.079356</td>\n",
       "      <td>0.173859</td>\n",
       "      <td>-0.231918</td>\n",
       "      <td>1.299347</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fresh      Milk   Grocery    Frozen  Detergents_Paper  Delicassen  \\\n",
       "0  0.052933  0.523568 -0.041115 -0.589367         -0.043569   -0.066339   \n",
       "1 -0.391302  0.544458  0.170318 -0.270136          0.086407    0.089151   \n",
       "2 -0.447029  0.408538 -0.028157 -0.137536          0.133232    2.243293   \n",
       "3  0.100111 -0.624020 -0.392977  0.687144         -0.498588    0.093411   \n",
       "4  0.840239 -0.052396 -0.079356  0.173859         -0.231918    1.299347   \n",
       "\n",
       "   Region_Oporto  Region_Other  Channel_Retail  \n",
       "0              0             1               1  \n",
       "1              0             1               1  \n",
       "2              0             1               1  \n",
       "3              0             1               0  \n",
       "4              0             1               1  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import preprocessing\n",
    "\n",
    "data[\n",
    "    [\"Fresh\", \"Milk\", \"Grocery\", \"Frozen\", \"Detergents_Paper\", \"Delicassen\"]\n",
    "] = preprocessing.scale(data[[\"Fresh\", \"Milk\", \"Grocery\", \"Frozen\", \"Detergents_Paper\", \"Delicassen\"]])\n",
    "\n",
    "data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Algorithm\n",
    "\n",
    "The algorithm works by keeping track of $k$ \"centroids\" which represent the centres of the $k$ cluters that we are trying to sort the data into.\n",
    "\n",
    "1. We start by randomly selecting $k$ points in the dataset to be the initial value for the centroids;\n",
    "2. Then, for each data point in the set, we calculate the distance (Euclidean distance in this case) from the data point to each centroid;\n",
    "3. We assign each point to the cluster of the centroid that it is closest to;\n",
    "4. We then calculate the average of each of these clusters and for each cluster make the centroid the average of the cluster;\n",
    "5. Repeat steps 2-4 until the datapoints in a cluster do not change between iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from random import sample\n",
    "\n",
    "def kmeans_clustering(data, k=3, iterations=1):\n",
    "    \"\"\"Perform the K-means Clustering algorithm.\n",
    "\n",
    "    This function performs the K-means Clustering algorithm with `k` clusters on the given dataset.\n",
    "    \n",
    "    Args:\n",
    "        data (pandas dataframe): the data to perform the algorithm on.\n",
    "        k (int): the number of clusters to separate the data into.\n",
    "    \n",
    "    Returns:\n",
    "        clusters (list): a list of pandas datasets, one for each cluster.\n",
    "\n",
    "    \"\"\"\n",
    "    \n",
    "    best_total_var = float(\"inf\")\n",
    "    best_clustering = None\n",
    "    \n",
    "    for iteration in range(iterations):\n",
    "        clustered_data = one_iteration_kmeans_clustering(data.copy(), k)\n",
    "        #data.groupby(\"cluster\").var().sum(axis=1).mean()\n",
    "        total_var = clustered_data.groupby(\"cluster\").var().sum(axis=1).sum()\n",
    "        #print(f\"iteration {iteration}: variance = {total_var}\")\n",
    "        if total_var < best_total_var:\n",
    "            best_total_var = total_var\n",
    "            best_clustering = clustered_data\n",
    "        #print(f\"best variance = {best_total_var}\")\n",
    "    \n",
    "    return best_clustering\n",
    "    \n",
    "\n",
    "def one_iteration_kmeans_clustering(data, k):\n",
    "    \"\"\"Perform the K-means Clustering algorithm.\n",
    "\n",
    "    This function performs the K-means Clustering algorithm with `k` clusters on the given dataset.\n",
    "    \n",
    "    Args:\n",
    "        data (pandas dataframe): the data to perform the algorithm on.\n",
    "        k (int): the number of clusters to separate the data into.\n",
    "    \n",
    "    Returns:\n",
    "        clusters (list): a list of pandas datasets, one for each cluster.\n",
    "\n",
    "    \"\"\"\n",
    "    centroids = select_initial_centroids(data, k)\n",
    "    data[\"cluster\"] = None\n",
    "    clusters = get_clusters(data, centroids)\n",
    "    count = 0\n",
    "    \n",
    "    while has_cluster_variance_changed(data, clusters):\n",
    "        count += 1\n",
    "        data.cluster = clusters\n",
    "        centroids = iterate_centroids(data)\n",
    "        clusters = get_clusters(data, centroids)\n",
    "        \n",
    "    return data\n",
    "\n",
    "def select_initial_centroids(data, k):\n",
    "    \"\"\"Select the initial centroids for the start of the algorithm.\n",
    "    \n",
    "    Args:\n",
    "        data (pandas dataframe): the data to perform the algorithm on.\n",
    "        k (int): the number of clusters to separate the data into.\n",
    "    \n",
    "    Returns:\n",
    "        initial_centroids (list): a list of numpy arrays representing the initial centroid vectors.\n",
    "    \n",
    "    \"\"\"\n",
    "    N = len(data)\n",
    "    indexes = sample(list(range(N)), k)\n",
    "    return [data.iloc[indexes.pop(), :].copy().values for i in range(k)]\n",
    "\n",
    "def closest_centroid(centroids):\n",
    "    \n",
    "    def closest_to(row):\n",
    "        \"\"\"Curried function to get the closest centroid for a given data point.\n",
    "        \n",
    "        Given a list of centroids, and then a data point, this curried function returns\n",
    "        the index of the centroid that that data point was closest to.\n",
    "        \n",
    "        Args:\n",
    "            centroids (list): a list of centroids (numpy vectors).\n",
    "            row (pandas series): one row of data from the dataset.\n",
    "        \n",
    "        Returns:\n",
    "            closest_index (int): the index of the closest centroid.\n",
    "        \n",
    "        \n",
    "        \"\"\"\n",
    "        \n",
    "        p = row.values\n",
    "        distances = [np.sqrt(np.sum((c-p) ** 2)) for c in centroids]\n",
    "        return np.argmin(distances)\n",
    "        \n",
    "    return closest_to\n",
    "\n",
    "def get_clusters(data, centroids):\n",
    "    \"\"\"Get a column showing the nearest cluster for each data point.\"\"\"\n",
    "    return data.drop(\"cluster\", axis=1).apply(closest_centroid(centroids), axis=1)\n",
    "\n",
    "def has_cluster_variance_changed(data, clusters):\n",
    "    \"\"\"Determine whether the clustering has changed between iterations.\"\"\"\n",
    "    return not (data.cluster == clusters).all()\n",
    "\n",
    "def iterate_centroids(data):\n",
    "    \"\"\"Calculate the next iteration of the centroid from the current clustering.\n",
    "    \n",
    "    Take data that has already been clustered and determine the average points for each cluster.\n",
    "    These will then become the centroids in the next iteration of the algorithm.\n",
    "    \n",
    "    Args:\n",
    "        data (pandas dataframe): the dataset that we are operating on.\n",
    "    \n",
    "    Returns:\n",
    "        centroids (list): a list of the next `k` centroids.\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    cluster_means = data.groupby(\"cluster\").apply(average_dataset)\n",
    "    return [np.array(row[1:]) for row in cluster_means.itertuples()]\n",
    "\n",
    "def average_dataset(dataframe):\n",
    "    \"\"\"Return the column average of the dataset without the cluster column.\"\"\"\n",
    "    return dataframe.drop(\"cluster\", axis=1).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "clustered_data = kmeans_clustering(data.copy(), k=2, iterations=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choosing K\n",
    "\n",
    "We want to choose an appropriate value of $k$ to run the algorithm with. A \"good\" value of $k$ would be one where the variance between datapoints in each cluster was small. This would mean that there was little variation within a cluster.\n",
    "\n",
    "The problem is that as we increase $k$ the variance will always decrease. Especially note that when we choose $k$ to be the same as the number of data points we have, $N$, then the variance of each cluster will be $1$, as each cluster will only contain one datapoint.\n",
    "\n",
    "We want to find the point where the rate of change of the variance as we increase $k$ starts to drop off, i.e. when the gains from increasing $k$ stop being \"meaningful\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fresh</th>\n",
       "      <th>Milk</th>\n",
       "      <th>Grocery</th>\n",
       "      <th>Frozen</th>\n",
       "      <th>Detergents_Paper</th>\n",
       "      <th>Delicassen</th>\n",
       "      <th>Region_Oporto</th>\n",
       "      <th>Region_Other</th>\n",
       "      <th>Channel_Retail</th>\n",
       "      <th>cluster</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>435</th>\n",
       "      <td>1.401312</td>\n",
       "      <td>0.848446</td>\n",
       "      <td>0.850760</td>\n",
       "      <td>2.075222</td>\n",
       "      <td>-0.566831</td>\n",
       "      <td>0.241091</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>436</th>\n",
       "      <td>2.155293</td>\n",
       "      <td>-0.592142</td>\n",
       "      <td>-0.757165</td>\n",
       "      <td>0.296561</td>\n",
       "      <td>-0.585519</td>\n",
       "      <td>0.291501</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>437</th>\n",
       "      <td>0.200326</td>\n",
       "      <td>1.314671</td>\n",
       "      <td>2.348386</td>\n",
       "      <td>-0.543380</td>\n",
       "      <td>2.511218</td>\n",
       "      <td>0.121456</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>438</th>\n",
       "      <td>-0.135384</td>\n",
       "      <td>-0.517536</td>\n",
       "      <td>-0.602514</td>\n",
       "      <td>-0.419441</td>\n",
       "      <td>-0.569770</td>\n",
       "      <td>0.213046</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>439</th>\n",
       "      <td>-0.729307</td>\n",
       "      <td>-0.555924</td>\n",
       "      <td>-0.573227</td>\n",
       "      <td>-0.620094</td>\n",
       "      <td>-0.504888</td>\n",
       "      <td>-0.522869</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Fresh      Milk   Grocery    Frozen  Detergents_Paper  Delicassen  \\\n",
       "435  1.401312  0.848446  0.850760  2.075222         -0.566831    0.241091   \n",
       "436  2.155293 -0.592142 -0.757165  0.296561         -0.585519    0.291501   \n",
       "437  0.200326  1.314671  2.348386 -0.543380          2.511218    0.121456   \n",
       "438 -0.135384 -0.517536 -0.602514 -0.419441         -0.569770    0.213046   \n",
       "439 -0.729307 -0.555924 -0.573227 -0.620094         -0.504888   -0.522869   \n",
       "\n",
       "     Region_Oporto  Region_Other  Channel_Retail  cluster  \n",
       "435              0             1               0        0  \n",
       "436              0             1               0        0  \n",
       "437              0             1               1        1  \n",
       "438              0             1               0        0  \n",
       "439              0             1               0        0  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clustered_data.tail(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17.262495028726256"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clustered_data.groupby(\"cluster\").var().sum(axis=1).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "k = 2: variance 17.262495028726256\n",
      "k = 3: variance 24.79222632821415\n",
      "k = 4: variance 39.74902487542304\n",
      "k = 5: variance 30.551500974621735\n",
      "k = 6: variance 62.933500294439575\n",
      "k = 7: variance 70.13630760754856\n",
      "k = 8: variance 41.33591957633465\n",
      "k = 9: variance 49.00478679960785\n",
      "k = 10: variance 48.37822555284168\n",
      "k = 11: variance 78.09587616597196\n",
      "k = 12: variance 59.73673388851804\n",
      "k = 13: variance 47.98410929653397\n",
      "k = 14: variance 78.9412197021831\n",
      "k = 15: variance 75.3887494043107\n",
      "k = 16: variance 68.35877618055781\n",
      "k = 17: variance 67.83476702756894\n",
      "k = 18: variance 73.26252633401438\n",
      "k = 19: variance 76.13309052596324\n",
      "k = 20: variance 76.5524487816797\n"
     ]
    }
   ],
   "source": [
    "for k in range(2, 21):\n",
    "    clustered_data = kmeans_clustering(data.copy(), k=k, iterations=10)\n",
    "    var = clustered_data.groupby(\"cluster\").var().sum(axis=1).sum()\n",
    "    print(f\"k = {k}: variance {var}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}