lab_jupyter_cf_knn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMML321ENSkillsNetwork817-2022-01-01\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Collaborative Filtering based Recommender System using K Nearest Neighbor**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **60** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Collaborative filtering is probably the most commonly used recommendation algorithm, there are two main types of methods: \n",
    " - **User-based** collaborative filtering is based on the user similarity or neighborhood\n",
    " - **Item-based** collaborative filtering is based on similarity among items\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They both work similarly, let's briefly explain how user-based collaborative filtering works.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "User-based collaborative filtering looks for users who are similar. This is very similar to the user clustering method done previously; where we employed explicit user profiles to calculate user similarity. However, the user profiles may not be available, so how can we determine if two users are similar?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### User-item interaction matrix \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For most collaborative filtering-based recommender systems, the main dataset format is a 2-D matrix called the user-item interaction matrix. In the matrix,  its row is labeled as the user id/index and column labelled to be the item id/index, and the element `(i, j)` represents the rating of user `i` to item `j`.  \n",
    "\n",
    "Below is a simple example of a user-item interaction matrix:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-ML321EN-SkillsNetwork/labs/module_4/images/user_item_matrix.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### KNN-based collaborative filtering\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from above, each row vector represents the rating history of a user and each column vector represents the users who rated the item. A user-item interaction matrix is usually very sparse as you can imagine one user very likely only interacts with a very small subset of items and one item is very likely to be interacted by a small subset of users.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to determine if two users are similar, we can simply calculate the similarities between their row vectors in the interaction matrix. Then based on the similarity measurements, we can find the `k` nearest neighbor as the similar users.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Item-based collaborative filtering works similarly, we just need to look at the user-item matrix vertically. Instead of finding similar users, we are trying to find similar items (courses). If two courses are enrolled by two groups of similar users, then we could consider the two items are similar and use the known ratings from the other users to predict the unknown ratings.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we formulate the KNN based collaborative filtering,  the predicted rating of user $u$ to item $i$, $\\hat{r}_{ui}$ is given by:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**User-based** collaborative filtering:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\hat{r}_{ui} = \\frac{\n",
    "\\sum\\limits_{v \\in N^k_i(u)} \\text{similarity}(u, v) \\cdot r_{vi}}\n",
    "{\\sum\\limits_{v \\in N^k_i(u)} \\text{similarity}(u, v)}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Item-based** collaborative filtering:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\hat{r}_{ui} = \\frac{\n",
    "\\sum\\limits_{j \\in N^k_u(i)} \\text{similarity}(i, j) \\cdot r_{uj}}\n",
    "{\\sum\\limits_{j \\in N^k_u(i)} \\text{similarity}(i, j)}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here $N^k_i(u)$ notates the nearest k neighbors of $u$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's illustrate how the equation works using a simple example. From the above figure, suppose we want to predict the rating of `user6` to item `Machine Learning Capstone` course. After some similarity measurements, we found that k = 4 nearest neighbors: `user2, user3, user4, user5` with similarities in array ```knn_sims```:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import math"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# An example similarity array stores the similarity of user2, user3, user4, and user5 to user6\n",
    "knn_sims = np.array([0.8, 0.92, 0.75, 0.83])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also their rating on the `Machine Learning Capstone` course are:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2.0 means audit and 3.0 means complete the course\n",
    "knn_ratings = np.array([3.0, 3.0, 2.0, 3.0]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the predicted rating of `user6` to item `Machine Learning Capstone` course can be calculated as:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.7727272727272725"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r_u6_ml =  np.dot(knn_sims, knn_ratings)/ sum(knn_sims)\n",
    "r_u6_ml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we already know the true rating to be 3.0, then we get a prediction error RMSE (Rooted Mean Squared Error) as:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.22727272727272751"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "true_rating = 3.0\n",
    "rmse = math.sqrt(true_rating - r_u6_ml) ** 2\n",
    "rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The predicted rating is around 2.7 (close to 3.0 with RMSE 0.22), which indicates that `user6` is also likely to complete the course `Machine Learning Capstone`. As such, we may recommend it to user6 with high confidence.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After completing this lab you will be able to:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Perform KNN-based collaborative filtering on the user-item interaction matrix\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and exploring dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first load our dataset, i.e., a user-item (learn-course) interaction matrix\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "rating_url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-ML321EN-SkillsNetwork/labs/datasets/ratings.csv\"\n",
    "rating_df = pd.read_csv(rating_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>item</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1889878</td>\n",
       "      <td>CC0101EN</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1342067</td>\n",
       "      <td>CL0101EN</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1990814</td>\n",
       "      <td>ML0120ENv3</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>380098</td>\n",
       "      <td>BD0211EN</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>779563</td>\n",
       "      <td>DS0101EN</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user        item  rating\n",
       "0  1889878    CC0101EN     3.0\n",
       "1  1342067    CL0101EN     3.0\n",
       "2  1990814  ML0120ENv3     3.0\n",
       "3   380098    BD0211EN     3.0\n",
       "4   779563    DS0101EN     3.0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rating_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset contains three columns, `user id` (learner), `item id`(course), and `rating`(enrollment mode). \n",
    "\n",
    "Note that this matrix is presented as the dense or vertical form, and you may convert it to a sparse matrix using `pivot` :\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>AI0111EN</th>\n",
       "      <th>BC0101EN</th>\n",
       "      <th>BC0201EN</th>\n",
       "      <th>BC0202EN</th>\n",
       "      <th>BD0101EN</th>\n",
       "      <th>BD0111EN</th>\n",
       "      <th>BD0115EN</th>\n",
       "      <th>BD0121EN</th>\n",
       "      <th>BD0123EN</th>\n",
       "      <th>...</th>\n",
       "      <th>SW0201EN</th>\n",
       "      <th>TA0105</th>\n",
       "      <th>TA0105EN</th>\n",
       "      <th>TA0106EN</th>\n",
       "      <th>TMP0101EN</th>\n",
       "      <th>TMP0105EN</th>\n",
       "      <th>TMP0106</th>\n",
       "      <th>TMP107</th>\n",
       "      <th>WA0101EN</th>\n",
       "      <th>WA0103EN</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 127 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  AI0111EN  BC0101EN  BC0201EN  BC0202EN  BD0101EN  BD0111EN  BD0115EN  \\\n",
       "0     2       0.0       3.0       0.0       0.0       3.0       2.0       0.0   \n",
       "1     4       0.0       0.0       0.0       0.0       2.0       2.0       2.0   \n",
       "2     5       2.0       2.0       2.0       0.0       2.0       0.0       0.0   \n",
       "3     7       0.0       0.0       0.0       0.0       0.0       0.0       0.0   \n",
       "4     8       0.0       0.0       0.0       0.0       0.0       2.0       0.0   \n",
       "\n",
       "   BD0121EN  BD0123EN  ...  SW0201EN  TA0105  TA0105EN  TA0106EN  TMP0101EN  \\\n",
       "0       2.0       2.0  ...       0.0     2.0       0.0       3.0        0.0   \n",
       "1       2.0       2.0  ...       0.0     2.0       0.0       0.0        0.0   \n",
       "2       0.0       2.0  ...       0.0     0.0       2.0       2.0        2.0   \n",
       "3       0.0       0.0  ...       0.0     0.0       0.0       0.0        0.0   \n",
       "4       0.0       0.0  ...       0.0     0.0       0.0       0.0        0.0   \n",
       "\n",
       "   TMP0105EN  TMP0106  TMP107  WA0101EN  WA0103EN  \n",
       "0        2.0      2.0     0.0       3.0       0.0  \n",
       "1        2.0      2.0     0.0       2.0       2.0  \n",
       "2        2.0      2.0     2.0       0.0       2.0  \n",
       "3        0.0      0.0     0.0       0.0       0.0  \n",
       "4        0.0      0.0     0.0       0.0       0.0  \n",
       "\n",
       "[5 rows x 127 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rating_sparse_df = rating_df.pivot(index='user', columns='item', values='rating').fillna(0).reset_index().rename_axis(index=None, columns=None)\n",
    "rating_sparse_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually, the dense format is more preferred as it saves a lot of storage and memory space. While the benefit of the sparse matrix is it is in the nature matrix format and you could apply computations such as cosine similarity directly.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you need to perform KNN-based collaborative filtering on the user-item interaction matrix. \n",
    "You may choose one of the two following implementation options of KNN-based collaborative filtering. \n",
    "- The first one is to use `scikit-surprise` which is a popular and easy-to-use Python recommendation system library. \n",
    "- The second way is to implement it with standard `numpy`, `pandas`, and `sklearn`. You may need to write a lot of low-level implementation code along the way.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation Option 1: Use **Surprise** library (recommended)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Surprise* is a Python sci-kit library for recommender systems. It is simple and comprehensive to build and test different recommendation algorithms. \n",
    "\n",
    "First, let's install it:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: scikit-surprise==1.1.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (1.1.1)\n",
      "Requirement already satisfied: joblib>=0.11 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from scikit-surprise==1.1.1) (1.2.0)\n",
      "Requirement already satisfied: six>=1.10.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from scikit-surprise==1.1.1) (1.16.0)\n",
      "Requirement already satisfied: scipy>=1.0.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from scikit-surprise==1.1.1) (1.7.3)\n",
      "Requirement already satisfied: numpy>=1.11.2 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from scikit-surprise==1.1.1) (1.21.6)\n"
     ]
    }
   ],
   "source": [
    "!pip install scikit-surprise==1.1.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we import required classes and methods\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from surprise import KNNBasic\n",
    "from surprise import Dataset, Reader\n",
    "from surprise.model_selection import train_test_split\n",
    "from surprise import accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, let's take a look at a code example how easily to perform KNN collaborative filtering on a sample movie review dataset, which contains about 100k movie ratings from users.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing the msd similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "RMSE: 0.9821\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9820519821481768"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the movielens-100k dataset (download it if needed),\n",
    "data = Dataset.load_builtin('ml-100k', prompt=False)\n",
    "\n",
    "# sample random trainset and testset\n",
    "# test set is made of 25% of the ratings.\n",
    "trainset, testset = train_test_split(data, test_size=.25)\n",
    "\n",
    "# We'll use the famous KNNBasic algorithm.\n",
    "algo = KNNBasic()\n",
    "\n",
    "# Train the algorithm on the trainset, and predict ratings for the testset\n",
    "algo.fit(trainset)\n",
    "predictions = algo.test(testset)\n",
    "\n",
    "# Then compute RMSE\n",
    "accuracy.rmse(predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, just a couple of lines and you can apply KNN collaborative filtering on the sample movie lens dataset. The main evaluation metric is `Root Mean Square Error (RMSE)` which is a very popular rating estimation error metric used in recommender systems as well as many regression model evaluations.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's load our own course rating dataset:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "rating_df.to_csv(\"course_ratings.csv\", index=False)\n",
    "# Read the course rating dataset with columns user item rating\n",
    "reader = Reader(\n",
    "        line_format='user item rating', sep=',', skip_lines=1, rating_scale=(2, 3))\n",
    "\n",
    "coruse_dataset = Dataset.load_from_file(\"course_ratings.csv\", reader=reader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split it into trainset and testset:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainset, testset = train_test_split(coruse_dataset, test_size=.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "then check how many users and items we can use to fit a KNN model:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total 31301 users and 124 items in the trainingset\n"
     ]
    }
   ],
   "source": [
    "print(f\"Total {trainset.n_users} users and {trainset.n_items} items in the trainingset\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TASK: Perform KNN-based collaborative filtering on the user-item interaction matrix\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_TODO: Fit the KNN-based collaborative filtering model using the trainset and evaluate the results using the testset:_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing the cosine similarity matrix...\n",
      "Done computing similarity matrix.\n",
      "RMSE: 0.2082\n",
      "0.20819341858353788\n"
     ]
    }
   ],
   "source": [
    "## WRITE YOUR CODE HERE:\n",
    "\n",
    "\n",
    "# - Define a KNNBasic() model\n",
    "# Note there are some arguments such as:\n",
    "# max_k and min_k, representing the max and min number of neighors for rating estimations\n",
    "# sim_option, representing similarity measurement such as cosine and whether you want it to be user_based or items_based \n",
    "# e.g., sim_option = {\n",
    "#        'name': 'cosine', 'user_based': False,\n",
    "#    }\n",
    "#\n",
    "# more KNN model hyperparamets can be found here:\n",
    "# https://surprise.readthedocs.io/en/stable/knn_inspired.html\n",
    "# \n",
    "# You may try different hyperparamet combinations to see which one has the best performance\n",
    "\n",
    "\n",
    "# - Train the KNNBasic model on the trainset, and predict ratings for the testset\n",
    "\n",
    "# - Then compute RMSE\n",
    "sim_option ={\n",
    "    \"name\": \"cosine\",\n",
    "    \"user_based\": False\n",
    "}\n",
    "mod = KNNBasic(mak_k = 50, min_k = 5, sim_options = sim_option)\n",
    "mod.fit(trainset)\n",
    "predictions = mod.test(testset)\n",
    "print(accuracy.rmse(predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Hints</summary>\n",
    "\n",
    "* Create a model by calling `KNNBasic()` class. \n",
    "* Fit it with `trainset` by using `model.fit(trainset)`.  \n",
    "* Record predictions to the `testset`  by using `model.test(testset).\n",
    "* Compute the accuracy by using `accuracy.rmse(predictions)`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To learn more detailed usages about _Surprise_ library, visit its website from [here](https://surprise.readthedocs.io/en/stable/getting_started.html?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMML321ENSkillsNetwork817-2022-01-01)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation Option 2: Use `numpy`, `pandas`, and `sklearn`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you do not prefer the one-stop Suprise solution and want more hardcore coding practices, you may implement the KNN model using `numpy`, `pandas`, and possibly `sklearn`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "## WRITE YOUR CODE HERE:\n",
    "\n",
    "## One solution could be:\n",
    "## - Calculate the similarity between two users using their rating history (the row vectors of interaction matrix)\n",
    "\n",
    "## - Build a similarity matrix for each pair of users with the training dataset\n",
    "\n",
    "## - For each user, find its k nearest neighbors in the sim matrix\n",
    "\n",
    "## - For each rating in the test dataset, estimate its rating using the KNN collaborative filtering equations shown before\n",
    "\n",
    "## - Calculate RMSE for the entire test dataset\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In this lab, you have learned and implemented KNN-based collaborative filtering. It is probably the simplest but very effective and intuitive collaborative filtering algorithm. Since it is based on KNN, it inherits the main characteristics of KNN such as memory-intensive because you need to maintain a huge similarity matrix among users or items. In the future labs, we will learn other types of collaborative filtering which do not rely on such a huge similarity matrix to make rating predictions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Yan Luo](https://www.linkedin.com/in/yan-luo-96288783/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMML321ENSkillsNetwork817-2022-01-01)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Date (YYYY-MM-DD)|Version|Changed By|Change Description|\n",
    "|-|-|-|-|\n",
    "|2021-10-25|1.0|Yan|Created the initial version|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2021 IBM Corporation. All rights reserved.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}