word2vec_evaluate_rnn2.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install pandas\n",
    "# !pip install numpy\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensure reproducibility\n",
    "Use a fixed seed such that all steps and results can be reproduced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# seed handpicked to ensure all of the cleaning/pre-processing steps were visually shown\n",
    "SEED = 544\n",
    "np.random.seed(SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Recurrent NN on my Word2Vec Model\n",
    "In this section, we will train an Recurrent Neural Network for sentiment analysis classification for both the binary and ternary cases.\n",
    "\n",
    "For part (a), use RNN cell with hidden state size of 50. Limit review length to 50, truncating longer ones and padding shorter reviews with zeros. For part (b), use a gated RNN cell instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load dataset\n",
    "Load the pandas dataset from Q1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the data from disk\n",
    "data = pd.read_pickle('dataset.pkl')\n",
    "\n",
    "# Because this homework takes ages to run, I was forced to reduce the dataset's size by 60%\n",
    "data = data.drop(data[data['label'] == 0].sample(frac=.6).index)\n",
    "data = data.drop(data[data['label'] == 1].sample(frac=.6).index)\n",
    "data = data.drop(data[data['label'] == 2].sample(frac=.6).index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load my Word2Vec model\n",
    "Load the w2v model from Q2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import KeyedVectors\n",
    "w2v_own = KeyedVectors.load('my_w2v.w2v')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the vectors for the reviews\n",
    "The input feature is the first 50 vectors of the review. Word with no encoding vectors are ignored. If a review has less than 50 vectors, the rest are filled with 0s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transform the given review body text into the first50 Word2Vec vector using a given trained word2vec model\n",
    "def create_first50_input_feature(text, wv):\n",
    "    vectors = []\n",
    "    # Will skip words that have no vectors\n",
    "    for word in str(text).split():\n",
    "        if word in wv:\n",
    "            vec = np.array(wv[word], np.float32)\n",
    "            vectors.append(vec)\n",
    "            # If we have our first 50 vectors, we can exit the loop\n",
    "            if len(vectors) == 50:\n",
    "                break\n",
    "    # The review does not have enough vectors, so we fill the rest with zeros\n",
    "    while len(vectors) < 50:\n",
    "        vectors.append(np.zeros((300,), dtype=np.float32))\n",
    "    return vectors\n",
    "\n",
    "# Replace the existing column for the review's input feature for both our w2v and googles w2v\n",
    "data['own_input_features'] = data['cleaned_reviews'].apply( \n",
    "    lambda text: create_first50_input_feature(text, w2v_own)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Common Functionality\n",
    "Below are functions and classes that group up implementations for code reuse and understanding. It is used for Q5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "device = torch.device('cpu')\n",
    "\n",
    "# Returns training and testing data loaders\n",
    "def prepare_data(X_train, X_test, y_train, y_test, batch_size):\n",
    "    train_data = torch.tensor(X_train, device=device)\n",
    "    train_label = torch.tensor(y_train.values, dtype=torch.long, device=device)\n",
    "    train_tensor = TensorDataset(train_data, train_label)\n",
    "    train_loader = DataLoader(dataset=train_tensor, batch_size=batch_size, shuffle=True)\n",
    "    \n",
    "    test_data = torch.tensor(X_test, device=device)\n",
    "    test_label = torch.tensor(y_test.values, dtype=torch.long, device=device)\n",
    "    test_tensor = TensorDataset(test_data, test_label)\n",
    "    test_loader = DataLoader(dataset=test_tensor, batch_size=batch_size, shuffle=True)\n",
    "    \n",
    "    return train_loader, test_loader\n",
    "\n",
    "# Trains the model given the data_loader with max_epochs\n",
    "def train_model(model, max_epochs, data_loader):\n",
    "    model.train()\n",
    "    for epoch in range(max_epochs):\n",
    "        for idx, (X, y) in enumerate(data_loader):\n",
    "            # Match the shape needed for GRU which is (50, 300)\n",
    "            X = X.reshape(-1, model.sequence_length, model.input_size).to(device=device)\n",
    "            y = y.to(device=device)\n",
    "            y_pred = model(X)\n",
    "            loss = model.criterion(y_pred, y)\n",
    "            model.optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            model.optimizer.step()\n",
    "            if (idx+1)%100 == 0:\n",
    "                print (f'Epoch [{epoch+1}/{max_epochs}], Step [{idx+1}], Loss: {loss.item():.3f}')\n",
    "\n",
    "# Returns the model's accuracy (0-1) given the data_loader\n",
    "def evaluate_model(model, data_loader):\n",
    "    model.eval()\n",
    "    num_correct = 0\n",
    "    num_samples = 0\n",
    "    with torch.no_grad():\n",
    "        for X, y in data_loader:\n",
    "            X = X.reshape(-1, model.sequence_length, model.input_size).to(device=device)\n",
    "            y = y.to(device=device)\n",
    "            scores = model(X)\n",
    "            _, predictions = scores.max(1)\n",
    "            num_correct += (predictions == y).sum()\n",
    "            num_samples += predictions.size(0)\n",
    "    return float(num_correct) / float(num_samples)\n",
    "\n",
    "# Reports the accuracy of the model\n",
    "def report_accuracy(model, text, data_loader):\n",
    "    accuracy = evaluate_model(model, data_loader)\n",
    "    print(f'{text}: accuracy is {accuracy:.3f}.')\n",
    "    print()\n",
    "\n",
    "# A Recurrent Neural Network with 1 hidden layer that uses CrossEntropyLoss and Adam optimizer\n",
    "class RNN(torch.nn.Module):\n",
    "    # Initializes the model with a single RNN layer using CrossEntropyLoss and Adam optimizer\n",
    "    def __init__(self, input_size, hidden_size, num_classes, learning_rate):\n",
    "        super(RNN, self).__init__()\n",
    "        self.input_size = input_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.sequence_length = 50\n",
    "        self.rnn = torch.nn.RNN(input_size, hidden_size, 1, batch_first=True)\n",
    "        self.linear = torch.nn.Linear(hidden_size, num_classes)\n",
    "        self.criterion = torch.nn.CrossEntropyLoss()\n",
    "        self.optimizer = torch.optim.Adam(self.parameters(), lr=learning_rate)\n",
    "    \n",
    "    # Performs a forward pass\n",
    "    def forward(self, x):\n",
    "        # Initial the hidden state to zeros\n",
    "        h0 = torch.zeros(1, x.size(0), self.hidden_size).to(device)\n",
    "        out, _ = self.rnn(x, h0)\n",
    "        # Get the last output and discard all intermediate outputs\n",
    "        out = out[:,-1,:]\n",
    "        out = self.linear(out)\n",
    "        return out\n",
    "        \n",
    "# A Gated Recurrent Neural Network with 1 hidden layer that uses CrossEntropyLoss and Adam optimizer\n",
    "class GRU(torch.nn.Module):\n",
    "    # Initializes the model with a single GRU layer using CrossEntropyLoss and Adam optimizer\n",
    "    def __init__(self, input_size, hidden_size, num_classes, learning_rate):\n",
    "        super(GRU, self).__init__()\n",
    "        self.input_size = input_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.sequence_length = 50\n",
    "        self.gru = torch.nn.GRU(input_size, hidden_size, 1, batch_first=True)\n",
    "        self.linear = torch.nn.Linear(hidden_size, num_classes)\n",
    "        self.criterion = torch.nn.CrossEntropyLoss()\n",
    "        self.optimizer = torch.optim.Adam(self.parameters(), lr=learning_rate)\n",
    "    \n",
    "    # Performs a forward pass\n",
    "    def forward(self, x):\n",
    "        # Initial the hidden state to zeros\n",
    "        h0 = torch.zeros(1, x.size(0), self.hidden_size).to(device)\n",
    "        out, _ = self.gru(x, h0)\n",
    "        # Get the last output and discard all intermediate outputs\n",
    "        out = out[:,-1,:]\n",
    "        out = self.linear(out)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. RNN and GRU on My Word2Vec\n",
    "In this section, we will use an (a)RNN and (b)GRU as the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and Testing data split (Binary)\n",
    "Split the data into two distinct parts (80% training, 20% testing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "binary_data = data[data['label'] <= 1] # Only select class 0 (positive) and class 1 (negative)\n",
    "own_input_features = binary_data['own_input_features']\n",
    "binary_labels = binary_data['label']\n",
    "\n",
    "# Perform an 80-20 split for training and testing data on the binary data only\n",
    "X_train_own, X_test_own, y_train_own, y_test_own = train_test_split(\n",
    "    own_input_features,\n",
    "    binary_labels,\n",
    "    test_size=0.2,\n",
    "    random_state=SEED\n",
    ")\n",
    "\n",
    "# Reshape from (num_samples,) to (num_samples, 50, 300)\n",
    "X_train_own = np.dstack(X_train_own) # (num_samples,) -> (50, 300, num_samples)\n",
    "X_train_own = np.moveaxis(X_train_own, -1, 0) # (50, 300, num_samples) -> (num_samples, 50, 300)\n",
    "X_test_own = np.dstack(X_test_own) # (num_samples,) -> (50, 300, num_samples)\n",
    "X_test_own = np.moveaxis(X_test_own, -1, 0) # (50, 300, num_samples) -> (num_samples, 50, 300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5a.1 RNN with My Word2Vec (Binary Case)\n",
    "Train an RNN for binary classification using my Word2Vec model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [1/5], Step [100], Loss: 0.693\n",
      "Epoch [1/5], Step [200], Loss: 0.705\n",
      "Epoch [1/5], Step [300], Loss: 0.654\n",
      "Epoch [1/5], Step [400], Loss: 0.687\n",
      "Epoch [1/5], Step [500], Loss: 0.673\n",
      "Epoch [1/5], Step [600], Loss: 0.684\n",
      "Epoch [1/5], Step [700], Loss: 0.676\n",
      "Epoch [1/5], Step [800], Loss: 0.674\n",
      "Epoch [1/5], Step [900], Loss: 0.714\n",
      "Epoch [1/5], Step [1000], Loss: 0.703\n",
      "Epoch [1/5], Step [1100], Loss: 0.715\n",
      "Epoch [1/5], Step [1200], Loss: 0.715\n",
      "Epoch [1/5], Step [1300], Loss: 0.695\n",
      "Epoch [1/5], Step [1400], Loss: 0.676\n",
      "Epoch [1/5], Step [1500], Loss: 0.666\n",
      "Epoch [1/5], Step [1600], Loss: 0.718\n",
      "Epoch [1/5], Step [1700], Loss: 0.689\n",
      "Epoch [1/5], Step [1800], Loss: 0.695\n",
      "Epoch [1/5], Step [1900], Loss: 0.696\n",
      "Epoch [2/5], Step [100], Loss: 0.569\n",
      "Epoch [2/5], Step [200], Loss: 0.676\n",
      "Epoch [2/5], Step [300], Loss: 0.586\n",
      "Epoch [2/5], Step [400], Loss: 0.656\n",
      "Epoch [2/5], Step [500], Loss: 0.680\n",
      "Epoch [2/5], Step [600], Loss: 0.730\n",
      "Epoch [2/5], Step [700], Loss: 0.764\n",
      "Epoch [2/5], Step [800], Loss: 0.594\n",
      "Epoch [2/5], Step [900], Loss: 0.613\n",
      "Epoch [2/5], Step [1000], Loss: 0.528\n",
      "Epoch [2/5], Step [1100], Loss: 0.552\n",
      "Epoch [2/5], Step [1200], Loss: 0.730\n",
      "Epoch [2/5], Step [1300], Loss: 0.600\n",
      "Epoch [2/5], Step [1400], Loss: 0.580\n",
      "Epoch [2/5], Step [1500], Loss: 0.605\n",
      "Epoch [2/5], Step [1600], Loss: 0.610\n",
      "Epoch [2/5], Step [1700], Loss: 0.614\n",
      "Epoch [2/5], Step [1800], Loss: 0.610\n",
      "Epoch [2/5], Step [1900], Loss: 0.659\n",
      "Epoch [3/5], Step [100], Loss: 0.651\n",
      "Epoch [3/5], Step [200], Loss: 0.606\n",
      "Epoch [3/5], Step [300], Loss: 0.620\n",
      "Epoch [3/5], Step [400], Loss: 0.465\n",
      "Epoch [3/5], Step [500], Loss: 0.472\n",
      "Epoch [3/5], Step [600], Loss: 0.507\n",
      "Epoch [3/5], Step [700], Loss: 0.534\n",
      "Epoch [3/5], Step [800], Loss: 0.556\n",
      "Epoch [3/5], Step [900], Loss: 0.557\n",
      "Epoch [3/5], Step [1000], Loss: 0.600\n",
      "Epoch [3/5], Step [1100], Loss: 0.567\n",
      "Epoch [3/5], Step [1200], Loss: 0.789\n",
      "Epoch [3/5], Step [1300], Loss: 0.692\n",
      "Epoch [3/5], Step [1400], Loss: 0.674\n",
      "Epoch [3/5], Step [1500], Loss: 0.598\n",
      "Epoch [3/5], Step [1600], Loss: 0.675\n",
      "Epoch [3/5], Step [1700], Loss: 0.477\n",
      "Epoch [3/5], Step [1800], Loss: 0.460\n",
      "Epoch [3/5], Step [1900], Loss: 0.615\n",
      "Epoch [4/5], Step [100], Loss: 0.569\n",
      "Epoch [4/5], Step [200], Loss: 0.611\n",
      "Epoch [4/5], Step [300], Loss: 0.676\n",
      "Epoch [4/5], Step [400], Loss: 0.520\n",
      "Epoch [4/5], Step [500], Loss: 0.666\n",
      "Epoch [4/5], Step [600], Loss: 0.579\n",
      "Epoch [4/5], Step [700], Loss: 0.574\n",
      "Epoch [4/5], Step [800], Loss: 0.537\n",
      "Epoch [4/5], Step [900], Loss: 0.467\n",
      "Epoch [4/5], Step [1000], Loss: 0.599\n",
      "Epoch [4/5], Step [1100], Loss: 0.435\n",
      "Epoch [4/5], Step [1200], Loss: 0.686\n",
      "Epoch [4/5], Step [1300], Loss: 0.605\n",
      "Epoch [4/5], Step [1400], Loss: 0.555\n",
      "Epoch [4/5], Step [1500], Loss: 0.540\n",
      "Epoch [4/5], Step [1600], Loss: 0.566\n",
      "Epoch [4/5], Step [1700], Loss: 0.501\n",
      "Epoch [4/5], Step [1800], Loss: 0.546\n",
      "Epoch [4/5], Step [1900], Loss: 0.446\n",
      "Epoch [5/5], Step [100], Loss: 0.483\n",
      "Epoch [5/5], Step [200], Loss: 0.413\n",
      "Epoch [5/5], Step [300], Loss: 0.443\n",
      "Epoch [5/5], Step [400], Loss: 0.632\n",
      "Epoch [5/5], Step [500], Loss: 0.542\n",
      "Epoch [5/5], Step [600], Loss: 0.619\n",
      "Epoch [5/5], Step [700], Loss: 0.545\n",
      "Epoch [5/5], Step [800], Loss: 0.713\n",
      "Epoch [5/5], Step [900], Loss: 0.529\n",
      "Epoch [5/5], Step [1000], Loss: 0.459\n",
      "Epoch [5/5], Step [1100], Loss: 0.519\n",
      "Epoch [5/5], Step [1200], Loss: 0.452\n",
      "Epoch [5/5], Step [1300], Loss: 0.547\n",
      "Epoch [5/5], Step [1400], Loss: 0.465\n",
      "Epoch [5/5], Step [1500], Loss: 0.707\n",
      "Epoch [5/5], Step [1600], Loss: 0.469\n",
      "Epoch [5/5], Step [1700], Loss: 0.650\n",
      "Epoch [5/5], Step [1800], Loss: 0.588\n",
      "Epoch [5/5], Step [1900], Loss: 0.645\n",
      "Own W2V - RNN - Binary: accuracy is 0.713.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Binary RNN - Own Trained Word2Vec\n",
    "\n",
    "# Prepare the data for PyTorch\n",
    "train_loader, test_loader = prepare_data(X_train_own, X_test_own, y_train_own, y_test_own, batch_size=32)\n",
    "\n",
    "# Create RNN for binary classification\n",
    "model = RNN(input_size=300, hidden_size=50, num_classes=2, learning_rate=1e-4)\n",
    "\n",
    "# Train the RNN\n",
    "train_model(model, max_epochs=5, data_loader=train_loader)\n",
    "\n",
    "# Evaluate RNN\n",
    "report_accuracy(model, 'Own W2V - RNN - Binary', data_loader=test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5b.1 GRU with My Word2Vec (Binary Case)\n",
    "Train an GRU for binary classification using my Word2Vec model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [1/5], Step [100], Loss: 0.703\n",
      "Epoch [1/5], Step [200], Loss: 0.700\n",
      "Epoch [1/5], Step [300], Loss: 0.719\n",
      "Epoch [1/5], Step [400], Loss: 0.692\n",
      "Epoch [1/5], Step [500], Loss: 0.659\n",
      "Epoch [1/5], Step [600], Loss: 0.681\n",
      "Epoch [1/5], Step [700], Loss: 0.683\n",
      "Epoch [1/5], Step [800], Loss: 0.674\n",
      "Epoch [1/5], Step [900], Loss: 0.677\n",
      "Epoch [1/5], Step [1000], Loss: 0.678\n",
      "Epoch [1/5], Step [1100], Loss: 0.653\n",
      "Epoch [1/5], Step [1200], Loss: 0.627\n",
      "Epoch [1/5], Step [1300], Loss: 0.674\n",
      "Epoch [1/5], Step [1400], Loss: 0.662\n",
      "Epoch [1/5], Step [1500], Loss: 0.678\n",
      "Epoch [1/5], Step [1600], Loss: 0.674\n",
      "Epoch [1/5], Step [1700], Loss: 0.680\n",
      "Epoch [1/5], Step [1800], Loss: 0.656\n",
      "Epoch [1/5], Step [1900], Loss: 0.666\n",
      "Epoch [2/5], Step [100], Loss: 0.542\n",
      "Epoch [2/5], Step [200], Loss: 0.475\n",
      "Epoch [2/5], Step [300], Loss: 0.811\n",
      "Epoch [2/5], Step [400], Loss: 0.402\n",
      "Epoch [2/5], Step [500], Loss: 0.476\n",
      "Epoch [2/5], Step [600], Loss: 0.511\n",
      "Epoch [2/5], Step [700], Loss: 0.487\n",
      "Epoch [2/5], Step [800], Loss: 0.481\n",
      "Epoch [2/5], Step [900], Loss: 0.530\n",
      "Epoch [2/5], Step [1000], Loss: 0.562\n",
      "Epoch [2/5], Step [1100], Loss: 0.613\n",
      "Epoch [2/5], Step [1200], Loss: 0.450\n",
      "Epoch [2/5], Step [1300], Loss: 0.387\n",
      "Epoch [2/5], Step [1400], Loss: 0.592\n",
      "Epoch [2/5], Step [1500], Loss: 0.366\n",
      "Epoch [2/5], Step [1600], Loss: 0.469\n",
      "Epoch [2/5], Step [1700], Loss: 0.613\n",
      "Epoch [2/5], Step [1800], Loss: 0.422\n",
      "Epoch [2/5], Step [1900], Loss: 0.281\n",
      "Epoch [3/5], Step [100], Loss: 0.602\n",
      "Epoch [3/5], Step [200], Loss: 0.387\n",
      "Epoch [3/5], Step [300], Loss: 0.356\n",
      "Epoch [3/5], Step [400], Loss: 0.276\n",
      "Epoch [3/5], Step [500], Loss: 0.405\n",
      "Epoch [3/5], Step [600], Loss: 0.437\n",
      "Epoch [3/5], Step [700], Loss: 0.445\n",
      "Epoch [3/5], Step [800], Loss: 0.397\n",
      "Epoch [3/5], Step [900], Loss: 0.276\n",
      "Epoch [3/5], Step [1000], Loss: 0.319\n",
      "Epoch [3/5], Step [1100], Loss: 0.311\n",
      "Epoch [3/5], Step [1200], Loss: 0.310\n",
      "Epoch [3/5], Step [1300], Loss: 0.375\n",
      "Epoch [3/5], Step [1400], Loss: 0.371\n",
      "Epoch [3/5], Step [1500], Loss: 0.436\n",
      "Epoch [3/5], Step [1600], Loss: 0.351\n",
      "Epoch [3/5], Step [1700], Loss: 0.416\n",
      "Epoch [3/5], Step [1800], Loss: 0.388\n",
      "Epoch [3/5], Step [1900], Loss: 0.273\n",
      "Epoch [4/5], Step [100], Loss: 0.285\n",
      "Epoch [4/5], Step [200], Loss: 0.359\n",
      "Epoch [4/5], Step [300], Loss: 0.243\n",
      "Epoch [4/5], Step [400], Loss: 0.442\n",
      "Epoch [4/5], Step [500], Loss: 0.389\n",
      "Epoch [4/5], Step [600], Loss: 0.567\n",
      "Epoch [4/5], Step [700], Loss: 0.427\n",
      "Epoch [4/5], Step [800], Loss: 0.248\n",
      "Epoch [4/5], Step [900], Loss: 0.580\n",
      "Epoch [4/5], Step [1000], Loss: 0.379\n",
      "Epoch [4/5], Step [1100], Loss: 0.262\n",
      "Epoch [4/5], Step [1200], Loss: 0.321\n",
      "Epoch [4/5], Step [1300], Loss: 0.659\n",
      "Epoch [4/5], Step [1400], Loss: 0.333\n",
      "Epoch [4/5], Step [1500], Loss: 0.449\n",
      "Epoch [4/5], Step [1600], Loss: 0.368\n",
      "Epoch [4/5], Step [1700], Loss: 0.426\n",
      "Epoch [4/5], Step [1800], Loss: 0.301\n",
      "Epoch [4/5], Step [1900], Loss: 0.427\n",
      "Epoch [5/5], Step [100], Loss: 0.214\n",
      "Epoch [5/5], Step [200], Loss: 0.436\n",
      "Epoch [5/5], Step [300], Loss: 0.223\n",
      "Epoch [5/5], Step [400], Loss: 0.455\n",
      "Epoch [5/5], Step [500], Loss: 0.237\n",
      "Epoch [5/5], Step [600], Loss: 0.469\n",
      "Epoch [5/5], Step [700], Loss: 0.291\n",
      "Epoch [5/5], Step [800], Loss: 0.373\n",
      "Epoch [5/5], Step [900], Loss: 0.292\n",
      "Epoch [5/5], Step [1000], Loss: 0.397\n",
      "Epoch [5/5], Step [1100], Loss: 0.331\n",
      "Epoch [5/5], Step [1200], Loss: 0.387\n",
      "Epoch [5/5], Step [1300], Loss: 0.409\n",
      "Epoch [5/5], Step [1400], Loss: 0.325\n",
      "Epoch [5/5], Step [1500], Loss: 0.303\n",
      "Epoch [5/5], Step [1600], Loss: 0.236\n",
      "Epoch [5/5], Step [1700], Loss: 0.281\n",
      "Epoch [5/5], Step [1800], Loss: 0.324\n",
      "Epoch [5/5], Step [1900], Loss: 0.381\n",
      "Own W2V - GRU - Binary: accuracy is 0.845.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Binary GRU - Own Trained Word2Vec\n",
    "\n",
    "# Prepare the data for PyTorch\n",
    "train_loader, test_loader = prepare_data(X_train_own, X_test_own, y_train_own, y_test_own, batch_size=32)\n",
    "\n",
    "# Create GRU for binary classification\n",
    "model = GRU(input_size=300, hidden_size=50, num_classes=2, learning_rate=1e-4)\n",
    "\n",
    "# Train the GRU\n",
    "train_model(model, max_epochs=5, data_loader=train_loader)\n",
    "\n",
    "# Evaluate RNN\n",
    "report_accuracy(model, 'Own W2V - GRU - Binary', data_loader=test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and Testing data split (Ternary)\n",
    "Split the data into two distinct parts (80% training, 20% testing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "own_input_features = data['own_input_features']\n",
    "ternary_labels = data['label']\n",
    "\n",
    "# Perform an 80-20 split for training and testing data on the complete dataset\n",
    "X_train_own, X_test_own, y_train_own, y_test_own = train_test_split(\n",
    "    own_input_features,\n",
    "    ternary_labels,\n",
    "    test_size=0.2,\n",
    "    random_state=SEED\n",
    ")\n",
    "\n",
    "# Reshape from (num_samples,) to (num_samples, 50, 300)\n",
    "X_train_own = np.dstack(X_train_own) # (num_samples,) -> (50, 300, num_samples)\n",
    "X_train_own = np.moveaxis(X_train_own, -1, 0) # (50, 300, num_samples) -> (num_samples, 50, 300)\n",
    "X_test_own = np.dstack(X_test_own) # (num_samples,) -> (50, 300, num_samples)\n",
    "X_test_own = np.moveaxis(X_test_own, -1, 0) # (50, 300, num_samples) -> (num_samples, 50, 300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5a.2 RNN with My Word2Vec (Ternary Case)\n",
    "Train an RNN for ternary classification using my Word2Vec model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [1/5], Step [100], Loss: 1.047\n",
      "Epoch [1/5], Step [200], Loss: 0.997\n",
      "Epoch [1/5], Step [300], Loss: 1.042\n",
      "Epoch [1/5], Step [400], Loss: 1.106\n",
      "Epoch [1/5], Step [500], Loss: 1.030\n",
      "Epoch [1/5], Step [600], Loss: 1.113\n",
      "Epoch [1/5], Step [700], Loss: 1.019\n",
      "Epoch [1/5], Step [800], Loss: 1.049\n",
      "Epoch [1/5], Step [900], Loss: 1.017\n",
      "Epoch [1/5], Step [1000], Loss: 1.056\n",
      "Epoch [1/5], Step [1100], Loss: 0.992\n",
      "Epoch [1/5], Step [1200], Loss: 1.106\n",
      "Epoch [1/5], Step [1300], Loss: 1.127\n",
      "Epoch [1/5], Step [1400], Loss: 1.114\n",
      "Epoch [1/5], Step [1500], Loss: 1.073\n",
      "Epoch [1/5], Step [1600], Loss: 1.013\n",
      "Epoch [1/5], Step [1700], Loss: 1.100\n",
      "Epoch [1/5], Step [1800], Loss: 0.984\n",
      "Epoch [1/5], Step [1900], Loss: 1.017\n",
      "Epoch [1/5], Step [2000], Loss: 1.004\n",
      "Epoch [1/5], Step [2100], Loss: 1.121\n",
      "Epoch [1/5], Step [2200], Loss: 1.034\n",
      "Epoch [1/5], Step [2300], Loss: 1.094\n",
      "Epoch [1/5], Step [2400], Loss: 1.132\n",
      "Epoch [2/5], Step [100], Loss: 1.048\n",
      "Epoch [2/5], Step [200], Loss: 1.037\n",
      "Epoch [2/5], Step [300], Loss: 1.167\n",
      "Epoch [2/5], Step [400], Loss: 1.031\n",
      "Epoch [2/5], Step [500], Loss: 0.965\n",
      "Epoch [2/5], Step [600], Loss: 1.084\n",
      "Epoch [2/5], Step [700], Loss: 1.007\n",
      "Epoch [2/5], Step [800], Loss: 0.901\n",
      "Epoch [2/5], Step [900], Loss: 1.104\n",
      "Epoch [2/5], Step [1000], Loss: 0.976\n",
      "Epoch [2/5], Step [1100], Loss: 1.113\n",
      "Epoch [2/5], Step [1200], Loss: 1.016\n",
      "Epoch [2/5], Step [1300], Loss: 1.044\n",
      "Epoch [2/5], Step [1400], Loss: 1.017\n",
      "Epoch [2/5], Step [1500], Loss: 1.041\n",
      "Epoch [2/5], Step [1600], Loss: 0.968\n",
      "Epoch [2/5], Step [1700], Loss: 0.998\n",
      "Epoch [2/5], Step [1800], Loss: 1.060\n",
      "Epoch [2/5], Step [1900], Loss: 0.995\n",
      "Epoch [2/5], Step [2000], Loss: 1.006\n",
      "Epoch [2/5], Step [2100], Loss: 0.942\n",
      "Epoch [2/5], Step [2200], Loss: 1.008\n",
      "Epoch [2/5], Step [2300], Loss: 0.872\n",
      "Epoch [2/5], Step [2400], Loss: 0.867\n",
      "Epoch [3/5], Step [100], Loss: 1.037\n",
      "Epoch [3/5], Step [200], Loss: 0.791\n",
      "Epoch [3/5], Step [300], Loss: 0.945\n",
      "Epoch [3/5], Step [400], Loss: 1.034\n",
      "Epoch [3/5], Step [500], Loss: 1.142\n",
      "Epoch [3/5], Step [600], Loss: 1.042\n",
      "Epoch [3/5], Step [700], Loss: 1.016\n",
      "Epoch [3/5], Step [800], Loss: 1.201\n",
      "Epoch [3/5], Step [900], Loss: 0.858\n",
      "Epoch [3/5], Step [1000], Loss: 0.843\n",
      "Epoch [3/5], Step [1100], Loss: 0.836\n",
      "Epoch [3/5], Step [1200], Loss: 0.966\n",
      "Epoch [3/5], Step [1300], Loss: 0.990\n",
      "Epoch [3/5], Step [1400], Loss: 1.129\n",
      "Epoch [3/5], Step [1500], Loss: 0.848\n",
      "Epoch [3/5], Step [1600], Loss: 1.109\n",
      "Epoch [3/5], Step [1700], Loss: 0.915\n",
      "Epoch [3/5], Step [1800], Loss: 0.951\n",
      "Epoch [3/5], Step [1900], Loss: 1.138\n",
      "Epoch [3/5], Step [2000], Loss: 1.147\n",
      "Epoch [3/5], Step [2100], Loss: 0.935\n",
      "Epoch [3/5], Step [2200], Loss: 0.974\n",
      "Epoch [3/5], Step [2300], Loss: 0.974\n",
      "Epoch [3/5], Step [2400], Loss: 0.979\n",
      "Epoch [4/5], Step [100], Loss: 0.971\n",
      "Epoch [4/5], Step [200], Loss: 0.823\n",
      "Epoch [4/5], Step [300], Loss: 0.972\n",
      "Epoch [4/5], Step [400], Loss: 0.995\n",
      "Epoch [4/5], Step [500], Loss: 0.957\n",
      "Epoch [4/5], Step [600], Loss: 1.083\n",
      "Epoch [4/5], Step [700], Loss: 0.998\n",
      "Epoch [4/5], Step [800], Loss: 1.061\n",
      "Epoch [4/5], Step [900], Loss: 0.883\n",
      "Epoch [4/5], Step [1000], Loss: 0.890\n",
      "Epoch [4/5], Step [1100], Loss: 1.111\n",
      "Epoch [4/5], Step [1200], Loss: 1.063\n",
      "Epoch [4/5], Step [1300], Loss: 1.072\n",
      "Epoch [4/5], Step [1400], Loss: 0.915\n",
      "Epoch [4/5], Step [1500], Loss: 0.995\n",
      "Epoch [4/5], Step [1600], Loss: 0.933\n",
      "Epoch [4/5], Step [1700], Loss: 0.885\n",
      "Epoch [4/5], Step [1800], Loss: 1.013\n",
      "Epoch [4/5], Step [1900], Loss: 0.981\n",
      "Epoch [4/5], Step [2000], Loss: 0.963\n",
      "Epoch [4/5], Step [2100], Loss: 0.846\n",
      "Epoch [4/5], Step [2200], Loss: 1.008\n",
      "Epoch [4/5], Step [2300], Loss: 0.886\n",
      "Epoch [4/5], Step [2400], Loss: 1.012\n",
      "Epoch [5/5], Step [100], Loss: 0.912\n",
      "Epoch [5/5], Step [200], Loss: 0.996\n",
      "Epoch [5/5], Step [300], Loss: 0.947\n",
      "Epoch [5/5], Step [400], Loss: 1.035\n",
      "Epoch [5/5], Step [500], Loss: 0.945\n",
      "Epoch [5/5], Step [600], Loss: 0.883\n",
      "Epoch [5/5], Step [700], Loss: 0.921\n",
      "Epoch [5/5], Step [800], Loss: 0.953\n",
      "Epoch [5/5], Step [900], Loss: 0.904\n",
      "Epoch [5/5], Step [1000], Loss: 0.972\n",
      "Epoch [5/5], Step [1100], Loss: 0.839\n",
      "Epoch [5/5], Step [1200], Loss: 1.121\n",
      "Epoch [5/5], Step [1300], Loss: 0.998\n",
      "Epoch [5/5], Step [1400], Loss: 0.920\n",
      "Epoch [5/5], Step [1500], Loss: 1.009\n",
      "Epoch [5/5], Step [1600], Loss: 0.826\n",
      "Epoch [5/5], Step [1700], Loss: 0.734\n",
      "Epoch [5/5], Step [1800], Loss: 0.990\n",
      "Epoch [5/5], Step [1900], Loss: 0.887\n",
      "Epoch [5/5], Step [2000], Loss: 1.057\n",
      "Epoch [5/5], Step [2100], Loss: 0.975\n",
      "Epoch [5/5], Step [2200], Loss: 0.914\n",
      "Epoch [5/5], Step [2300], Loss: 1.025\n",
      "Epoch [5/5], Step [2400], Loss: 0.810\n",
      "Own W2V - RNN - Ternary: accuracy is 0.564.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Ternary RNN - Own Trained Word2Vec\n",
    "\n",
    "# Prepare the data for PyTorch\n",
    "train_loader, test_loader = prepare_data(X_train_own, X_test_own, y_train_own, y_test_own, batch_size=32)\n",
    "\n",
    "# Create RNN for ternary classification\n",
    "model = RNN(input_size=300, hidden_size=50, num_classes=3, learning_rate=1e-4)\n",
    "\n",
    "# Train the RNN\n",
    "train_model(model, max_epochs=5, data_loader=train_loader)\n",
    "\n",
    "# Evaluate RNN\n",
    "report_accuracy(model, 'Own W2V - RNN - Ternary', data_loader=test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5b.2 GRU with My Word2Vec (Ternary Case)\n",
    "Train an GRU for ternary classification using my Word2Vec model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [1/5], Step [100], Loss: 1.087\n",
      "Epoch [1/5], Step [200], Loss: 1.091\n",
      "Epoch [1/5], Step [300], Loss: 1.108\n",
      "Epoch [1/5], Step [400], Loss: 1.086\n",
      "Epoch [1/5], Step [500], Loss: 1.016\n",
      "Epoch [1/5], Step [600], Loss: 1.034\n",
      "Epoch [1/5], Step [700], Loss: 1.044\n",
      "Epoch [1/5], Step [800], Loss: 1.005\n",
      "Epoch [1/5], Step [900], Loss: 1.099\n",
      "Epoch [1/5], Step [1000], Loss: 1.032\n",
      "Epoch [1/5], Step [1100], Loss: 1.069\n",
      "Epoch [1/5], Step [1200], Loss: 1.036\n",
      "Epoch [1/5], Step [1300], Loss: 1.078\n",
      "Epoch [1/5], Step [1400], Loss: 1.035\n",
      "Epoch [1/5], Step [1500], Loss: 1.032\n",
      "Epoch [1/5], Step [1600], Loss: 1.039\n",
      "Epoch [1/5], Step [1700], Loss: 1.040\n",
      "Epoch [1/5], Step [1800], Loss: 1.022\n",
      "Epoch [1/5], Step [1900], Loss: 0.944\n",
      "Epoch [1/5], Step [2000], Loss: 1.017\n",
      "Epoch [1/5], Step [2100], Loss: 1.098\n",
      "Epoch [1/5], Step [2200], Loss: 1.060\n",
      "Epoch [1/5], Step [2300], Loss: 1.042\n",
      "Epoch [1/5], Step [2400], Loss: 1.078\n",
      "Epoch [2/5], Step [100], Loss: 1.091\n",
      "Epoch [2/5], Step [200], Loss: 0.977\n",
      "Epoch [2/5], Step [300], Loss: 1.040\n",
      "Epoch [2/5], Step [400], Loss: 1.072\n",
      "Epoch [2/5], Step [500], Loss: 1.103\n",
      "Epoch [2/5], Step [600], Loss: 1.035\n",
      "Epoch [2/5], Step [700], Loss: 1.144\n",
      "Epoch [2/5], Step [800], Loss: 1.013\n",
      "Epoch [2/5], Step [900], Loss: 0.960\n",
      "Epoch [2/5], Step [1000], Loss: 0.978\n",
      "Epoch [2/5], Step [1100], Loss: 1.032\n",
      "Epoch [2/5], Step [1200], Loss: 1.087\n",
      "Epoch [2/5], Step [1300], Loss: 1.055\n",
      "Epoch [2/5], Step [1400], Loss: 0.922\n",
      "Epoch [2/5], Step [1500], Loss: 0.877\n",
      "Epoch [2/5], Step [1600], Loss: 0.993\n",
      "Epoch [2/5], Step [1700], Loss: 0.740\n",
      "Epoch [2/5], Step [1800], Loss: 0.957\n",
      "Epoch [2/5], Step [1900], Loss: 0.908\n",
      "Epoch [2/5], Step [2000], Loss: 0.816\n",
      "Epoch [2/5], Step [2100], Loss: 0.939\n",
      "Epoch [2/5], Step [2200], Loss: 0.900\n",
      "Epoch [2/5], Step [2300], Loss: 0.915\n",
      "Epoch [2/5], Step [2400], Loss: 1.131\n",
      "Epoch [3/5], Step [100], Loss: 0.778\n",
      "Epoch [3/5], Step [200], Loss: 0.775\n",
      "Epoch [3/5], Step [300], Loss: 0.778\n",
      "Epoch [3/5], Step [400], Loss: 0.928\n",
      "Epoch [3/5], Step [500], Loss: 0.869\n",
      "Epoch [3/5], Step [600], Loss: 0.831\n",
      "Epoch [3/5], Step [700], Loss: 0.807\n",
      "Epoch [3/5], Step [800], Loss: 0.839\n",
      "Epoch [3/5], Step [900], Loss: 0.804\n",
      "Epoch [3/5], Step [1000], Loss: 0.785\n",
      "Epoch [3/5], Step [1100], Loss: 0.969\n",
      "Epoch [3/5], Step [1200], Loss: 0.798\n",
      "Epoch [3/5], Step [1300], Loss: 0.836\n",
      "Epoch [3/5], Step [1400], Loss: 0.785\n",
      "Epoch [3/5], Step [1500], Loss: 0.763\n",
      "Epoch [3/5], Step [1600], Loss: 0.895\n",
      "Epoch [3/5], Step [1700], Loss: 0.555\n",
      "Epoch [3/5], Step [1800], Loss: 0.880\n",
      "Epoch [3/5], Step [1900], Loss: 0.698\n",
      "Epoch [3/5], Step [2000], Loss: 0.539\n",
      "Epoch [3/5], Step [2100], Loss: 0.853\n",
      "Epoch [3/5], Step [2200], Loss: 0.819\n",
      "Epoch [3/5], Step [2300], Loss: 0.800\n",
      "Epoch [3/5], Step [2400], Loss: 0.824\n",
      "Epoch [4/5], Step [100], Loss: 0.799\n",
      "Epoch [4/5], Step [200], Loss: 0.748\n",
      "Epoch [4/5], Step [300], Loss: 0.604\n",
      "Epoch [4/5], Step [400], Loss: 0.903\n",
      "Epoch [4/5], Step [500], Loss: 1.017\n",
      "Epoch [4/5], Step [600], Loss: 0.817\n",
      "Epoch [4/5], Step [700], Loss: 0.795\n",
      "Epoch [4/5], Step [800], Loss: 0.817\n",
      "Epoch [4/5], Step [900], Loss: 0.601\n",
      "Epoch [4/5], Step [1000], Loss: 0.878\n",
      "Epoch [4/5], Step [1100], Loss: 0.770\n",
      "Epoch [4/5], Step [1200], Loss: 0.964\n",
      "Epoch [4/5], Step [1300], Loss: 0.490\n",
      "Epoch [4/5], Step [1400], Loss: 0.788\n",
      "Epoch [4/5], Step [1500], Loss: 0.904\n",
      "Epoch [4/5], Step [1600], Loss: 0.846\n",
      "Epoch [4/5], Step [1700], Loss: 0.711\n",
      "Epoch [4/5], Step [1800], Loss: 0.773\n",
      "Epoch [4/5], Step [1900], Loss: 0.971\n",
      "Epoch [4/5], Step [2000], Loss: 0.569\n",
      "Epoch [4/5], Step [2100], Loss: 0.762\n",
      "Epoch [4/5], Step [2200], Loss: 0.656\n",
      "Epoch [4/5], Step [2300], Loss: 0.571\n",
      "Epoch [4/5], Step [2400], Loss: 0.760\n",
      "Epoch [5/5], Step [100], Loss: 0.910\n",
      "Epoch [5/5], Step [200], Loss: 0.527\n",
      "Epoch [5/5], Step [300], Loss: 0.616\n",
      "Epoch [5/5], Step [400], Loss: 0.909\n",
      "Epoch [5/5], Step [500], Loss: 0.699\n",
      "Epoch [5/5], Step [600], Loss: 0.637\n",
      "Epoch [5/5], Step [700], Loss: 0.853\n",
      "Epoch [5/5], Step [800], Loss: 0.723\n",
      "Epoch [5/5], Step [900], Loss: 0.939\n",
      "Epoch [5/5], Step [1000], Loss: 0.429\n",
      "Epoch [5/5], Step [1100], Loss: 0.695\n",
      "Epoch [5/5], Step [1200], Loss: 0.547\n",
      "Epoch [5/5], Step [1300], Loss: 0.822\n",
      "Epoch [5/5], Step [1400], Loss: 0.447\n",
      "Epoch [5/5], Step [1500], Loss: 0.683\n",
      "Epoch [5/5], Step [1600], Loss: 0.742\n",
      "Epoch [5/5], Step [1700], Loss: 0.575\n",
      "Epoch [5/5], Step [1800], Loss: 1.038\n",
      "Epoch [5/5], Step [1900], Loss: 0.583\n",
      "Epoch [5/5], Step [2000], Loss: 0.592\n",
      "Epoch [5/5], Step [2100], Loss: 0.655\n",
      "Epoch [5/5], Step [2200], Loss: 0.782\n",
      "Epoch [5/5], Step [2300], Loss: 0.775\n",
      "Epoch [5/5], Step [2400], Loss: 0.755\n",
      "Own W2V - GRU - Ternary: accuracy is 0.682.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Ternary GRU - Own Trained Word2Vec\n",
    "\n",
    "# Prepare the data for PyTorch\n",
    "train_loader, test_loader = prepare_data(X_train_own, X_test_own, y_train_own, y_test_own, batch_size=32)\n",
    "\n",
    "# Create GRU for ternary classification\n",
    "model = GRU(input_size=300, hidden_size=50, num_classes=3, learning_rate=1e-4)\n",
    "\n",
    "# Train the GRU\n",
    "train_model(model, max_epochs=5, data_loader=train_loader)\n",
    "\n",
    "# Evaluate RNN\n",
    "report_accuracy(model, 'Own W2V - GRU - Ternary', data_loader=test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "At first glance, the accuracies don't seem that great. However, keep in mind, that only 40% of the dataset is being used for training because my PC cannot handle the full dataset. Moreover, the number of epochs was limited to 5 because it was already too time consuming (hours). Having said that, accuracies in the 60s and 70s is decent and I strongly believe that given more RAM and computation power, the RNN and GRU models will reach the 90% accuracy territory because the loss is slowly decreasing throughout the 5 epochs as shown above. Moreover, GRU seems to outperform RNN in our experiment."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}