Assignment5-text-analysis.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 5 - Text Analysis\n",
    "An explanation this assignment could be found in the .pdf explanation document"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Materials to review for this assignment\n",
    "<h4>From Moodle:</h4> \n",
    "<h5><u>Review the notebooks regarding the following python topics</u>:</h5>\n",
    "<div class=\"alert alert-info\">\n",
    "&#x2714; <b>Working with strings</b> (tutorial notebook)<br/>\n",
    "&#x2714; <b>Text Analysis</b> (tutorial notebook)<br/>\n",
    "&#x2714; <b>Hebrew text analysis tools (tokenizer, wordnet)</b> (moodle example)<br/>\n",
    "&#x2714; <b>(brief review) All previous notebooks</b><br/>\n",
    "</div> \n",
    "<h5><u>Review the presentations regarding the following topics</u>:</h5>\n",
    "<div class=\"alert alert-info\">\n",
    "&#x2714; <b>Text Analysis</b> (lecture presentation)<br/>\n",
    "&#x2714; <b>(brief review) All other presentations</b><br/>\n",
    "</div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preceding Step - import modules (packages)\n",
    "This step is necessary in order to use external modules (packages). <br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# --------------------------------------\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "# --------------------------------------\n",
    "\n",
    "\n",
    "# --------------------------------------\n",
    "# ------------- visualizations:\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.colors import ListedColormap\n",
    "# --------------------------------------\n",
    "\n",
    "\n",
    "# ---------------------------------------\n",
    "import sklearn\n",
    "from sklearn import preprocessing, metrics, pipeline, model_selection, feature_extraction \n",
    "from sklearn import naive_bayes, linear_model, svm, neural_network, neighbors, tree\n",
    "from sklearn import decomposition, cluster\n",
    "\n",
    "from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV \n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix\n",
    "from sklearn.metrics import precision_score, recall_score, f1_score\n",
    "from sklearn.metrics import mean_squared_error, r2_score, silhouette_score\n",
    "from sklearn.preprocessing import MinMaxScaler, StandardScaler, LabelEncoder\n",
    "\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.linear_model import Perceptron, SGDClassifier\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.naive_bayes import MultinomialNB, GaussianNB\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "# ---------------------------------------\n",
    "\n",
    "\n",
    "# ----------------- output and visualizations: \n",
    "import warnings\n",
    "from sklearn.exceptions import ConvergenceWarning\n",
    "warnings.simplefilter(\"ignore\")\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
    "warnings.simplefilter(\"ignore\", category=ConvergenceWarning)\n",
    "# show several prints in one cell. This will allow us to condence every trick in one cell.\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "%matplotlib inline\n",
    "pd.pandas.set_option('display.max_columns', None)\n",
    "pd.set_option('display.float_format', lambda x: '%.3f' % x)\n",
    "# ---------------------------------------"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text analysis and String manipulation imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# --------------------------------------\n",
    "# --------- Text analysis and Hebrew text analysis imports:\n",
    "# vectorizers:\n",
    "from sklearn.feature_extraction import text\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "\n",
    "# regular expressions:\n",
    "import re\n",
    "# --------------------------------------"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (optional) Hebrew text analysis - WordNet (for Hebrew)\n",
    "Note: the WordNet is not a must"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) Only if you didn't install Wordnet (for Hebrew) use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: wn in c:\\users\\user\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (0.9.3)\n",
      "Requirement already satisfied: requests in c:\\users\\user\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from wn) (2.28.1)\n",
      "Requirement already satisfied: tomli in c:\\users\\user\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from wn) (2.0.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\user\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests->wn) (2022.12.7)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\\users\\user\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests->wn) (1.26.13)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\user\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests->wn) (3.4)\n",
      "Requirement already satisfied: charset-normalizer<3,>=2 in c:\\users\\user\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (from requests->wn) (2.1.1)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 23.0 -> 23.1.2\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n",
      "\n",
      "\u001b[KCached file found: C:\\Users\\User\\.wn_data\\downloads\\7ecf10e89326bc0ac26ad94b40fe60a7b6ac3302\n",
      "\n",
      "\u001b[KChecking C:\\Users\\User\\AppData\\Local\\Temp\\tmpbe1vg4vj\\omw-he\\omw-he.xml\n",
      "\u001b[KSkipping omw-he:1.4 (Hebrew Wordnet); already added\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# word net installation:\n",
    "\n",
    "# unmark if you want to use and need to install\n",
    "!pip install wn\n",
    "!python -m wn download omw-he:1.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# word net import:\n",
    "\n",
    "# unmark if you want to use:\n",
    "import wn"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (optional) Hebrew text analysis - hebrew_tokenizer (Tokenizer for Hebrew)\n",
    "Note: the hebrew_tokenizer is not a must"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) Only if you didn't install hebrew_tokenizer use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: hebrew_tokenizer in c:\\users\\user\\appdata\\local\\programs\\python\\python310\\lib\\site-packages (2.3.0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 23.0 -> 23.1.2\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "# Hebrew tokenizer installation:\n",
    "\n",
    "# unmark if you want to use and need to install:\n",
    "!pip install hebrew_tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hebrew tokenizer import:\n",
    "\n",
    "# unmark if you want to use:\n",
    "import hebrew_tokenizer as ht"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading input files\n",
    "Reading input files for train annotated corpus (raw text data) corpus and for the test corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_filename = 'annotated_corpus_for_train.csv'\n",
    "test_filename  = 'corpus_for_test.csv'\n",
    "df_train = pd.read_csv(train_filename, index_col=None, encoding='utf-8')\n",
    "df_test  = pd.read_csv(test_filename, index_col=None, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>story</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>כשחבר הזמין אותי לחול, לא באמת חשבתי שזה יקרה,...</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>לפני שהתגייסתי לצבא עשיתי כל מני מיונים ליחידו...</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>מאז שהתחילו הלימודים חלומו של כל סטודנט זה הפנ...</td>\n",
       "      <td>f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>כשהייתי ילד, מטוסים היה הדבר שהכי ריתק אותי. ב...</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>‏הייתי מדריכה בכפר נוער ומתאם הכפר היינו צריכי...</td>\n",
       "      <td>f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>לפני כ3 חודשים טסתי לרומא למשך שבוע. טסתי במטו...</td>\n",
       "      <td>f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>אני כבר שנתיים נשוי והשנה אני ואישתי סוף סוף י...</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>השנה התחלנו שיפוץ בדירה שלנו בתל אביב. הדירה ה...</td>\n",
       "      <td>f</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               story gender\n",
       "0  כשחבר הזמין אותי לחול, לא באמת חשבתי שזה יקרה,...      m\n",
       "1  לפני שהתגייסתי לצבא עשיתי כל מני מיונים ליחידו...      m\n",
       "2  מאז שהתחילו הלימודים חלומו של כל סטודנט זה הפנ...      f\n",
       "3  כשהייתי ילד, מטוסים היה הדבר שהכי ריתק אותי. ב...      m\n",
       "4  ‏הייתי מדריכה בכפר נוער ומתאם הכפר היינו צריכי...      f\n",
       "5  לפני כ3 חודשים טסתי לרומא למשך שבוע. טסתי במטו...      f\n",
       "6  אני כבר שנתיים נשוי והשנה אני ואישתי סוף סוף י...      m\n",
       "7  השנה התחלנו שיפוץ בדירה שלנו בתל אביב. הדירה ה...      f"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(753, 2)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head(8)\n",
    "df_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_example_id</th>\n",
       "      <th>story</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>כל קיץ אני והמשפחה נוסעים לארצות הברית לוס אנג...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>הגעתי לשירות המדינה אחרי שנתיים כפעיל בתנועת \"...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>אחת האהבות הגדולות שלי אלו הכלבים שלי ושל אישת...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   test_example_id                                              story\n",
       "0                0  כל קיץ אני והמשפחה נוסעים לארצות הברית לוס אנג...\n",
       "1                1  הגעתי לשירות המדינה אחרי שנתיים כפעיל בתנועת \"...\n",
       "2                2  אחת האהבות הגדולות שלי אלו הכלבים שלי ושל אישת..."
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(323, 2)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test.head(3)\n",
    "df_test.shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your implementation:\n",
    "Write your code solution in the following code-cells"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_tokenizer(hebrew_text):\n",
    "    token = ht.tokenize(hebrew_text)\n",
    "    tokens_list = list(filter(lambda x: (x[0] == 'HEBREW'),token))\n",
    "    tokens_list = [tuple[1] for tuple in tokens_list]\n",
    "    tokens_list\n",
    "    return tokens_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    כשחבר הזמין אותי לחול, לא באמת חשבתי שזה יקרה,...\n",
       "1    לפני שהתגייסתי לצבא עשיתי כל מני מיונים ליחידו...\n",
       "2    מאז שהתחילו הלימודים חלומו של כל סטודנט זה הפנ...\n",
       "3    כשהייתי ילד, מטוסים היה הדבר שהכי ריתק אותי. ב...\n",
       "4    ‏הייתי מדריכה בכפר נוער ומתאם הכפר היינו צריכי...\n",
       "Name: story, dtype: object"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    1\n",
       "2    0\n",
       "3    1\n",
       "4    0\n",
       "Name: gender, dtype: int32"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train['gender'] = np.where(df_train['gender'] == 'm',1,0)\n",
    "X_train = df_train['story']\n",
    "y_train = df_train['gender']\n",
    "X_train.head(5)\n",
    "y_train.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating SGDClassifier...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-6 {color: black;background-color: white;}#sk-container-id-6 pre{padding: 0;}#sk-container-id-6 div.sk-toggleable {background-color: white;}#sk-container-id-6 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-6 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-6 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-6 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-6 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-6 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-6 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-6 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-6 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-6 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-6 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-6 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-6 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-6 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-6 div.sk-item {position: relative;z-index: 1;}#sk-container-id-6 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-6 div.sk-item::before, #sk-container-id-6 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-6 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-6 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-6 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-6 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-6 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-6 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-6 div.sk-label-container {text-align: center;}#sk-container-id-6 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-6 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-6\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                                        TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                                       (&#x27;normalize&#x27;, Normalizer()),\n",
       "                                       (&#x27;clf&#x27;,\n",
       "                                        SGDClassifier(random_state=42))]),\n",
       "             param_grid={&#x27;clf__alpha&#x27;: [0.0001, 0.001, 0.01, 0.1],\n",
       "                         &#x27;clf__max_iter&#x27;: [1000, 2000, 3000],\n",
       "                         &#x27;clf__penalty&#x27;: [&#x27;l1&#x27;, &#x27;l2&#x27;, &#x27;elasticnet&#x27;],\n",
       "                         &#x27;tfidf__max_df&#x27;: [0.5, 0.75, 1.0]},\n",
       "             scoring=&#x27;f1_macro&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-18\" type=\"checkbox\" ><label for=\"sk-estimator-id-18\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                                        TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                                       (&#x27;normalize&#x27;, Normalizer()),\n",
       "                                       (&#x27;clf&#x27;,\n",
       "                                        SGDClassifier(random_state=42))]),\n",
       "             param_grid={&#x27;clf__alpha&#x27;: [0.0001, 0.001, 0.01, 0.1],\n",
       "                         &#x27;clf__max_iter&#x27;: [1000, 2000, 3000],\n",
       "                         &#x27;clf__penalty&#x27;: [&#x27;l1&#x27;, &#x27;l2&#x27;, &#x27;elasticnet&#x27;],\n",
       "                         &#x27;tfidf__max_df&#x27;: [0.5, 0.75, 1.0]},\n",
       "             scoring=&#x27;f1_macro&#x27;)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-19\" type=\"checkbox\" ><label for=\"sk-estimator-id-19\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                (&#x27;normalize&#x27;, Normalizer()),\n",
       "                (&#x27;clf&#x27;, SGDClassifier(random_state=42))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-20\" type=\"checkbox\" ><label for=\"sk-estimator-id-20\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-21\" type=\"checkbox\" ><label for=\"sk-estimator-id-21\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Normalizer</label><div class=\"sk-toggleable__content\"><pre>Normalizer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-22\" type=\"checkbox\" ><label for=\"sk-estimator-id-22\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SGDClassifier</label><div class=\"sk-toggleable__content\"><pre>SGDClassifier(random_state=42)</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[('tfidf',\n",
       "                                        TfidfVectorizer(tokenizer=<function my_tokenizer at 0x000002269C541000>)),\n",
       "                                       ('normalize', Normalizer()),\n",
       "                                       ('clf',\n",
       "                                        SGDClassifier(random_state=42))]),\n",
       "             param_grid={'clf__alpha': [0.0001, 0.001, 0.01, 0.1],\n",
       "                         'clf__max_iter': [1000, 2000, 3000],\n",
       "                         'clf__penalty': ['l1', 'l2', 'elasticnet'],\n",
       "                         'tfidf__max_df': [0.5, 0.75, 1.0]},\n",
       "             scoring='f1_macro')"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best hyperparameters: {'clf__alpha': 0.0001, 'clf__max_iter': 1000, 'clf__penalty': 'l1', 'tfidf__max_df': 0.75}\n",
      "CV score=0.710\n",
      "Evaluating Naive Bayes...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-7 {color: black;background-color: white;}#sk-container-id-7 pre{padding: 0;}#sk-container-id-7 div.sk-toggleable {background-color: white;}#sk-container-id-7 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-7 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-7 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-7 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-7 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-7 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-7 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-7 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-7 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-7 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-7 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-7 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-7 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-7 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-7 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-7 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-7 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-7 div.sk-item {position: relative;z-index: 1;}#sk-container-id-7 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-7 div.sk-item::before, #sk-container-id-7 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-7 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-7 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-7 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-7 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-7 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-7 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-7 div.sk-label-container {text-align: center;}#sk-container-id-7 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-7 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-7\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                                        TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                                       (&#x27;normalize&#x27;, Normalizer()),\n",
       "                                       (&#x27;clf&#x27;, MultinomialNB())]),\n",
       "             param_grid={&#x27;clf__alpha&#x27;: [0.0001, 0.001, 0.01, 0.1],\n",
       "                         &#x27;tfidf__max_df&#x27;: [0.5, 0.75, 1.0]},\n",
       "             scoring=&#x27;f1_macro&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-23\" type=\"checkbox\" ><label for=\"sk-estimator-id-23\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                                        TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                                       (&#x27;normalize&#x27;, Normalizer()),\n",
       "                                       (&#x27;clf&#x27;, MultinomialNB())]),\n",
       "             param_grid={&#x27;clf__alpha&#x27;: [0.0001, 0.001, 0.01, 0.1],\n",
       "                         &#x27;tfidf__max_df&#x27;: [0.5, 0.75, 1.0]},\n",
       "             scoring=&#x27;f1_macro&#x27;)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-24\" type=\"checkbox\" ><label for=\"sk-estimator-id-24\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                (&#x27;normalize&#x27;, Normalizer()), (&#x27;clf&#x27;, MultinomialNB())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-25\" type=\"checkbox\" ><label for=\"sk-estimator-id-25\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-26\" type=\"checkbox\" ><label for=\"sk-estimator-id-26\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Normalizer</label><div class=\"sk-toggleable__content\"><pre>Normalizer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-27\" type=\"checkbox\" ><label for=\"sk-estimator-id-27\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">MultinomialNB</label><div class=\"sk-toggleable__content\"><pre>MultinomialNB()</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[('tfidf',\n",
       "                                        TfidfVectorizer(tokenizer=<function my_tokenizer at 0x000002269C541000>)),\n",
       "                                       ('normalize', Normalizer()),\n",
       "                                       ('clf', MultinomialNB())]),\n",
       "             param_grid={'clf__alpha': [0.0001, 0.001, 0.01, 0.1],\n",
       "                         'tfidf__max_df': [0.5, 0.75, 1.0]},\n",
       "             scoring='f1_macro')"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best hyperparameters: {'clf__alpha': 0.01, 'tfidf__max_df': 0.5}\n",
      "CV score=0.507\n",
      "Evaluating Linear SVC...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-8 {color: black;background-color: white;}#sk-container-id-8 pre{padding: 0;}#sk-container-id-8 div.sk-toggleable {background-color: white;}#sk-container-id-8 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-8 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-8 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-8 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-8 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-8 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-8 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-8 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-8 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-8 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-8 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-8 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-8 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-8 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-8 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-8 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-8 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-8 div.sk-item {position: relative;z-index: 1;}#sk-container-id-8 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-8 div.sk-item::before, #sk-container-id-8 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-8 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-8 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-8 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-8 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-8 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-8 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-8 div.sk-label-container {text-align: center;}#sk-container-id-8 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-8 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-8\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                                        TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                                       (&#x27;normalize&#x27;, Normalizer()),\n",
       "                                       (&#x27;classifier&#x27;, LinearSVC())]),\n",
       "             param_grid={&#x27;classifier__C&#x27;: [0.1, 1, 10],\n",
       "                         &#x27;classifier__max_iter&#x27;: [1000, 2000, 3000],\n",
       "                         &#x27;classifier__penalty&#x27;: [&#x27;l1&#x27;, &#x27;l2&#x27;],\n",
       "                         &#x27;tfidf__max_df&#x27;: [0.5, 0.75, 1.0]},\n",
       "             scoring=&#x27;f1_macro&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-28\" type=\"checkbox\" ><label for=\"sk-estimator-id-28\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                                        TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                                       (&#x27;normalize&#x27;, Normalizer()),\n",
       "                                       (&#x27;classifier&#x27;, LinearSVC())]),\n",
       "             param_grid={&#x27;classifier__C&#x27;: [0.1, 1, 10],\n",
       "                         &#x27;classifier__max_iter&#x27;: [1000, 2000, 3000],\n",
       "                         &#x27;classifier__penalty&#x27;: [&#x27;l1&#x27;, &#x27;l2&#x27;],\n",
       "                         &#x27;tfidf__max_df&#x27;: [0.5, 0.75, 1.0]},\n",
       "             scoring=&#x27;f1_macro&#x27;)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-29\" type=\"checkbox\" ><label for=\"sk-estimator-id-29\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                (&#x27;normalize&#x27;, Normalizer()), (&#x27;classifier&#x27;, LinearSVC())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-30\" type=\"checkbox\" ><label for=\"sk-estimator-id-30\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-31\" type=\"checkbox\" ><label for=\"sk-estimator-id-31\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Normalizer</label><div class=\"sk-toggleable__content\"><pre>Normalizer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-32\" type=\"checkbox\" ><label for=\"sk-estimator-id-32\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LinearSVC</label><div class=\"sk-toggleable__content\"><pre>LinearSVC()</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[('tfidf',\n",
       "                                        TfidfVectorizer(tokenizer=<function my_tokenizer at 0x000002269C541000>)),\n",
       "                                       ('normalize', Normalizer()),\n",
       "                                       ('classifier', LinearSVC())]),\n",
       "             param_grid={'classifier__C': [0.1, 1, 10],\n",
       "                         'classifier__max_iter': [1000, 2000, 3000],\n",
       "                         'classifier__penalty': ['l1', 'l2'],\n",
       "                         'tfidf__max_df': [0.5, 0.75, 1.0]},\n",
       "             scoring='f1_macro')"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best hyperparameters: {'classifier__C': 10, 'classifier__max_iter': 1000, 'classifier__penalty': 'l2', 'tfidf__max_df': 0.75}\n",
      "CV score=0.640\n",
      "Evaluating MLP Classifier...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-9 {color: black;background-color: white;}#sk-container-id-9 pre{padding: 0;}#sk-container-id-9 div.sk-toggleable {background-color: white;}#sk-container-id-9 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-9 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-9 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-9 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-9 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-9 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-9 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-9 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-9 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-9 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-9 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-9 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-9 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-9 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-9 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-9 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-9 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-9 div.sk-item {position: relative;z-index: 1;}#sk-container-id-9 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-9 div.sk-item::before, #sk-container-id-9 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-9 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-9 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-9 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-9 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-9 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-9 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-9 div.sk-label-container {text-align: center;}#sk-container-id-9 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-9 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-9\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                                        TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                                       (&#x27;normalize&#x27;, Normalizer()),\n",
       "                                       (&#x27;classifier&#x27;, MLPClassifier())]),\n",
       "             param_grid={&#x27;classifier__activation&#x27;: [&#x27;logistic&#x27;, &#x27;relu&#x27;],\n",
       "                         &#x27;classifier__hidden_layer_sizes&#x27;: [(50,), (100,)],\n",
       "                         &#x27;classifier__max_iter&#x27;: [1000, 2000],\n",
       "                         &#x27;classifier__solver&#x27;: [&#x27;sgd&#x27;, &#x27;adam&#x27;],\n",
       "                         &#x27;tfidf__max_df&#x27;: [0.5, 0.75, 1.0]},\n",
       "             scoring=&#x27;f1_macro&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-33\" type=\"checkbox\" ><label for=\"sk-estimator-id-33\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                                        TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                                       (&#x27;normalize&#x27;, Normalizer()),\n",
       "                                       (&#x27;classifier&#x27;, MLPClassifier())]),\n",
       "             param_grid={&#x27;classifier__activation&#x27;: [&#x27;logistic&#x27;, &#x27;relu&#x27;],\n",
       "                         &#x27;classifier__hidden_layer_sizes&#x27;: [(50,), (100,)],\n",
       "                         &#x27;classifier__max_iter&#x27;: [1000, 2000],\n",
       "                         &#x27;classifier__solver&#x27;: [&#x27;sgd&#x27;, &#x27;adam&#x27;],\n",
       "                         &#x27;tfidf__max_df&#x27;: [0.5, 0.75, 1.0]},\n",
       "             scoring=&#x27;f1_macro&#x27;)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-34\" type=\"checkbox\" ><label for=\"sk-estimator-id-34\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                (&#x27;normalize&#x27;, Normalizer()), (&#x27;classifier&#x27;, MLPClassifier())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-35\" type=\"checkbox\" ><label for=\"sk-estimator-id-35\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-36\" type=\"checkbox\" ><label for=\"sk-estimator-id-36\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Normalizer</label><div class=\"sk-toggleable__content\"><pre>Normalizer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-37\" type=\"checkbox\" ><label for=\"sk-estimator-id-37\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">MLPClassifier</label><div class=\"sk-toggleable__content\"><pre>MLPClassifier()</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[('tfidf',\n",
       "                                        TfidfVectorizer(tokenizer=<function my_tokenizer at 0x000002269C541000>)),\n",
       "                                       ('normalize', Normalizer()),\n",
       "                                       ('classifier', MLPClassifier())]),\n",
       "             param_grid={'classifier__activation': ['logistic', 'relu'],\n",
       "                         'classifier__hidden_layer_sizes': [(50,), (100,)],\n",
       "                         'classifier__max_iter': [1000, 2000],\n",
       "                         'classifier__solver': ['sgd', 'adam'],\n",
       "                         'tfidf__max_df': [0.5, 0.75, 1.0]},\n",
       "             scoring='f1_macro')"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best hyperparameters: {'classifier__activation': 'logistic', 'classifier__hidden_layer_sizes': (50,), 'classifier__max_iter': 2000, 'classifier__solver': 'adam', 'tfidf__max_df': 0.5}\n",
      "CV score=0.542\n",
      "The best model is Pipeline(steps=[('tfidf',\n",
      "                 TfidfVectorizer(max_df=0.75,\n",
      "                                 tokenizer=<function my_tokenizer at 0x000002269C541000>)),\n",
      "                ('normalize', Normalizer()),\n",
      "                ('clf', SGDClassifier(penalty='l1', random_state=42))]) with an F1 score of 0.709772807622721.\n"
     ]
    }
   ],
   "source": [
    "SGDClassifier_pipeline = Pipeline([\n",
    "    ('tfidf', TfidfVectorizer(tokenizer=my_tokenizer)),\n",
    "    ('normalize', preprocessing.Normalizer(norm='l2')),\n",
    "    ('clf', SGDClassifier(max_iter=1000, random_state=42))\n",
    "])\n",
    "\n",
    "\n",
    "#Naive bayes\n",
    "NB_pipeline = Pipeline([\n",
    "    ('tfidf', TfidfVectorizer(tokenizer=my_tokenizer)),\n",
    "    ('normalize', preprocessing.Normalizer(norm='l2')),\n",
    "    ('clf', MultinomialNB())\n",
    "])\n",
    "\n",
    "\n",
    "#LineaerSVC\n",
    "svc_pipeline = Pipeline([\n",
    "    ('tfidf', TfidfVectorizer(tokenizer=my_tokenizer)),\n",
    "    ('normalize', preprocessing.Normalizer(norm='l2')),\n",
    "    ('classifier', LinearSVC())\n",
    "])\n",
    "\n",
    "#MLPClassifier\n",
    "mlp_pipeline = Pipeline([\n",
    "    ('tfidf', TfidfVectorizer(tokenizer=my_tokenizer)),\n",
    "    ('normalize', preprocessing.Normalizer(norm='l2')),\n",
    "    ('classifier', MLPClassifier())\n",
    "])\n",
    "\n",
    "best_model = None\n",
    "best_f1_score = 0\n",
    "\n",
    "# Define the parameter grids for each pipeline\n",
    "SGDClassifier_param_grid = {\n",
    "    'tfidf__max_df': [0.5, 0.75, 1.0],\n",
    "    'clf__alpha': [0.0001, 0.001, 0.01, 0.1],\n",
    "    'clf__penalty': ['l1', 'l2', 'elasticnet'],\n",
    "    'clf__max_iter': [1000, 2000, 3000]\n",
    "}\n",
    "\n",
    "NB_param_grid = {\n",
    "    'tfidf__max_df': [0.5, 0.75, 1.0],\n",
    "    'clf__alpha': [0.0001, 0.001, 0.01, 0.1]\n",
    "}\n",
    "\n",
    "svc_param_grid = {\n",
    "    'tfidf__max_df': [0.5, 0.75, 1.0],\n",
    "    'classifier__C': [0.1, 1, 10],\n",
    "    'classifier__penalty': ['l1', 'l2'],\n",
    "    'classifier__max_iter': [1000, 2000, 3000]\n",
    "}\n",
    "\n",
    "mlp_param_grid = {\n",
    "    'tfidf__max_df': [0.5, 0.75, 1.0],\n",
    "    'classifier__hidden_layer_sizes': [(50,), (100,),],\n",
    "    'classifier__activation': ['logistic', 'relu'],\n",
    "    'classifier__solver': ['sgd', 'adam'],\n",
    "    'classifier__max_iter': [1000, 2000]\n",
    "}\n",
    "\n",
    "# Define the list of pipelines to evaluate\n",
    "pipelines = [\n",
    "    ('SGDClassifier', SGDClassifier_pipeline, SGDClassifier_param_grid),\n",
    "    ('Naive Bayes', NB_pipeline, NB_param_grid),\n",
    "    ('Linear SVC', svc_pipeline, svc_param_grid),\n",
    "    ('MLP Classifier', mlp_pipeline, mlp_param_grid)\n",
    "]\n",
    "\n",
    "# Loop over the pipelines and perform grid search with cross-validation\n",
    "for name, pipeline, param_grid in pipelines:\n",
    "    print(f\"Evaluating {name}...\")\n",
    "    grid_search = GridSearchCV(pipeline, param_grid, cv=10,scoring='f1_macro')\n",
    "    grid_search.fit(X_train, y_train)\n",
    "    print(f\"Best hyperparameters: {grid_search.best_params_}\")\n",
    "    print(\"CV score=%0.3f\" % grid_search.best_score_)\n",
    "    if grid_search.best_score_ > best_f1_score:\n",
    "        best_f1_score = grid_search.best_score_\n",
    "        best_model = grid_search.best_estimator_\n",
    "\n",
    "# Print the best model\n",
    "print(f\"The best model is {best_model} with an F1 score of {best_f1_score}.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-10 {color: black;background-color: white;}#sk-container-id-10 pre{padding: 0;}#sk-container-id-10 div.sk-toggleable {background-color: white;}#sk-container-id-10 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-10 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-10 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-10 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-10 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-10 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-10 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-10 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-10 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-10 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-10 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-10 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-10 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-10 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-10 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-10 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-10 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-10 div.sk-item {position: relative;z-index: 1;}#sk-container-id-10 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-10 div.sk-item::before, #sk-container-id-10 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-10 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-10 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-10 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-10 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-10 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-10 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-10 div.sk-label-container {text-align: center;}#sk-container-id-10 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-10 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-10\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                                        TfidfVectorizer(max_df=0.65,\n",
       "                                                        ngram_range=(1, 2),\n",
       "                                                        tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                                       (&#x27;normalize&#x27;, Normalizer()),\n",
       "                                       (&#x27;clf&#x27;,\n",
       "                                        SGDClassifier(random_state=42))]),\n",
       "             param_grid={&#x27;clf__alpha&#x27;: [0.0001, 0.001, 0.01, 0.1],\n",
       "                         &#x27;clf__loss&#x27;: [&#x27;hinge&#x27;, &#x27;log&#x27;],\n",
       "                         &#x27;clf__max_iter&#x27;: [30, 40, 50],\n",
       "                         &#x27;clf__penalty&#x27;: [&#x27;l1&#x27;, &#x27;l2&#x27;, &#x27;elasticnet&#x27;],\n",
       "                         &#x27;clf__validation_fraction&#x27;: [0.1, 0.2],\n",
       "                         &#x27;tfidf__max_df&#x27;: [0.5, 0.75, 1.0]},\n",
       "             scoring=&#x27;f1_macro&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-38\" type=\"checkbox\" ><label for=\"sk-estimator-id-38\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                                        TfidfVectorizer(max_df=0.65,\n",
       "                                                        ngram_range=(1, 2),\n",
       "                                                        tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                                       (&#x27;normalize&#x27;, Normalizer()),\n",
       "                                       (&#x27;clf&#x27;,\n",
       "                                        SGDClassifier(random_state=42))]),\n",
       "             param_grid={&#x27;clf__alpha&#x27;: [0.0001, 0.001, 0.01, 0.1],\n",
       "                         &#x27;clf__loss&#x27;: [&#x27;hinge&#x27;, &#x27;log&#x27;],\n",
       "                         &#x27;clf__max_iter&#x27;: [30, 40, 50],\n",
       "                         &#x27;clf__penalty&#x27;: [&#x27;l1&#x27;, &#x27;l2&#x27;, &#x27;elasticnet&#x27;],\n",
       "                         &#x27;clf__validation_fraction&#x27;: [0.1, 0.2],\n",
       "                         &#x27;tfidf__max_df&#x27;: [0.5, 0.75, 1.0]},\n",
       "             scoring=&#x27;f1_macro&#x27;)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-39\" type=\"checkbox\" ><label for=\"sk-estimator-id-39\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(max_df=0.65, ngram_range=(1, 2),\n",
       "                                 tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                (&#x27;normalize&#x27;, Normalizer()),\n",
       "                (&#x27;clf&#x27;, SGDClassifier(random_state=42))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-40\" type=\"checkbox\" ><label for=\"sk-estimator-id-40\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(max_df=0.65, ngram_range=(1, 2),\n",
       "                tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-41\" type=\"checkbox\" ><label for=\"sk-estimator-id-41\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Normalizer</label><div class=\"sk-toggleable__content\"><pre>Normalizer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-42\" type=\"checkbox\" ><label for=\"sk-estimator-id-42\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SGDClassifier</label><div class=\"sk-toggleable__content\"><pre>SGDClassifier(random_state=42)</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=10,\n",
       "             estimator=Pipeline(steps=[('tfidf',\n",
       "                                        TfidfVectorizer(max_df=0.65,\n",
       "                                                        ngram_range=(1, 2),\n",
       "                                                        tokenizer=<function my_tokenizer at 0x000002269C541000>)),\n",
       "                                       ('normalize', Normalizer()),\n",
       "                                       ('clf',\n",
       "                                        SGDClassifier(random_state=42))]),\n",
       "             param_grid={'clf__alpha': [0.0001, 0.001, 0.01, 0.1],\n",
       "                         'clf__loss': ['hinge', 'log'],\n",
       "                         'clf__max_iter': [30, 40, 50],\n",
       "                         'clf__penalty': ['l1', 'l2', 'elasticnet'],\n",
       "                         'clf__validation_fraction': [0.1, 0.2],\n",
       "                         'tfidf__max_df': [0.5, 0.75, 1.0]},\n",
       "             scoring='f1_macro')"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best hyperparameters: {'clf__alpha': 0.0001, 'clf__loss': 'hinge', 'clf__max_iter': 30, 'clf__penalty': 'l1', 'clf__validation_fraction': 0.1, 'tfidf__max_df': 0.75}\n",
      "Cross-validation F1 score: 0.7237609620302812\n"
     ]
    }
   ],
   "source": [
    "SGDClassifier_pipeline = Pipeline([\n",
    "    ('tfidf', TfidfVectorizer(tokenizer=my_tokenizer,max_df=0.65,ngram_range=(1,2))),\n",
    "    ('normalize', preprocessing.Normalizer(norm='l2')),\n",
    "    ('clf', SGDClassifier(random_state=42))\n",
    "])\n",
    "\n",
    "# Set up the parameter grid\n",
    "SGDClassifier_param_grid = {\n",
    "    'tfidf__max_df': [0.5, 0.75, 1.0],\n",
    "    'clf__alpha': [0.0001, 0.001, 0.01, 0.1],\n",
    "    'clf__penalty': ['l1', 'l2', 'elasticnet'],\n",
    "    'clf__max_iter': [30, 40, 50],\n",
    "    'clf__loss': ['hinge', 'log'],\n",
    "    'clf__validation_fraction': [0.1, 0.2]\n",
    "}\n",
    "\n",
    "# Perform grid search\n",
    "grid_search = GridSearchCV(SGDClassifier_pipeline,\n",
    "                           SGDClassifier_param_grid, cv=10, scoring='f1_macro')\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "# Print the best hyperparameters and cross-validation score\n",
    "print(\"Best hyperparameters:\", grid_search.best_params_)\n",
    "print(\"Cross-validation F1 score:\", grid_search.best_score_)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the best avgf1score with TfidfVectorizer and the best params fromm above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-12 {color: black;background-color: white;}#sk-container-id-12 pre{padding: 0;}#sk-container-id-12 div.sk-toggleable {background-color: white;}#sk-container-id-12 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-12 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-12 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-12 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-12 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-12 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-12 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-12 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-12 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-12 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-12 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-12 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-12 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-12 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-12 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-12 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-12 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-12 div.sk-item {position: relative;z-index: 1;}#sk-container-id-12 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-12 div.sk-item::before, #sk-container-id-12 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-12 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-12 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-12 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-12 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-12 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-12 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-12 div.sk-label-container {text-align: center;}#sk-container-id-12 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-12 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-12\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(max_df=0.75, ngram_range=(1, 2),\n",
       "                                 tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                (&#x27;normalize&#x27;, Normalizer()),\n",
       "                (&#x27;clf&#x27;,\n",
       "                 SGDClassifier(max_iter=30, penalty=&#x27;l1&#x27;, random_state=42))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-47\" type=\"checkbox\" ><label for=\"sk-estimator-id-47\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 TfidfVectorizer(max_df=0.75, ngram_range=(1, 2),\n",
       "                                 tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                (&#x27;normalize&#x27;, Normalizer()),\n",
       "                (&#x27;clf&#x27;,\n",
       "                 SGDClassifier(max_iter=30, penalty=&#x27;l1&#x27;, random_state=42))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-48\" type=\"checkbox\" ><label for=\"sk-estimator-id-48\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(max_df=0.75, ngram_range=(1, 2),\n",
       "                tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-49\" type=\"checkbox\" ><label for=\"sk-estimator-id-49\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Normalizer</label><div class=\"sk-toggleable__content\"><pre>Normalizer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-50\" type=\"checkbox\" ><label for=\"sk-estimator-id-50\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SGDClassifier</label><div class=\"sk-toggleable__content\"><pre>SGDClassifier(max_iter=30, penalty=&#x27;l1&#x27;, random_state=42)</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('tfidf',\n",
       "                 TfidfVectorizer(max_df=0.75, ngram_range=(1, 2),\n",
       "                                 tokenizer=<function my_tokenizer at 0x000002269C541000>)),\n",
       "                ('normalize', Normalizer()),\n",
       "                ('clf',\n",
       "                 SGDClassifier(max_iter=30, penalty='l1', random_state=42))])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 score for best model: 0.7237609620302812\n"
     ]
    }
   ],
   "source": [
    "best_params = {\n",
    "    'tfidf__max_df': 0.75,\n",
    "    'clf__alpha': 0.0001,\n",
    "    'clf__penalty': 'l1',\n",
    "    'clf__max_iter': 30,\n",
    "    'clf__loss': 'hinge',\n",
    "    'clf__validation_fraction': 0.1\n",
    "}\n",
    "\n",
    "best_model_tfidf = Pipeline([\n",
    "    ('tfidf', TfidfVectorizer(tokenizer=my_tokenizer, max_df=best_params['tfidf__max_df'], ngram_range=(1, 2))),\n",
    "    ('normalize', preprocessing.Normalizer(norm='l2')),\n",
    "    ('clf', SGDClassifier(random_state=42, alpha=best_params['clf__alpha'], penalty=best_params['clf__penalty'], \n",
    "                           max_iter=best_params['clf__max_iter'], loss=best_params['clf__loss'],\n",
    "                           validation_fraction=best_params['clf__validation_fraction']))\n",
    "])\n",
    "\n",
    "# SGDClassifier_pipeline = Pipeline([\n",
    "#     ('tfidf', TfidfVectorizer(tokenizer=my_tokenizer,max_df=0.65,ngram_range=(1,2))),\n",
    "#     ('normalize', preprocessing.Normalizer(norm='l2')),\n",
    "#     ('clf', SGDClassifier(random_state=42))\n",
    "# ])\n",
    "\n",
    "best_model_tfidf.fit(X_train, y_train)\n",
    "f1_scores = cross_val_score(best_model_tfidf, X_train, y_train, cv=10, scoring='f1_macro')\n",
    "avg_f1_score = np.mean(f1_scores)\n",
    "print(\"F1 score for best model:\", avg_f1_score)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the best avgf1score with CountVectorizer and the best params from above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-13 {color: black;background-color: white;}#sk-container-id-13 pre{padding: 0;}#sk-container-id-13 div.sk-toggleable {background-color: white;}#sk-container-id-13 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-13 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-13 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-13 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-13 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-13 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-13 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-13 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-13 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-13 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-13 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-13 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-13 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-13 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-13 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-13 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-13 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-13 div.sk-item {position: relative;z-index: 1;}#sk-container-id-13 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-13 div.sk-item::before, #sk-container-id-13 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-13 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-13 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-13 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-13 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-13 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-13 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-13 div.sk-label-container {text-align: center;}#sk-container-id-13 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-13 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-13\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 CountVectorizer(max_df=0.65, ngram_range=(1, 2),\n",
       "                                 tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                (&#x27;normalize&#x27;, Normalizer()),\n",
       "                (&#x27;clf&#x27;,\n",
       "                 SGDClassifier(eta0=1, max_iter=22, n_iter_no_change=6,\n",
       "                               penalty=&#x27;l1&#x27;, random_state=42))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-51\" type=\"checkbox\" ><label for=\"sk-estimator-id-51\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;,\n",
       "                 CountVectorizer(max_df=0.65, ngram_range=(1, 2),\n",
       "                                 tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)),\n",
       "                (&#x27;normalize&#x27;, Normalizer()),\n",
       "                (&#x27;clf&#x27;,\n",
       "                 SGDClassifier(eta0=1, max_iter=22, n_iter_no_change=6,\n",
       "                               penalty=&#x27;l1&#x27;, random_state=42))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-52\" type=\"checkbox\" ><label for=\"sk-estimator-id-52\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">CountVectorizer</label><div class=\"sk-toggleable__content\"><pre>CountVectorizer(max_df=0.65, ngram_range=(1, 2),\n",
       "                tokenizer=&lt;function my_tokenizer at 0x000002269C541000&gt;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-53\" type=\"checkbox\" ><label for=\"sk-estimator-id-53\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Normalizer</label><div class=\"sk-toggleable__content\"><pre>Normalizer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-54\" type=\"checkbox\" ><label for=\"sk-estimator-id-54\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SGDClassifier</label><div class=\"sk-toggleable__content\"><pre>SGDClassifier(eta0=1, max_iter=22, n_iter_no_change=6, penalty=&#x27;l1&#x27;,\n",
       "              random_state=42)</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('tfidf',\n",
       "                 CountVectorizer(max_df=0.65, ngram_range=(1, 2),\n",
       "                                 tokenizer=<function my_tokenizer at 0x000002269C541000>)),\n",
       "                ('normalize', Normalizer()),\n",
       "                ('clf',\n",
       "                 SGDClassifier(eta0=1, max_iter=22, n_iter_no_change=6,\n",
       "                               penalty='l1', random_state=42))])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average F1 score: 0.6858596615032063\n"
     ]
    }
   ],
   "source": [
    "best_model_countvectorizer = Pipeline(\n",
    "    [   ('tfidf', CountVectorizer(tokenizer=my_tokenizer,max_df=0.65, ngram_range=(1, 2))),\n",
    "        ('normalize', preprocessing.Normalizer(norm='l2')),\n",
    "        ('clf', SGDClassifier(max_iter=22, random_state=42, alpha=0.0001, penalty='l1', loss='hinge',\n",
    "                              learning_rate='optimal', eta0=1, validation_fraction=0.1, n_iter_no_change=6))\n",
    "    ])\n",
    "\n",
    "best_model_countvectorizer.fit(X_train, y_train)\n",
    "f1_scores = cross_val_score(best_model_countvectorizer, X_train, y_train, cv=10, scoring='f1_macro')\n",
    "avg_f1_score = np.mean(f1_scores)\n",
    "print(\"Average F1 score:\", avg_f1_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_example_id</th>\n",
       "      <th>predicted_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>318</th>\n",
       "      <td>318</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>319</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>320</th>\n",
       "      <td>320</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>321</th>\n",
       "      <td>321</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>322</th>\n",
       "      <td>322</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>323 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     test_example_id predicted_category\n",
       "0                  0                  m\n",
       "1                  1                  m\n",
       "2                  2                  m\n",
       "3                  3                  m\n",
       "4                  4                  f\n",
       "..               ...                ...\n",
       "318              318                  m\n",
       "319              319                  m\n",
       "320              320                  m\n",
       "321              321                  m\n",
       "322              322                  m\n",
       "\n",
       "[323 rows x 2 columns]"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_predicted = pd.DataFrame(columns = ['test_example_id','predicted_category'])\n",
    "y_pred = best_model_tfidf.predict(df_test[\"story\"])\n",
    "df_predicted['test_example_id'] = df_test['test_example_id']\n",
    "df_predicted['predicted_category'] = np.where(y_pred == 1, 'm', 'f')\n",
    "df_predicted"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save output to csv (optional)\n",
    "After you're done save your output to the 'classification_results.csv' csv file.<br/>\n",
    "We assume that the dataframe with your results contain the following columns:\n",
    "* column 1 (left column): 'test_example_id'  - the same id associated to each of the test stories to be predicted.\n",
    "* column 2 (right column): 'predicted_category' - the predicted gender value for each of the associated story. \n",
    "\n",
    "Assuming your predicted values are in the `df_predicted` dataframe, you should save you're results as following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_predicted.to_csv('classification_results.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Create Assignment",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  },
  "vscode": {
   "interpreter": {
    "hash": "afb734500600fd355917ca529030176ea0ca205570884b88f2f6f7d791fd3fbe"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}