movie_sentiment_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Welcome\n",
    "This is a quick tutorial of how you can use existing LLMs to perform precise sentiment analysis on sample data.\n",
    "\n",
    "**Created By: Chen Wenlong**\n",
    "\n",
    "**Date: 6 March 2024**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick Start\n",
    "\n",
    "Before we begin, since we will be using the imdb dataset from [huggingface](https://huggingface.co/datasets/imdb), we first need to install the datasets library using pip. We will also need to install some dependencies: transformers to tokenise our text, torch to handle our tensors, scipy (softmark) to normalise the probabilities the sentiment scores and pandas to manipulate our dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install datasets transformers torch pandas scipy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simulate the model, I have copied over some sample data from the imdb dataset (top 5 and bottom 5), the first 5 are negative comments while the last 5 are positive comments. You can skip this if you want to access the full dataset below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "neg_data = [{\"text\":\"I rented I AM CURIOUS-YELLOW from my video store because of all the controversy that surrounded it when it was first released in 1967. I also heard that at first it was seized by U.S. customs if it ever tried to enter this country, therefore being a fan of films considered \\\"controversial\\\" I really had to see this for myself.<br \\/><br \\/>The plot is centered around a young Swedish drama student named Lena who wants to learn everything she can about life. In particular she wants to focus her attentions to making some sort of documentary on what the average Swede thought about certain political issues such as the Vietnam War and race issues in the United States. In between asking politicians and ordinary denizens of Stockholm about their opinions on politics, she has sex with her drama teacher, classmates, and married men.<br \\/><br \\/>What kills me about I AM CURIOUS-YELLOW is that 40 years ago, this was considered pornographic. Really, the sex and nudity scenes are few and far between, even then it's not shot like some cheaply made porno. While my countrymen mind find it shocking, in reality sex and nudity are a major staple in Swedish cinema. Even Ingmar Bergman, arguably their answer to good old boy John Ford, had sex scenes in his films.<br \\/><br \\/>I do commend the filmmakers for the fact that any sex shown in the film is shown for artistic purposes rather than just to shock people and make money to be shown in pornographic theaters in America. I AM CURIOUS-YELLOW is a good film for anyone wanting to study the meat and potatoes (no pun intended) of Swedish cinema. But really, this film doesn't have much of a plot.\",\"label\":0},\n",
    "{\"text\":\"\\\"I Am Curious: Yellow\\\" is a risible and pretentious steaming pile. It doesn't matter what one's political views are because this film can hardly be taken seriously on any level. As for the claim that frontal male nudity is an automatic NC-17, that isn't true. I've seen R-rated films with male nudity. Granted, they only offer some fleeting views, but where are the R-rated films with gaping vulvas and flapping labia? Nowhere, because they don't exist. The same goes for those crappy cable shows: schlongs swinging in the breeze but not a clitoris in sight. And those pretentious indie movies like The Brown Bunny, in which we're treated to the site of Vincent Gallo's throbbing johnson, but not a trace of pink visible on Chloe Sevigny. Before crying (or implying) \\\"double-standard\\\" in matters of nudity, the mentally obtuse should take into account one unavoidably obvious anatomical difference between men and women: there are no genitals on display when actresses appears nude, and the same cannot be said for a man. In fact, you generally won't see female genitals in an American film in anything short of porn or explicit erotica. This alleged double-standard is less a double standard than an admittedly depressing ability to come to terms culturally with the insides of women's bodies.\",\"label\":0},\n",
    "{\"text\":\"If only to avoid making this type of film in the future. This film is interesting as an experiment but tells no cogent story.<br \\/><br \\/>One might feel virtuous for sitting thru it because it touches on so many IMPORTANT issues but it does so without any discernable motive. The viewer comes away with no new perspectives (unless one comes up with one while one's mind wanders, as it will invariably do during this pointless film).<br \\/><br \\/>One might better spend one's time staring out a window at a tree growing.<br \\/><br \\/>\",\"label\":0},\n",
    "{\"text\":\"This film was probably inspired by Godard's Masculin, f\\u00e9minin and I urge you to see that film instead.<br \\/><br \\/>The film has two strong elements and those are, (1) the realistic acting (2) the impressive, undeservedly good, photo. Apart from that, what strikes me most is the endless stream of silliness. Lena Nyman has to be most annoying actress in the world. She acts so stupid and with all the nudity in this film,...it's unattractive. Comparing to Godard's film, intellectuality has been replaced with stupidity. Without going too far on this subject, I would say that follows from the difference in ideals between the French and the Swedish society.<br \\/><br \\/>A movie of its time, and place. 2\\/10.\",\"label\":0},\n",
    "{\"text\":\"Oh, brother...after hearing about this ridiculous film for umpteen years all I can think of is that old Peggy Lee song..<br \\/><br \\/>\\\"Is that all there is??\\\" ...I was just an early teen when this smoked fish hit the U.S. I was too young to get in the theater (although I did manage to sneak into \\\"Goodbye Columbus\\\"). Then a screening at a local film museum beckoned - Finally I could see this film, except now I was as old as my parents were when they schlepped to see it!!<br \\/><br \\/>The ONLY reason this film was not condemned to the anonymous sands of time was because of the obscenity case sparked by its U.S. release. MILLIONS of people flocked to this stinker, thinking they were going to see a sex film...Instead, they got lots of closeups of gnarly, repulsive Swedes, on-street interviews in bland shopping malls, asinie political pretension...and feeble who-cares simulated sex scenes with saggy, pale actors.<br \\/><br \\/>Cultural icon, holy grail, historic artifact..whatever this thing was, shred it, burn it, then stuff the ashes in a lead box!<br \\/><br \\/>Elite esthetes still scrape to find value in its boring pseudo revolutionary political spewings..But if it weren't for the censorship scandal, it would have been ignored, then forgotten.<br \\/><br \\/>Instead, the \\\"I Am Blank, Blank\\\" rhythymed title was repeated endlessly for years as a titilation for porno films (I am Curious, Lavender - for gay films, I Am Curious, Black - for blaxploitation films, etc..) and every ten years or so the thing rises from the dead, to be viewed by a new generation of suckers who want to see that \\\"naughty sex film\\\" that \\\"revolutionized the film industry\\\"...<br \\/><br \\/>Yeesh, avoid like the plague..Or if you MUST see it - rent the video and fast forward to the \\\"dirty\\\" parts, just to get it over with.<br \\/><br \\/>\",\"label\":0}]\n",
    "\n",
    "pos_data = [{\"text\":\"A hit at the time but now better categorised as an Australian cult film. The humour is broad, unsubtle and, in the final scene where a BBC studio fire is extinguished by urinating on it, crude. Contains just about every cliche about the traditional Australian pilgrimage to 'the old country', and every cliche about those rapacious, stuck up, whinging, Tory Brits. Would be acceptable to the British because of its strong cast of well known actors, and to Australians of that generation, who can 'get' the humour. Americans -- forget it. The language and jokes are in the Australian dialect of English and as such will be unintelligible.\",\"label\":1},\n",
    "{\"text\":\"I love this movie like no other. Another time I will try to explain its virtues to the uninitiated, but for the moment let me quote a few of pieces the remarkable dialogue, which, please remember, is all tongue in cheek. Aussies and Poms will understand, everyone else-well?<br \\/><br \\/>(title song lyric)\\\"he can sink a beer, he can pick a queer, in his latest double-breasted Bondi gear.\\\"<br \\/><br \\/>(another song lyric) \\\"All pommies are bastards, bastards, or worse, and England is the a**e-hole of the universe.\\\"<br \\/><br \\/>(during a television interview on an \\\"arty program\\\"): Mr Mackenzie what artists have impressed you most since you've been in England? (Barry's response)Flamin' bull-artists!<br \\/><br \\/>(while chatting up a naive young pom girl): Mr Mackenzie, I suppose you have hordes of Aboriginal servants back in Australia? (Barry's response) Abos? I've never seen an Abo in me life. Mum does most of the solid yacca (ie hard work) round our place.<br \\/><br \\/>This is just a taste of the hilarious farce of this bonser Aussie flick. If you can get a copy of it, watch and enjoy.\",\"label\":1},\n",
    "{\"text\":\"This film and it's sequel Barry Mckenzie holds his own, are the two greatest comedies to ever be produced. A great story a young Aussie bloke travels to england to claim his inheritance and meets up with his mates, who are just as loveable and innocent as he is.<br \\/><br \\/>It's chock a block full of great, sayings , where else could you find someone who needs a drink so bad that he's as dry as a dead dingoes donger? great characters, top acting, and it's got great sheilas and more Fosters consumption then any other three films put together. Top notch.<br \\/><br \\/>And some of the funniest songs you'll ever hear, and it's full of great celebrities. Definitely my two favourite films of all time, I watch them at least once a fortnight.\",\"label\":1},\n",
    "{\"text\":\"'The Adventures Of Barry McKenzie' started life as a satirical comic strip in 'Private Eye', written by Barry Humphries and based on an idea by Peter Cook. McKenzie ( 'Bazza' to his friends ) is a lanky, loud, hat-wearing Australian whose two main interests in life are sex ( despite never having had any ) and Fosters lager. In 1972, he found his way to the big screen for the first of two outings. It must have been tempting for Humphries to cast himself as 'Bazza', but he wisely left the job to Barry Crocker ( later to sing the theme to the television soap opera 'Neighbours'! ). Humphries instead played multiple roles in true Peter Sellers fashion, most notably Bazza's overbearing Aunt 'Edna Everage' ( this was before she became a Dame ).<br \\/><br \\/>You know this is not going to be 'The Importance Of Being Ernest' when its censorship classification N.P.A. stands for 'No Poofters Allowed'. Pom-hating Bazza is told by a Sydney solicitor that in order to inherit a share in his father's will he must go to England to absorb British culture. With Aunt Edna in tow, he catches a Quantas flight to Hong Kong, and then on to London. An over-efficient customs officer makes Bazza pay import duties on everything he bought over there, including a suitcase full of 'tubes of Fosters lager'. As he puts it: \\\"when it comes to fleecing you, the Poms have got the edge on the gyppos!\\\". A crafty taxi driver ( Bernard Spear ) maximises the fare by taking Bazza and Edna first to Stonehenge, then Scotland. The streets of London are filthy, and their hotel is a hovel run by a seedy landlord ( Spike Milligan ) who makes Bazza put pound notes in the electricity meter every twenty minutes. There is some good news for our hero though; he meets up with other Aussies in Earls Court, and Fosters is on sale in British pubs.<br \\/><br \\/>What happens next is a series of comical escapades that take Bazza from starring in his own cigarette commercial, putting curry down his pants in the belief it is some form of aphrodisiac, a bizarre encounter with Dennis Price as an upper-class pervert who loves being spanked while wearing a schoolboy's uniform, a Young Conservative dance in Rickmansworth to a charity rock concert where his song about 'chundering' ( vomiting ) almost makes him an international star, and finally to the B.B.C. T.V. Centre where he pulls his pants down on a live talk-show hosted by the thinking man's crumpet herself, Joan Bakewell. A fire breaks out, and Bazza's friends come to the rescue - downing cans of Fosters, they urinate on the flames en masse.<br \\/><br \\/>This is a far cry from Bruce Beresford's later works - 'Breaker Morant' and 'Driving Miss Daisy'. On release, it was savaged by critics for being too 'vulgar'. Well, yes, it is, but it is also great non-P.C. fun. 'Bazza' is a disgusting creation, but his zest for life is unmistakable, you cannot help but like the guy. His various euphemisms for urinating ( 'point Percy at the porcelain' ) and vomiting ( 'the Technicolour yawn' ) have passed into the English language without a lot of people knowing where they came from. Other guest stars include Dick Bentley ( as a detective who chases Bazza everywhere ), Peter Cook, Julie Covington ( later to star in 'Rock Follies' ), and even future arts presenter Russell Davies.<br \\/><br \\/>A sequel - the wonderfully-named 'Barry McKenzie Holds His Own - came out two years later. At its premiere, Humphries took the opportunity to blast the critics who had savaged the first film. Good for him.<br \\/><br \\/>What must have been of greater concern to him, though, was the release of 'Crocodile Dundee' in 1985. It also featured a lanky, hat-wearing Aussie struggling to come to terms with a foreign culture. And made tonnes more money.<br \\/><br \\/>The song on the end credits ( performed by Snacka Fitzgibbon ) is magnificent. You have a love a lyric that includes the line: \\\"If you want to send your sister in a frenzy, introduce her to Barry McKenzie!\\\". Time to end this review. I have to go the dunny to shake hands with the unemployed...\",\"label\":1},\n",
    "{\"text\":\"The story centers around Barry McKenzie who must go to England if he wishes to claim his inheritance. Being about the grossest Aussie shearer ever to set foot outside this great Nation of ours there is something of a culture clash and much fun and games ensue. The songs of Barry McKenzie(Barry Crocker) are highlights.\",\"label\":1}]\n",
    "\n",
    "data = neg_data + pos_data\n",
    "\n",
    "df = pd.DataFrame(data)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we have downloaded the datasets library, we will import the imdb dataset into our local environment and load in into a `dataset` variable. The dataset is large and has 25000 entires, so it might take slightly longer to load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "import pandas as pd\n",
    "\n",
    "# Load the IMDb dataset, only the train split\n",
    "dataset = load_dataset(\"imdb\", split=\"train\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we can see how many data points we want to run through for sentiment analysis. \n",
    "\n",
    "Edit the `datapoints` variable to set the number of datapoints you want to run through!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datapoints = 100\n",
    "\n",
    "# take from the first 50 (negative) and the last 50 (positive) examples\n",
    "df = pd.concat([pd.DataFrame(dataset[:int(datapoints/2)]), pd.DataFrame(dataset[-int(datapoints/2):])])\n",
    "\n",
    "# Reset the index\n",
    "df = df.reset_index(drop=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now instantiate the tokenizer and model. This may take a minute or more because we are instantiating the model, and downloading 500MB of model data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n",
    "\n",
    "tokenizer_tr = AutoTokenizer.from_pretrained(\"cardiffnlp/twitter-roberta-base-sentiment-latest\")\n",
    "model_tr = AutoModelForSequenceClassification.from_pretrained(\"cardiffnlp/twitter-roberta-base-sentiment-latest\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will create a method that will be used to parse each text (tokens) through to retrieve the sentiment analysis results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.special import softmax\n",
    "\n",
    "def sentiment_score(review):\n",
    "    tokens = tokenizer_tr.encode(review, return_tensors='pt') # encode the review, return_tensors='pt' returns pytorch tensors\n",
    "    result = model_tr(tokens)\n",
    "    score = result[0][0].detach().numpy()\n",
    "    score = softmax(score)\n",
    "    scores_dict = {\n",
    "        \"negative\": score[0].item(),\n",
    "        \"neutral\": score[1].item(),\n",
    "        \"positive\": score[2].item()\n",
    "    }\n",
    "    return scores_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we iterate through the dataset and return the sentiment scores. I placed checks here to ensure we know which lines are not fully parsed due to exceeding token count and handle the exception accordingly.\n",
    "\n",
    "Uncomment the last line to see how many rows did not parse fully due to token limit exceeded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "count = 0 # counter of number of rows that did not parse fully\n",
    "rows_skipped = [] # list of errors\n",
    "\n",
    "for i, row in tqdm(df.iterrows(), total=len(df)): # tqdm helps us track the progress of the loop via a progress bar\n",
    "    try:\n",
    "        df.at[i, 'full_sentiment_tr'] = sentiment_score(row['text'])\n",
    "    except RuntimeError as e: # catch error, likely to be because the string is too long.\n",
    "        df.at[i, 'full_sentiment_tr'] = sentiment_score(row['text'][:512]) \n",
    "        count+=1\n",
    "        rows_skipped.append(i)\n",
    "    finally:\n",
    "        df.at[i, 'sentiment_tr']= max(df.at[i, 'full_sentiment_tr'], key=df.at[i, 'full_sentiment_tr'].get)\n",
    "\n",
    "df # prints out the df\n",
    "\n",
    "# Print the number of rows that did not parse fully\n",
    "# print(count, \"rows did not parse fully\")\n",
    "# print(\"Rows skipped:\", rows_skipped) # print the rows that did not parse fully\n",
    "\n",
    "# Concise way to do the same thing using native pandas methods (note for self)\n",
    "# df['full_sentiment_tr'] = df['text'].apply(lambda x: sentiment_score(x[:512]))\n",
    "# df['highest_tr']= df['full_sentiment_tr'].apply(lambda x: max(x, key=x.get))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another interesting way to do this is via another form of rating on a scale of 1 to 5, similar to how an actual movie review is done."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we instantiate the bert multilingual model. This may take awhile."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n",
    "\n",
    "tokenizer_bm = AutoTokenizer.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment')\n",
    "model_bm = AutoModelForSequenceClassification.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we define a new function to analyse the text, similar to the method structure from the previous method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "def sentiment_score_bm(review):\n",
    "    tokens = tokenizer_bm.encode(review, return_tensors='pt') # encode the review, return_tensors='pt' returns pytorch tensors\n",
    "    result = model_bm(tokens)\n",
    "    return int(torch.argmax(result.logits))+1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We run this model and append the score to the dataframe. We will obtain the raw score and then create a new column where it will reflect neutral for `score == 3`, positive for `score > 3` and negative for `score < 3`.\n",
    "\n",
    "Note that number of lines partially parsed can be different between the 2 methods because of the different tokenizer we are using for each respective model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "count = 0 # counter of number of rows that did not parse fully\n",
    "rows_skipped = []\n",
    "for i, row in tqdm(df.iterrows(), total=len(df)): # tqdm helps us track the progress of the loop via a progress bar\n",
    "    try:\n",
    "        df.at[i, 'full_sentiment_bm'] = sentiment_score_bm(row['text'])\n",
    "    except RuntimeError as e: # catch error, likely to be because the string is too long.\n",
    "        df.at[i, 'full_sentiment_bm'] = sentiment_score_bm(row['text'][:512]) \n",
    "        count+=1\n",
    "        rows_skipped.append(i)\n",
    "    finally:\n",
    "        df.at[i, 'sentiment_bm']= \"positive\" if df.at[i, 'full_sentiment_bm']>3 else \"neutral\" if df.at[i, 'full_sentiment_bm']==3 else \"negative\"\n",
    "\n",
    "df\n",
    "\n",
    "# Print the number of rows that did not parse fully\n",
    "# print(count, \"rows did not parse fully\")\n",
    "# print(\"Rows skipped:\", rows_skipped) # print the rows that did not parse fully\n",
    "\n",
    "# Concise way to do the same thing using native pandas methods (note for self)\n",
    "# df['full_sentiment_bm'] = df['text'].apply(lambda x: sentiment_score_bm(x[:512])) # we limit the token size as per model restriction\n",
    "# df['highest_bm'] = df['full_sentiment_bm'].apply(lambda x: \"positive\" if x==3 else \"neutral\" if x==2 else \"negative\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** What is the accuracy and how do they compare? To find out the efficacy of each model, we will find the `accruacy`, `precision`, `recall` and `F1 score`. Below are fomulas to calculate each.\n",
    "\n",
    "TP = True Positive, TN = True Negative, FP = False Positive, FN = False Negative.\n",
    "**Accuracy:**\n",
    "$$ \\text{Accuracy} = \\frac{TP + TN}{TP + FN + FP + TN} $$\n",
    "\n",
    "**Precision (Positive Predictive Value):**\n",
    "$$ \\text{Precision} = \\frac{TP}{TP + FP} $$\n",
    "\n",
    "**Recall (Sensitivity, True Positive Rate):**\n",
    "$$ \\text{Recall} = \\frac{TP}{TP + FN} $$\n",
    "\n",
    "**F1 Score (Harmonic Mean of Precision and Recall):**\n",
    "$$ \\text{F1 Score} = 2 \\times \\frac{\\text{Precision} \\times \\text{Recall}}{\\text{Precision} + \\text{Recall}} $$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate true positive of the sentiment analysis compared to the label twitter roberta\n",
    "true_positives = df.apply(lambda x: 1 if x['sentiment_tr'] == 'positive' and x['label'] == 1 else 0, axis=1).sum()\n",
    "false_positives = df.apply(lambda x: 1 if x['sentiment_tr'] == 'positive' and x['label'] == 0 else 0, axis=1).sum()\n",
    "true_negatives = df.apply(lambda x: 1 if x['sentiment_tr'] == 'negative' and x['label'] == 0 else 0, axis=1).sum()\n",
    "false_negatives = df.apply(lambda x: 1 if x['sentiment_tr'] == 'negative' and x['label'] == 1 else 0, axis=1).sum()\n",
    "# print(true_positives, false_positives, true_negatives, false_negatives)\n",
    "\n",
    "accuracy = (true_positives + true_negatives) / (true_positives + false_positives + true_negatives + false_negatives)\n",
    "precision = true_positives / (true_positives + false_positives)\n",
    "recall = true_positives / (true_positives + false_negatives)\n",
    "f1 = 2 * (precision * recall) / (precision + recall)\n",
    "\n",
    "print(\"Twi1tter RoBERTa\")\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Precision: {precision}\")\n",
    "print(f\"Recall: {recall}\")\n",
    "print(f\"F1 Score: {f1}\")\n",
    "\n",
    "# calculate true positive of the sentiment analysis compared to the label for BERT\n",
    "true_positives = df.apply(lambda x: 1 if x['sentiment_bm'] == 'positive' and x['label'] == 1 else 0, axis=1).sum()\n",
    "false_positives = df.apply(lambda x: 1 if x['sentiment_bm'] == 'positive' and x['label'] == 0 else 0, axis=1).sum()\n",
    "true_negatives = df.apply(lambda x: 1 if x['sentiment_bm'] == 'negative' and x['label'] == 0 else 0, axis=1).sum()\n",
    "false_negatives = df.apply(lambda x: 1 if x['sentiment_bm'] == 'negative' and x['label'] == 1 else 0, axis=1).sum()\n",
    "# print(true_positives, false_positives, true_negatives, false_negatives)\n",
    "\n",
    "accuracy = (true_positives + true_negatives) / (true_positives + false_positives + true_negatives + false_negatives)\n",
    "precision = true_positives / (true_positives + false_positives)\n",
    "recall = true_positives / (true_positives + false_negatives)\n",
    "f1 = 2 * (precision * recall) / (precision + recall)\n",
    "\n",
    "print(\"BERT Multilingual\")\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Precision: {precision}\")\n",
    "print(f\"Recall: {recall}\")\n",
    "print(f\"F1 Score: {f1}\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can create a heatmap to visualise the confusion matrix of the model.\n",
    "\n",
    "We need to download more libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install matplotlib seaborn scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Convert 'sentiment' to numerical values\n",
    "df['predicted_label'] = df['sentiment_tr'].apply(lambda x: 1 if x == 'positive' else 0)\n",
    "\n",
    "# Generate the confusion matrix\n",
    "cm = confusion_matrix(df['label'], df['predicted_label'])\n",
    "\n",
    "# Plotting the confusion matrix using Seaborn\n",
    "plt.figure(figsize=(8,6))\n",
    "sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')\n",
    "plt.title('Confusion Matrix For Twitter-RoBERTa')\n",
    "plt.xlabel('Predicted Label')\n",
    "plt.ylabel('True Label')\n",
    "plt.xticks([0.5, 1.5], ['Negative', 'Positive'])\n",
    "plt.yticks([0.5, 1.5], ['Negative', 'Positive'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do the same for the BERT multilingual model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Convert 'sentiment' to numerical values\n",
    "df['predicted_label'] = df['sentiment_bm'].apply(lambda x: 1 if x == 'positive' else 0)\n",
    "\n",
    "# Generate the confusion matrix\n",
    "cm = confusion_matrix(df['label'], df['predicted_label'])\n",
    "\n",
    "# Plotting the confusion matrix using Seaborn\n",
    "plt.figure(figsize=(8,6))\n",
    "sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')\n",
    "plt.title('Confusion Matrix For Twitter-RoBERTa')\n",
    "plt.xlabel('Predicted Label')\n",
    "plt.ylabel('True Label')\n",
    "plt.xticks([0.5, 1.5], ['Negative', 'Positive'])\n",
    "plt.yticks([0.5, 1.5], ['Negative', 'Positive'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final Thoughts\n",
    "\n",
    "In general, both models performed very well in terms identifying negative comments, whereas positive comments are not as accurate. Although not significant, the BERT model did perform better than the twitter roberta model. I used the roberta model because it was the most downloaded model at the time of making this (48.3M downloads). That's pretty insane. However, the second model, with its rating system (out of 5), seemed more fitting for the specific task of analysing movie rating. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}