02.2_Premium.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Detailed-Dynamic Context-Aware Summarization\n",
    "## MEthods Names-:\n",
    "* **TF-IDF Vectorization**: Used to convert text into a numerical format that can be easily analyzed. Specifically used for keyword extraction.\n",
    "* **Cosine Similarity**: To find the similarity between sentences.\n",
    "* **Named Entity Recognition (NER)**: To extract important names, places, and other entities from the text.\n",
    "* **Sentence Tokenization**: To break down the text into individual sentences.\n",
    "* **Stop Words Removal**: To ignore commonly occurring words that don't carry significant meaning.\n",
    "* **Dynamic Detail Levels**: To adjust the length and detail of the summary.\n",
    "* **Context-based Weightage**: To adjust the number of sentences in the summary based on the type of text (e.g., story, report, article)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import spacy\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics.pairwise import linear_kernel\n",
    "from nltk.tokenize import sent_tokenize\n",
    "# from sklearn.feature_extraction.stop_words import ENGLISH_STOP_WORDS\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, ENGLISH_STOP_WORDS\n",
    "\n",
    "# # # from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "# # # stop_words = TfidfVectorizer(stop_words='english').get_stop_words()\n",
    "\n",
    "\n",
    "# Function to extract important keywords using TF-IDF\n",
    "def extract_important_keywords(input_text, num_keywords=5):\n",
    "    my_stop_words = list(ENGLISH_STOP_WORDS)\n",
    "    tfidf_vectorizer = TfidfVectorizer(stop_words=my_stop_words, min_df=1)\n",
    "    tfidf_matrix = tfidf_vectorizer.fit_transform([input_text])\n",
    "    feature_names = np.array(tfidf_vectorizer.get_feature_names_out())\n",
    "    tfidf_values = tfidf_matrix.toarray()[0]\n",
    "    sorted_indices = tfidf_values.argsort()[::-1]\n",
    "    top_keywords = feature_names[sorted_indices][:num_keywords]\n",
    "    return top_keywords.tolist()\n",
    "\n",
    "# Function to extract named entities using SpaCy\n",
    "def extract_named_entities(input_text):\n",
    "    nlp = spacy.load(\"en_core_web_sm\")\n",
    "    doc = nlp(input_text)\n",
    "    named_entities = [ent.text.lower() for ent in doc.ents]\n",
    "    named_entities=named_entities\n",
    "    return list(set(named_entities))\n",
    "\n",
    "# Function to generate dynamic detailed summaries\n",
    "# Let's make a modification to the sentence ordering in the summary generation function.\n",
    "def dynamic_detailed_summarize_premium(input_text, detail_level=1, context_type='general', important_keywords=None, important_entities=None):\n",
    "    if important_keywords is None:\n",
    "        important_keywords = extract_important_keywords(input_text)\n",
    "    \n",
    "    if important_entities is None:\n",
    "        important_entities = extract_named_entities(input_text)\n",
    "\n",
    "\n",
    "    print(f'Keywords are-: {important_keywords}')\n",
    "    print(f'Entities are-: {important_entities}')    \n",
    "    # Combine keywords and named entities for a more comprehensive set of important words\n",
    "    important_words = set(important_keywords + important_entities)\n",
    "    \n",
    "    sentences = sent_tokenize(input_text)\n",
    "    num_sentences = len(sentences)\n",
    "    \n",
    "    tfidf_vectorizer = TfidfVectorizer(stop_words='english')\n",
    "    tfidf_matrix = tfidf_vectorizer.fit_transform(sentences)\n",
    "    \n",
    "    cosine_similarities = linear_kernel(tfidf_matrix[0:1], tfidf_matrix).flatten()\n",
    "    related_sentences_indices = cosine_similarities.argsort()[:-num_sentences:-1]\n",
    "\n",
    "    important_sentences = [sentences[i] for i in related_sentences_indices if any(kw.lower() in sentences[i].lower() for kw in important_words)]\n",
    "    \n",
    "    # Context weightage for adjusting the number of sentences based on context_type\n",
    "    context_weightage = {\n",
    "        'general': 1,\n",
    "        'story': 1.5,\n",
    "        'report': 1.2,\n",
    "        'article': 1.3  # Adding support for 'article' context\n",
    "    }\n",
    "    \n",
    "    num_output_sentences = max(2, int(detail_level * len(important_sentences) / 5 * context_weightage.get(context_type, 1)))\n",
    "    \n",
    "    # Sort the sentences based on their original order to maintain chronological sequence\n",
    "    selected_sentences = sorted(important_sentences[:num_output_sentences], key=lambda x: sentences.index(x))\n",
    "    \n",
    "    return ' '.join(selected_sentences)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords are-: ['spiralia', 'emily', 'town', 'moment', 'lost']\n",
      "Entities are-: ['tomorrow', 'four', 'south', 'yesterday', 'today', 'spiralia', 'west', 'east', 'north']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"In the town of Spiralia, where time twisted and turned upon itself, Emily, a young woman with raven-black hair, found herself standing at the crossroads. But when she reached the town, she found herself back in Spiralia, with no memory of the visions she'd seen. By the time Emily reached the town, she realized it wasn't a town at all. It was Spiralia, but a Spiralia of impossibilities and things that had never happened. Tired and disoriented, Emily sat down at the crossroads. Emily blinked, and suddenly she was back at her home, outside Spiralia.\""
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dynamic_detailed_summarize_premium(\"\"\"In the town of Spiralia, where time twisted and turned upon itself, Emily, a young woman with raven-black hair, found herself standing at the crossroads. The signposts pointed in four directions: North to \"Yesterday\", South to \"Tomorrow\", East to \"Now\", and West to \"Never\".\n",
    "\n",
    "She remembered coming to Spiralia for a reason, but the more she thought about it, the further the memory seemed to slip away. Emily decided to head North towards \"Yesterday\" in hopes of recovering her lost memories. But every step she took led her closer to a town that looked exactly like Spiralia, yet everyone she met claimed it was a place called \"Today\".\n",
    "\n",
    "Confused, Emily retraced her steps and decided to head South towards \"Tomorrow\". The path was lined with trees bearing fruits that showed visions of possible futures. Emily saw herself in various roles - a baker, a sailor, a librarian, and even Spiralia's mayor. But when she reached the town, she found herself back in Spiralia, with no memory of the visions she'd seen.\n",
    "\n",
    "Growing frustrated, she then ventured East towards \"Now\". As she walked, the world around her began to loop. She'd walk past a blue house, then a red barn, only to see the same blue house and red barn again moments later. It felt as if she was stuck in an endless loop of the present moment.\n",
    "\n",
    "Desperate to escape, she dashed West towards \"Never\". This path was the strangest of all. Unborn dreams floated in the air, and landscapes changed with every blink. There were mountains made of whispers and rivers that flowed backward. By the time Emily reached the town, she realized it wasn't a town at all. It was Spiralia, but a Spiralia of impossibilities and things that had never happened.\n",
    "\n",
    "Tired and disoriented, Emily sat down at the crossroads. Just then, an old woman approached her. \"Lost your way?\" she asked. Emily nodded. The woman smiled, \"In Spiralia, all roads lead back to where you started, for it's not about the journey but the realization that every moment is an intersection of past, present, future, and the unimagined.\"\n",
    "\n",
    "Emily blinked, and suddenly she was back at her home, outside Spiralia. She had her memories back, but they were intertwined with the confusing experiences of Spiralia. Had it all been a dream? Or had she truly visited a place where time lost its meaning? She might never know, but Emily felt a newfound appreciation for the present moment.\n",
    "\"\"\",1,'story')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords are-: ['meta', 'glasses', 'ray', 'ban', 'case']\n",
      "Entities are-: ['wayfarer', 'eight', 'bluetooth', 'meta', 'first', 'roughly 500', '299', 'instagram or facebook', '44', 'wayfarers', '379', '30 seconds', 'the ray-ban meta', '12', '150', 'jfk', 'alexa', 'ray-ban stories', 'ray-ban', 'wifi', 'u.s.', 'four hours', 'meta ai', 'headliner', 'less than four hours', 'facebook or instagram', 'all day', '36 hours', '329', 'meta ray-ban', '50.8', '32', 'two', 'gb', '100']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Somewhere between the Ray-Ban Meta and Meta Quest 3 sits an ideal mixed-reality headset. The Meta Quest Ray-Ban is a fantasy at the moment — albeit one that points in the direction of where its makers think this is all headed. Presently, the Ray-Ban Meta and Meta Quest 3 are very different devices, with little in the way of overlap, beyond being head-worn products with built-in sensors. The Meta Quest 3 is a mixed-reality headset designed to be worn exclusively indoors. You’ll need to download the Meta View app, choose between Meta Ray-Ban and Ray-Ban Stories and allow bluetooth to connect .'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dynamic_detailed_summarize_premium(\"\"\"Somewhere between the Ray-Ban Meta and Meta Quest 3 sits an ideal mixed-reality headset. It’s slim, light, offers hand tracking and passthrough and livestreams video when the moment calls for it. It’s designed to be worn inconspicuously outdoors, until the time comes for content capture.\n",
    "\n",
    "The Meta Quest Ray-Ban is a fantasy at the moment — albeit one that points in the direction of where its makers think this is all headed. Presently, the Ray-Ban Meta and Meta Quest 3 are very different devices, with little in the way of overlap, beyond being head-worn products with built-in sensors.\n",
    "\n",
    "The Meta Quest 3 is a mixed-reality headset designed to be worn exclusively indoors. It’s light, perhaps, compared to other headsets of its ilk, but wearing the thing while walking around outside frankly sounds a bit miserable. That’s precisely the use case the Ray-Ban Meta was designed for: freedom of movement outside the house that’s designed to go (mostly) unnoticed.\n",
    "\n",
    "Just prior to writing this, I slipped a pair on, before the JFK airport mobility cart drove my sciatica-ridden ass to the gate. I would say the pair was inconspicuous but for the fact that I was wearing a pair of sunglasses indoors. Well, that and the extremely necessary recording lighting that flashes on so you can’t creep shoot folks without their knowledge. We got our first glimpse of the Ray-Ban Meta at a briefing just ahead of the recent Connect conference. I was genuinely impressed by the industrial design the join team came up with. Most folks would hard-pressed to distinguish the charger from a standard Ray-Ban classic eyeglass case. It’s a little thicker than some, sure. A bit heavier. More rigid. But the team was able to make surprisingly few concessions.\n",
    "\n",
    "There are a lot of clever touches here. In the place of a snap is a ring. Open the case and it glows green when fully charged and orange when not. The orange starts blinking when the battery is low. Space has been maximized inside. The battery sits directly beneath the glasses’ folded temples. In front of this is a dock with two charging pins that lie flush with a pair of contact pads hidden on the underside of the glasses’ bridge, held in place with magnets and a small tab.\n",
    "\n",
    "The USB-C port is located on the outside bottom of the case, allowing it to sit on its back while being charged. Directly above this on the case’s rear is the Bluetooth pairing button. The case is slimmer than the last gen and can be carried in a pocket comfortably.\n",
    "\n",
    "Meta says the glasses get “up to” four hours on a charge, while the case gets a total of eight charging cycles, for a grand total of 36 hours. As the company notes, “Battery life varies by use, configuration, settings and many other factors.” That’s the case with all tech, of course, but I did notice that video is a power drainer.The companies really leaned into the style side of things here (not a bad decision when designing tech meant to be worn on the body). There are two main designs for the glasses. There’s the classic Wayfarer (which is probably what you think of when you think of sunglasses) and the new Headliner (not dissimilar from Wayfarer, but significantly more rounded on the top and bottom).\n",
    "\n",
    "According to Meta, there are 150 design combos possible, when you factor in all of the different design options, including frame color, style and lenses (including sunglasses, clear, prescription, transitions and polarized).\n",
    "\n",
    "The temples are thicker than most sunglasses — to be expected, seeing as how they contain the speakers and other components (there’s a transparent option, if you want to see for yourself) — but again, the designers have done a good job keeping size down, all things considered. And again, while slightly heavier that a standard pair of Wayfarers (50.8 g vs. 44 g), you can wear them comfortably all day if you want to (or at least the less than four hours the battery lasts). There’s a touchpad on the outside of the left temple. Swiping back and forth will adjust the volume (other features can be customized in-app). It also doubles as a control panel for live streaming, since you likely don’t want to futz with your phone or keep using the wake word. A tap can check Instagram or Facebook comments and viewers in real-time. The capture button sits next to the hinge on the left temple\n",
    "\n",
    "There are a pair of small circular modules on the end pieces. They look identical, for the sake of symmetry, but serve very different — albeit related — functions. On the top right (when facing the glasses) is the 12-megapixel camera. On the top left is an LED that turns on to alert people in your vicinity that you’re recording.\n",
    "\n",
    "When covered, the glasses send an audio alert that they’ve stopped recording. This is to avoid people sticking a piece of electrical tape to hide the light. Meta says they didn’t hear of any specific examples of this happening, but they almost certainly got that feedback. Again, privacy is paramount for a device like this, especially since it’s something that most people around you don’t know exists. When the battery is low, you’ll get a spoken alert and the light will blink orange and turn red right before shutting down. The light will blink white when receiving a call, do a single flash when taking a photo and glow steadily when recording.\n",
    "\n",
    "When pairing, it flashes blue, going solid when connected. The pairing process is pretty straight forward. You’ll need to download the Meta View app, choose between Meta Ray-Ban and Ray-Ban Stories and allow bluetooth to connect . Images and video will save to the glasses’ 32GB of internal storage (that’s roughly 500 photos or 100 videos at the maximum 30 seconds apiece). You’ll need to tap “Import” inside the app to connect via WiFi and download the contents to your phone. You can also set it up to auto import via settings.\n",
    "\n",
    "Once everything is paired, put the glasses on and open either Facebook or Instagram to livestream. Tap the plus icon and it will bring you to the livestream screen. Your phone’s camera is, understandably, the default, but double pressing the capture button will switch over to the glasses. Livestreaming is probably the single biggest killer app Ray-Ban Stories was missing.\n",
    "\n",
    "There are barely visible down firing speakers on the bottom of the temple tips. When I first tried the speakers in an otherwise silent room, they sounded surprisingly loud and clear. They’re open-ear speakers, rather than bone conduction, which has its pluses and minuses. Bone conduction tends to be quite quiet but does a decent job with ambient noise, since it’s arriving at your eardrums through a different method. As expected, I had to turn up the volume quite a bit among the airport din. I would recommend them for quieter environments, where possible, but obviously that isn’t always an option. Sound is integral to the headphones, beyond music listening. For instance, there’s an audible shutter click when you take a picture.\n",
    "\n",
    "There are on-board microphones as well, which listen for the “hey Meta” wake word. Voice certainly makes sense on a device like this. It can be used to take a picture, stop and start video and adjust volume (turns out voice is kind of an annoying way to do the latter). You can also ask the glasses for the time, weather and how much battery is left. You can also ask Alexa style-questions, and Meta AI will attempt to answer. That’s currently only available here in the U.S. through an open beta.\n",
    "\n",
    "The price starts at $299 for standard lenses. Polarized run $329 and transitions $379. Prescription lenses are on a sliding scale. The price will almost certainly be a deterrent for many — and understandably so. Ultimately, you need to ask yourself how much value a face-worn camera will bring to your life. If you make a living livestreaming, it may make sense. It’s a lot to pay however, for sheer novelty.\n",
    "\n",
    "It’s worth noting that future updates will bring more value to the device, including sign translation (through voice) and the ability to identify landmarks in front of you. One can see the future of head-worn computing laid out in front of your face — though it’s still going to be a while before we get there.\"\"\",0.5,'article')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords are-: ['netflix', 'quarter', 'subscribers', 'plan', 'ads']\n",
      "Entities are-: ['may', 'the quarter', 'first', 'the third quarter', 'france', '3.73', 'three', 'netflix', '22.99', 'the first full quarter', 'a couple of months ago', 'u.s.', '8.76 million', '$8.5 billion', '393.79', 'q2', 'dozens', '13.75%', 'almost 70%', 'the u.k. people', 'third quarter', 'nearly 9 million', 'this quarter', '247.15 million', 'u.k.', 'nearly a third', 'quarter-over-quarter', '2020', 'covid']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Netflix reported its third quarter earnings and things are going great right now for the streaming giant. While reports suggest that many customers are facing subscription fatigue and are thinking about canceling some streaming subscriptions, it seems like Netflix still has a lot of room for growth — especially with advertising revenue. When it comes to revenue, Netflix reported $8.5 billion this quarter with earnings of $3.73 per share. But subscribers aren’t going to be happy as it means that the company is going to raise its prices for some of its plans once again in three key markets.'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dynamic_detailed_summarize_premium(\"\"\"Netflix reported its third quarter earnings and things are going great right now for the streaming giant. The company added nearly 9 million subscribers globally, which means that revenue is up.\n",
    "\n",
    "And yet, it also means that Netflix is using this opportunity to raise the prices of some of its plans in the U.S., the U.K. and France to differentiate ad-free plans from its entry-level ad supported plan. For instance, the most expensive plan will now cost $22.99 per month for new subscribers.\n",
    "\n",
    "But let’s rewind first and look at Netflix’s current situation. In May, Netflix began its crackdown on password sharing in its home market and in dozens of global markets. In other words, we’re now seeing a clear picture of the effect of password sharing as the third quarter is the first full quarter under the new rules.\n",
    "\n",
    "The company also streamlined its offering a couple of months ago by removing the basic tier in the U.S. and the U.K. People now have to pay quite a bit of money to remove ads from Netflix.\n",
    "\n",
    "While reports suggest that many customers are facing subscription fatigue and are thinking about canceling some streaming subscriptions, it seems like Netflix still has a lot of room for growth — especially with advertising revenue.\n",
    "\n",
    "The company now has 247.15 million subscribers. And that number jumped by 8.76 million subscribers this quarter alone. Netflix hasn’t seen that kind of growth in subscribers since Q2 of 2020, also known as the quarter of strict Covid lockdowns around the world.\n",
    "\n",
    "When it comes to revenue, Netflix reported $8.5 billion this quarter with earnings of $3.73 per share. Ads are increasingly contributing to the bottom line as the number of subscribers on the ads plan is up almost 70% quarter-over-quarter. Nearly a third of new subscribers start with an ads plan.\n",
    "\n",
    "\n",
    "This is good news for shareholders, which is why Netflix shares are currently up 13.75% in pre-market trading ($393.79 per share). But subscribers aren’t going to be happy as it means that the company is going to raise its prices for some of its plans once again in three key markets. \n",
    "\"\"\",1,'article')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "def dynamic_detailed_summarize_premium_variability(input_text, detail_level=1, context_type='general', important_keywords=None, important_entities=None, randomize=False):\n",
    "    if important_keywords is None:\n",
    "        num_keywords = random.randint(4, 7) if randomize else 5  # Randomizing keyword extraction\n",
    "        important_keywords = extract_important_keywords(input_text, num_keywords)\n",
    "    \n",
    "    if important_entities is None:\n",
    "        important_entities = extract_named_entities(input_text)\n",
    "    \n",
    "    print(f'Keywords are-: {important_keywords}')\n",
    "    print(f'Entities are-: {important_entities}')    \n",
    "    important_words = set(important_keywords + important_entities)\n",
    "    \n",
    "    sentences = sent_tokenize(input_text)\n",
    "    \n",
    "    tfidf_vectorizer = TfidfVectorizer(stop_words='english')\n",
    "    tfidf_matrix = tfidf_vectorizer.fit_transform(sentences)\n",
    "    cosine_similarities = linear_kernel(tfidf_matrix[0:1], tfidf_matrix).flatten()\n",
    "    related_sentences_indices = cosine_similarities.argsort()[:-len(sentences):-1]\n",
    "    important_sentences = [sentences[i] for i in related_sentences_indices if any(kw.lower() in sentences[i].lower() for kw in important_words)]\n",
    "    \n",
    "    context_weightage = {\n",
    "        'general': 1,\n",
    "        'story': 1.5,\n",
    "        'report': 1.2,\n",
    "        'article': 1.3\n",
    "    }\n",
    "    \n",
    "    context_factor = context_weightage.get(context_type, 1)\n",
    "    if randomize:\n",
    "        context_factor += random.uniform(-0.1, 0.1)  # Randomizing context weightage\n",
    "    \n",
    "    num_output_sentences = max(2, int(detail_level * len(important_sentences) / 5 * context_factor))\n",
    "    \n",
    "    selected_sentences = sorted(important_sentences[:num_output_sentences], key=lambda x: sentences.index(x))\n",
    "    \n",
    "    if randomize:\n",
    "        random.shuffle(selected_sentences)  # Shuffling the selected sentences\n",
    "    \n",
    "    return ' '.join(selected_sentences)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords are-: ['ai', 'pokémon', 'whidden', 'game']\n",
      "Entities are-: ['cerulean city', 'peter whidden', 'kanto', 'more than 50,000 hours', '25', 'first', 'almost 10 years ago', 'the pokémon center', 'youtube', 'github', 'only 12', 'seattle', '33-minute', 'pallet town', 'game boy', 'one', 'alphago', 'a ton', 'pokémon', '2.2 million', 'pidgey', 'deepmind', 'twitch plays pokémon', 'gyarados', 'ai', 'million', 'magikarp', 'nine days', 'techcrunch', 'whidden', 'pokémon crystal', 'pavlovian', 'pokémon center', 'mt. moon', 'aaaaaaaaaa', 'the last few years', 'second', 'pokémon red']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Almost 10 years ago, the online phenomenon “Twitch Plays Pokémon” convened over a million people to play Pokémon Red at the same time, with each player’s keystrokes registering as commands for the one pixelated avatar. For the last few years, Seattle-based software engineer Peter Whidden has been training a reinforcement learning algorithm to navigate the classic first game of the Pokémon series — in that time, the AI has played more than 50,000 hours of the game. When you reach the second town in Pokémon Red, you’re given an item to bring back to the Pokémon Professor in Pallet Town.'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dynamic_detailed_summarize_premium_variability(\"\"\"Almost 10 years ago, the online phenomenon “Twitch Plays Pokémon” convened over a million people to play Pokémon Red at the same time, with each player’s keystrokes registering as commands for the one pixelated avatar. Now, like a Magikarp growing into a Gyarados, the evolution of technology begs a new question: can AI play Pokémon?\n",
    "\n",
    "For the last few years, Seattle-based software engineer Peter Whidden has been training a reinforcement learning algorithm to navigate the classic first game of the Pokémon series — in that time, the AI has played more than 50,000 hours of the game. Whidden posted a 33-minute YouTube video telling the story of the AI’s development, and after nine days, the video has amassed 2.2 million views.“What’s been super fun to see is how many people are engaging with it,” Whidden told TechCrunch. He uploaded the code he used to GitHub, along with instructions on how to operate and train the AI. “There’s a ton of people that seem really interested in actually doing this process of creating or designing.” One fan was able to apply his code to Pokémon Crystal, another retro Game Boy installment.\n",
    "\n",
    "The AI’s reinforcement model is Pavlovian, giving the AI point-based incentives to level up Pokémon, explore new areas, win battles and beat gym leaders. Sometimes, these incentives don’t perfectly align with progression in the game, yet the failures of the AI are weirdly charming, which is probably why Whidden’s video has gone viral.\n",
    "\n",
    "In one of the AI’s attempts, it simply stops to stare at the water in Pallet Town — the first place you visit in the game — and never moves. It gets stuck in an area with animated water, grass and NPCs who pace back and forth, meaning that every individual frame seems like a novel experience to the AI, even though it’s just sitting motionless without even getting its first Pokémon yet. But this AI isn’t in a rush to “catch ’em all.” It’s just enjoying the beauty of the Kanto region (or maybe it’s taking an ethical stance against forcing these cute little animals to battle each other… who can say).\n",
    "\n",
    "“So, according to our own objective, just hanging out and admiring the scenery is more rewarding than exploring the rest of the world,” Whidden explains in the video. “This is a paradox that we encounter in real life: curiosity leads us to our most important discoveries, but at the same time, it makes us vulnerable to distractions and gets us into trouble.”\n",
    "\n",
    "The AI somehow continues to pull on our heartstrings: Later, it experiences something akin to a traumatic event at the Pokémon Center. The AI’s success is measured in part by the total levels of all Pokémon in your party. But when an AI goes to the Pokémon Center and button smashes enough to deposit a Pokémon into storage, the sum of all levels reduces drastically, sending a strong negative signal to the AI. With both Pidgey and an unidentified creature nicknamed “AAAAAAAAAA” in its party, the sum of all levels was 25, but once Pidgey is deposited into the PC, the sum is only 12.\n",
    "\n",
    "“It doesn’t have emotions like a human does, but a single event with an extreme reward value can still leave a lasting impact on its behavior,” Whidden narrates. “In this case, losing its Pokémon only one time is enough to form a negative association with the whole Pokémon Center, and the AI will avoid it entirely in all future games.”\n",
    "\n",
    "Despite the AI’s ability to experience trauma and admire the pretty pixels of Pallet Town, it’s still just a computer. This AI isn’t able to read and interpret dialogue in the game, so in early iterations, the program would get stuck at an early crossroads in the game. When you reach the second town in Pokémon Red, you’re given an item to bring back to the Pokémon Professor in Pallet Town. But the AI was having a hard time backtracking to deliver the parcel, making it impossible to progress further. So, Whidden skipped ahead to make each game begin after delivering the package, and with Squirtle as the AI’s starter Pokémon, since the early game is generally easier with a water Pokémon at your service.\n",
    "\n",
    "“In the video, the farthest that [the AI] reaches is Mt. Moon, between the first and second gym,” Whidden told TechCrunch. Caves are notoriously frustrating to navigate in early Pokémon games, even if you have an actual human brain. But Whidden recently tweaked some of the rewards in his code and tried a different learning algorithm, and finally, the AI managed to exit the cave and arrive in Cerulean City.\n",
    "\n",
    "Other researchers have used reinforcement learning to study the use of AI in gaming, like with DeepMind’s AlphaGo, which was the first computer program to defeat a professional Go player. But Whidden’s video has garnered so much attention because he’s so adept at explaining unfamiliar concepts through a familiar medium: Pokémon.\n",
    "\n",
    "\"\"\",0.5,'article',randomize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords are-: ['ai', 'pokémon', 'whidden', 'game', 'video']\n",
      "Entities are-: ['cerulean city', 'peter whidden', 'kanto', 'more than 50,000 hours', '25', 'first', 'almost 10 years ago', 'the pokémon center', 'youtube', 'github', 'only 12', 'seattle', '33-minute', 'pallet town', 'game boy', 'one', 'alphago', 'a ton', 'pokémon', '2.2 million', 'pidgey', 'deepmind', 'twitch plays pokémon', 'gyarados', 'ai', 'million', 'magikarp', 'nine days', 'techcrunch', 'whidden', 'pokémon crystal', 'pavlovian', 'pokémon center', 'mt. moon', 'aaaaaaaaaa', 'the last few years', 'second', 'pokémon red']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Almost 10 years ago, the online phenomenon “Twitch Plays Pokémon” convened over a million people to play Pokémon Red at the same time, with each player’s keystrokes registering as commands for the one pixelated avatar. Now, like a Magikarp growing into a Gyarados, the evolution of technology begs a new question: can AI play Pokémon? For the last few years, Seattle-based software engineer Peter Whidden has been training a reinforcement learning algorithm to navigate the classic first game of the Pokémon series — in that time, the AI has played more than 50,000 hours of the game. Whidden posted a 33-minute YouTube video telling the story of the AI’s development, and after nine days, the video has amassed 2.2 million views.“What’s been super fun to see is how many people are engaging with it,” Whidden told TechCrunch. “There’s a ton of people that seem really interested in actually doing this process of creating or designing.” One fan was able to apply his code to Pokémon Crystal, another retro Game Boy installment. “In this case, losing its Pokémon only one time is enough to form a negative association with the whole Pokémon Center, and the AI will avoid it entirely in all future games.”\\n\\nDespite the AI’s ability to experience trauma and admire the pretty pixels of Pallet Town, it’s still just a computer. When you reach the second town in Pokémon Red, you’re given an item to bring back to the Pokémon Professor in Pallet Town.'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dynamic_detailed_summarize_premium(\"\"\"Almost 10 years ago, the online phenomenon “Twitch Plays Pokémon” convened over a million people to play Pokémon Red at the same time, with each player’s keystrokes registering as commands for the one pixelated avatar. Now, like a Magikarp growing into a Gyarados, the evolution of technology begs a new question: can AI play Pokémon?\n",
    "\n",
    "For the last few years, Seattle-based software engineer Peter Whidden has been training a reinforcement learning algorithm to navigate the classic first game of the Pokémon series — in that time, the AI has played more than 50,000 hours of the game. Whidden posted a 33-minute YouTube video telling the story of the AI’s development, and after nine days, the video has amassed 2.2 million views.“What’s been super fun to see is how many people are engaging with it,” Whidden told TechCrunch. He uploaded the code he used to GitHub, along with instructions on how to operate and train the AI. “There’s a ton of people that seem really interested in actually doing this process of creating or designing.” One fan was able to apply his code to Pokémon Crystal, another retro Game Boy installment.\n",
    "\n",
    "The AI’s reinforcement model is Pavlovian, giving the AI point-based incentives to level up Pokémon, explore new areas, win battles and beat gym leaders. Sometimes, these incentives don’t perfectly align with progression in the game, yet the failures of the AI are weirdly charming, which is probably why Whidden’s video has gone viral.\n",
    "\n",
    "In one of the AI’s attempts, it simply stops to stare at the water in Pallet Town — the first place you visit in the game — and never moves. It gets stuck in an area with animated water, grass and NPCs who pace back and forth, meaning that every individual frame seems like a novel experience to the AI, even though it’s just sitting motionless without even getting its first Pokémon yet. But this AI isn’t in a rush to “catch ’em all.” It’s just enjoying the beauty of the Kanto region (or maybe it’s taking an ethical stance against forcing these cute little animals to battle each other… who can say).\n",
    "\n",
    "“So, according to our own objective, just hanging out and admiring the scenery is more rewarding than exploring the rest of the world,” Whidden explains in the video. “This is a paradox that we encounter in real life: curiosity leads us to our most important discoveries, but at the same time, it makes us vulnerable to distractions and gets us into trouble.”\n",
    "\n",
    "The AI somehow continues to pull on our heartstrings: Later, it experiences something akin to a traumatic event at the Pokémon Center. The AI’s success is measured in part by the total levels of all Pokémon in your party. But when an AI goes to the Pokémon Center and button smashes enough to deposit a Pokémon into storage, the sum of all levels reduces drastically, sending a strong negative signal to the AI. With both Pidgey and an unidentified creature nicknamed “AAAAAAAAAA” in its party, the sum of all levels was 25, but once Pidgey is deposited into the PC, the sum is only 12.\n",
    "\n",
    "“It doesn’t have emotions like a human does, but a single event with an extreme reward value can still leave a lasting impact on its behavior,” Whidden narrates. “In this case, losing its Pokémon only one time is enough to form a negative association with the whole Pokémon Center, and the AI will avoid it entirely in all future games.”\n",
    "\n",
    "Despite the AI’s ability to experience trauma and admire the pretty pixels of Pallet Town, it’s still just a computer. This AI isn’t able to read and interpret dialogue in the game, so in early iterations, the program would get stuck at an early crossroads in the game. When you reach the second town in Pokémon Red, you’re given an item to bring back to the Pokémon Professor in Pallet Town. But the AI was having a hard time backtracking to deliver the parcel, making it impossible to progress further. So, Whidden skipped ahead to make each game begin after delivering the package, and with Squirtle as the AI’s starter Pokémon, since the early game is generally easier with a water Pokémon at your service.\n",
    "\n",
    "“In the video, the farthest that [the AI] reaches is Mt. Moon, between the first and second gym,” Whidden told TechCrunch. Caves are notoriously frustrating to navigate in early Pokémon games, even if you have an actual human brain. But Whidden recently tweaked some of the rewards in his code and tried a different learning algorithm, and finally, the AI managed to exit the cave and arrive in Cerulean City.\n",
    "\n",
    "Other researchers have used reinforcement learning to study the use of AI in gaming, like with DeepMind’s AlphaGo, which was the first computer program to defeat a professional Go player. But Whidden’s video has garnered so much attention because he’s so adept at explaining unfamiliar concepts through a familiar medium: Pokémon.\n",
    "\n",
    "\"\"\",1,'article')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords are-: ['tree', 'guardian', 'old', 'town', 'mara']\n",
      "Entities are-: ['this year', 'one', 'larksville', 'autumn', 'mara', 'winter', 'the next day', 'old guardian', \"old guardian's\", 'summer', 'weeks', 'recent years', 'only one']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"In the heart of a quaint town named Larksville, an ancient oak tree stood tall. It spoke of an ancient pact it had made with the founders of Larksville. The tree would stand guard over the town, protecting it from harm. She gathered the town's children and narrated Old Guardian's story. The following spring, Old Guardian sprouted more leaves than ever before, standing tall and proud, with the entire town of Larksville by its side.\""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dynamic_detailed_summarize_premium(\"\"\"In the heart of a quaint town named Larksville, an ancient oak tree stood tall. It had seen many seasons come and go, with children playing around its trunk in summer and couples seeking shelter beneath its expansive branches during rain. However, this year was different.\n",
    "\n",
    "As autumn approached, the tree, which locals affectionately called \"Old Guardian\", began to shed its leaves earlier than usual. The townspeople were puzzled. Was the tree finally succumbing to age?\n",
    "\n",
    "Mara, a young girl with fiery red hair and a penchant for adventure, decided to uncover the mystery. Every day after school, she would sit by Old Guardian, reading books and observing its changes. As days turned into weeks, only one leaf remained on the tree, clinging desperately to a fragile branch.\n",
    "\n",
    "One chilly evening, as Mara was about to head home, she heard a soft whisper. To her astonishment, it was coming from the tree. Old Guardian had a story to tell.\n",
    "\n",
    "It spoke of an ancient pact it had made with the founders of Larksville. The tree would stand guard over the town, protecting it from harm. In return, it asked for respect and care from its inhabitants. However, in recent years, as the town modernized, many had forgotten the legend, leading Old Guardian to feel abandoned and unloved.\n",
    "\n",
    "Mara, moved by the tree's tale, decided to act. She gathered the town's children and narrated Old Guardian's story. The next day, the townspeople came together to celebrate the tree, singing songs and sharing tales of its legacy.\n",
    "\n",
    "As winter approached, the last leaf fell, but it wasn't a sign of the tree's end. Instead, it marked a new beginning. The following spring, Old Guardian sprouted more leaves than ever before, standing tall and proud, with the entire town of Larksville by its side.\n",
    "\n",
    "\"\"\",1,'story')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords are-: ['points', 'community', 'reddit', 'program', 'users']\n",
      "Entities are-: ['1', 'vaults', 'over 5,000', 'first', 'reddit', '30-day', 'tim rathschmidt', 'rathschmidt', 'gifs', 'later that month', '2022', 'vault', 'at least 10', 'early november', 'earlier this year', 'one', 'monthly', 'mid-september', 'faq', 'api', 'techcrunch', 'gold', 'the years', 'this year', 'the contributor program', 'community points', 'ethereum', 'over 8,000', '48-hour', 'arbitrum nova', '2020', 'two']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Reddit is winding down Community Points — the blockchain-based “internet points” program designed to reward creators and developers — in favor of prioritizing rewards programs that are less difficult to scale. In the years since launching Community Points, Reddit has rolled out a number of community incentives, like the moderator rewards program and the Contributor Program, which awards actual money by allowing eligible users to convert their Reddit gold and karma into cash. Rathschmidt noted that the company isn’t phasing out Community Points to make way for the Contributor Program, but rather to prioritize on programs that are “more set up to scale and benefit more users,” like the Contributor Program.'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dynamic_detailed_summarize_premium(\"\"\"Reddit is winding down Community Points — the blockchain-based “internet points” program designed to reward creators and developers — in favor of prioritizing rewards programs that are less difficult to scale.\n",
    "\n",
    "“Though we saw some future opportunities for Community Points, the resourcing needed was unfortunately too high to justify,” Reddit’s director of consumer and product communications Tim Rathschmidt told TechCrunch. “The regulatory environment has since added to that effort. Though the moderators and communities that supported Community Points have been incredible partners — as it’s evolved, the product is no longer set up to scale.”\n",
    "\n",
    "Community Points, which will be phased out by early November, were promoted as a chance for Redditors to “own” a piece of their community. First launched in 2020, Community Points were awarded to users who positively engaged in select subreddits in order to incentivize better content and conversation. The points were essentially interchangeable Ethereum tokens stored in Reddit’s Vault, which operated as a cryptocurrency wallet.\n",
    "\n",
    "Once awarded, neither Reddit nor moderators could take back another user’s Community Points. The points could be used on special features, like memberships that unlocked unique badges and animated emojis. Once spent, the points were “burned.”\n",
    "\n",
    "As a measure of “reputation,” Community Points were displayed next to usernames in participating subreddits to mark the community’s biggest contributors. Since the points were on the blockchain, the program aimed to allow users to display their “reputation” anywhere online, and could be embedded in other sites or apps. If users were banned from Reddit, their points would still exist on the blockchain, but they would lose access to their Vault — rendering the points useless.\n",
    "\n",
    "But scaling Community Points proved to be an immense challenge for Reddit. The pilot program originally used Ethereum, which had high transaction fees and limited bandwidth.\n",
    "\n",
    "“Putting all Reddit users on the main Ethereum network, for example, would be infeasible and prohibitively expensive,” the Community Points page said. “Therefore, we have researched heavily into solutions that are decentralized, secure and yet highly scalable.”\n",
    "\n",
    "Reddit moved Community Points to Arbitrum Nova in 2022, which is built “on top” of Ethereum.\n",
    "\n",
    "“This enables it to apply scaling technologies that enable ultra-low-cost transactions, very high energy efficiency and strong security guarantees,” the FAQ stated.\n",
    "\n",
    "\n",
    "\n",
    "But even that was impractical for Reddit. In the years since launching Community Points, Reddit has rolled out a number of community incentives, like the moderator rewards program and the Contributor Program, which awards actual money by allowing eligible users to convert their Reddit gold and karma into cash. Under the Contributor Program, Redditors who earn at least 10 gold within a 30-day period are eligible for a monthly withdrawal. Reddit users who have over 5,000 karma (total upvotes) can earn $1 per gold.\n",
    "\n",
    "“We’re still working on ways to improve community governance and empower communities and contributions,” Rathschmidt said. “Part of why we’re moving past this product is that we’ve already launched, or are actively investing in, several products that accomplish what the Community Points program was trying to accomplish, while being easier to adopt and understand.”\n",
    "\n",
    "He added that the special features that Community Points can buy, like gifs, should be available to any community. Once the program is fully phased out, beta participants will no longer see their points in their Vaults, and will no longer earn points in their communities. Although Rathschmidt said he couldn’t give an exact count of the number of subreddits affected by the cut, he admitted that “you’d be able to count them on one hand and have a finger or two left over.”\n",
    "\n",
    "It’s not the only rewards program that Reddit has killed this year. Earlier this year, Reddit announced the end of its coin system, which allowed users to purchase Gold and other currency to award other users. The announcement came in the aftermath of the sitewide protests against Reddit’s API changes, which culminated in a 48-hour blackout of over 8,000 subreddits, and was wildly unpopular among Reddit’s already jaded users. Awards and existing coins under the previous reward program were available until mid-September. Reddit announced the Contributor Program later that month.\n",
    "\n",
    "Rathschmidt noted that the company isn’t phasing out Community Points to make way for the Contributor Program, but rather to prioritize on programs that are “more set up to scale and benefit more users,” like the Contributor Program.\n",
    "\n",
    "“It’s one example. Now, we’re able to scale several products that accomplish what the Community Points program was trying to accomplish — like subreddit karma and gifs,” he continued. “Many of the benefits of Community Points have already been built into the platform.”\"\"\",0.5,'article')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords are-: ['birbal', 'emperor', 'akbar', 'sad', 'man']\n",
      "Entities are-: ['birbal', 'one sunny morning', 'akbar', 'mughal empire', 'days', 'that day']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"Once upon a time, in the magnificent Mughal Empire, there lived a wise and witty minister named Birbal. He summoned Birbal to his court and presented him with a challenge. Emperor Akbar was impressed by Birbal's ingenious solution. The Emperor praised his minister for his cleverness and wit. Their friendship and the tales of their clever exchanges became legendary throughout the empire, and Birbal's reputation as a wise minister continued to flourish.\""
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dynamic_detailed_summarize_premium(\"\"\"Once upon a time, in the magnificent Mughal Empire, there lived a wise and witty minister named Birbal. Birbal was not just known for his intelligence, but also for his ability to solve the most complex problems with simplicity and humor. His sharp wit and clever solutions made him a favorite of Emperor Akbar.\n",
    "\n",
    "One sunny morning, Emperor Akbar decided to put Birbal's wit to the test. He summoned Birbal to his court and presented him with a challenge. The Emperor said, \"Birbal, I want you to bring me something that will make a happy man sad and a sad man happy. If you cannot do this, you shall lose your position as my minister.\"\n",
    "\n",
    "Birbal, known for his calm and composed demeanor, accepted the challenge without hesitation. He bowed to the Emperor and left the court, pondering the task at hand.\n",
    "\n",
    "After days of contemplation, Birbal finally had a brilliant idea. He requested a small, elaborately decorated box to be made, which he then presented to Emperor Akbar.\n",
    "\n",
    "The Emperor, intrigued by the beautifully crafted box, asked, \"Birbal, what is inside this box that can achieve what I asked for?\"\n",
    "\n",
    "Birbal replied, \"Your Majesty, inside this box is a simple mirror. When a happy man looks into it, he will see his own reflection and realize that his happiness is fleeting and can change. This thought may make him a bit sad. Conversely, when a sad man looks into the same mirror, he will see his reflection as well and realize that his sadness is temporary. This realization might bring a glimmer of happiness to his heart.\"\n",
    "\n",
    "Emperor Akbar was impressed by Birbal's ingenious solution. He couldn't help but smile as he understood the depth of Birbal's wisdom. The Emperor praised his minister for his cleverness and wit.\n",
    "\n",
    "From that day on, Emperor Akbar held Birbal's counsel in even higher regard, knowing that his wisdom was not only profound but also sprinkled with a touch of humor. Their friendship and the tales of their clever exchanges became legendary throughout the empire, and Birbal's reputation as a wise minister continued to flourish.\"\"\",1,'story')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "mic=\"\"\"In case you weren’t sure how Microsoft felt about AI, CEO Satya Nadella is here to tell you: they like it. They like it a lot. In the company’s annual report, he pens a letter to shareholders bigging up AI in every way. And it’s not hard to see why: He clearly believes this is the biggest, perhaps the only advance in computing that has truly mattered in over a decade.\n",
    "\n",
    "Though Microsoft’s business is sound, you could be forgiven for thinking they’ve been spinning their wheels somewhat. Their bids to get into mobile, search and hardware have all either stalled or fallen flat, and numerous product experiments have failed to penetrate their respective markets.\n",
    "\n",
    "On the other hand, their cloud business is very strong, and they’ve increasingly shaped the company and its products around that premise. But even that success was beginning to wear thin, for as profitable as it is, there is only so much room for innovation there.\n",
    "\n",
    "For years they must certainly have kept an eye on the trends, watching to see whether they some new development or another was worth adopting. Social web? Nah — too much work. Fitness? Simpler to be their infrastructure. Blockchain? Redundant and risky. Metaverse? Very funny.\n",
    "\n",
    "Like a surfer in a calm, Microsoft waited, placid and bobbing. Then the AI wave rose up under them — and they started paddling for dear life.\n",
    "\n",
    "This next generation of AI will reshape every software category and every business, including our own. Forty-eight years after its founding, Microsoft remains a consequential company because time and time again—from PC/Server, to Web/Internet, to Cloud/Mobile—we have adapted to technological paradigm shifts. Today, we are doing so once again, as we lead this new era.\n",
    "\n",
    "Then follows a couple dozen examples of where AI is being slotted into place across all their business units, products and long-term efforts. This isn’t a hobby for Microsoft — they have actually decided that this is the next phase of personal and business computing.\n",
    "\n",
    "And it’s not merely an enabler, either, like a silicon advance that makes data centers run twice as efficiently, or a battery that lasts twice as long. It is, so to speak, a transformer:\n",
    "\n",
    "The long arc of computing has, in many ways, been shaped by the pursuit of increasingly intuitive human-computer interfaces—keyboards, mice, touch screens. We believe we have now arrived at the next big step forward—natural language—and will quickly go beyond, to see, hear, interpret, and make sense of our intent and the world around us.\n",
    "\n",
    "You can almost see the stars in his eyes: Imagine being at the head of a major tech company like Microsoft during an upheaval of this magnitude! They’ve dabbled with the idea of moving past the mouse and keyboard before, but so far their natural language interfaces (like Cortana) and alternative hardware (like HoloLens) have not risen past the level of parlor tricks.\n",
    "\n",
    "But by either good fortune or foresight they happened to have backed the breakout leader in natural language AI: OpenAI. Not only does the technology actually look like a genuine game-changer, but the way the cookie crumbled put them in a fair way to give perennial rival Google a black eye. Google, for its part, has been caught flat-footed by the rapid shift to AI, despite having internally created the concepts that enabled it. They’re trying to bounce back, but the company has always struggled to rally successfully behind a unifying concept, and this time may prove no different.\n",
    "\n",
    "\n",
    "\n",
    "This alliance between Microsoft and OpenAI is freeing for both. OpenAI gets a combination investor and customer with effectively bottomless pockets and a sincere desire to integrate AI tools into every corner of its business. Microsoft is spared the embarrassing necessity of appearing — as it actually is — far behind the curve in AI development, because it can simply present the market-leading product as its own. Nadella makes no mention of Microsoft training its own foundation models, though they are likely doing so quietly to hedge against treachery, because their efforts pale in comparison to the forward momentum of its partnership.\n",
    "\n",
    "Imagine if the situation was reversed and it was Google who had made a fortuitous deal with OpenAI, leaving Microsoft out in the cold. Microsoft would be even worse off than Google, having to scramble to build LLMs a fraction as good, and every month they spent trying to catch up saw their competitor score another million users.\n",
    "\n",
    "So it should come as no surprise that Microsoft is expending enormous amounts of money to fortify their position and, to the extent possible, expand and deepen their partnership with OpenAI.\n",
    "\n",
    "\n",
    "\n",
    "One troubling note Nadella struck, however, was his characterization of the second of the two breakthroughs he feels define this era of AI: “the emergence of a powerful new reasoning engine.”\n",
    "\n",
    "If you are at all familiar with the way this generation of AI models works, you will know that they do not reason, any more than a calculator reasons when you ask it to multiply two numbers.\n",
    "\n",
    "Of course, Nadella is not naïve or uninformed on this matter. He knows what he is saying here — that these systems perform functions that are in many ways indistinguishable from reasoning. Asking a computer to summarize a long text document and having it do so, or even do so in iambic pentameter, feels magical — because until recently, only those with reasoning powers could do that.\n",
    "\n",
    "As it turns out, the patterns of language are predictable enough that some reasoning tasks can be reduced to statistical tasks. This is remarkable enough on its own that we need not gild the lily with magical thinking.\n",
    "\n",
    "But this language is indicative of the arguably unearned confidence AI systems have created in backers like Microsoft. They are capable of a great deal, but with only a couple years of existence, they are still in their infancy. They will grow more capable, yes, but we will also learn their limitations, and possibly only when those limitations have already created serious harms.\n",
    "\n",
    "As AI ethicists have warned repeatedly, the risks of AI are not some future apocalypse or theoretical systems displacing entire industries, but in overconfident and uninformed applications of the systems we have now. One CEO with stars in his eyes can do a lot of damage with AI models that are not inherently capable enough to do it themselves.\n",
    "\n",
    "\n",
    "\n",
    "The balancing act Microsoft must pull off is to invest at a pace that puts them ahead of their competitors, but not so far that they wind up in a minefield with everyone else watching from afar. It’s the curse of the innovator (or in this case integrator) that they should be the first to face new risks, and Microsoft seems ready to fill this role by putting AI to work in, as far as I can tell, nearly every single business unit and product where it can conceivably be included.\n",
    "\n",
    "Where will it find purchase? Where will it fail miserably? Where will it attract lawsuits? Where will it be regulated out of existence? Satya Nadella does not know, but he and his shareholders are going to find out, by god, one way or another. Things are getting exciting again.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords are-: ['ai', 'microsoft', 'like', 'business', 'way']\n",
      "Entities are-: ['first', 'hololens', 'only a couple years', 'two', 'a couple dozen', 'every month', 'satya nadella', 'this era', 'annual', 'cortana', 'forty-eight years', 'one', 'microsoft', 'blockchain', 'years', 'google', 'nadella', 'ai', 'million', 'cloud/mobile', 'today', 'openai', 'second', 'over a decade']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'In case you weren’t sure how Microsoft felt about AI, CEO Satya Nadella is here to tell you: they like it. They like it a lot. One CEO with stars in his eyes can do a lot of damage with AI models that are not inherently capable enough to do it themselves. It’s the curse of the innovator (or in this case integrator) that they should be the first to face new risks, and Microsoft seems ready to fill this role by putting AI to work in, as far as I can tell, nearly every single business unit and product where it can conceivably be included. Satya Nadella does not know, but he and his shareholders are going to find out, by god, one way or another.'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dynamic_detailed_summarize_premium(mic,0.5,'article')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "bir=\"\"\"On a cold winter day, a poor man came to the court of Emperor Akbar, shivering in his thin clothes. He complained that he had been fined unjustly by a judge for not wearing warm clothes in the winter, despite the fact that he couldn't afford them.\n",
    "\n",
    "Akbar wanted to know if the man was telling the truth. So, he turned to Birbal and asked, \"Is it really that cold outside?\"\n",
    "\n",
    "Birbal, sensing an opportunity to make a point, replied, \"I cannot say for sure, Your Majesty. Let me check.\"\n",
    "\n",
    "That evening, Birbal ordered a pot of khichdi (a type of rice and lentil dish) to be cooked in the open ground, far away from his home. He then asked a servant to place it where Akbar could see it from his palace.\n",
    "\n",
    "The next morning, Birbal invited Akbar to his home to show him the khichdi. The Emperor was puzzled. The pot was still there, but the khichdi was not cooked, despite the fact that the pot had been on the fire for a whole night.\n",
    "\n",
    "\"Why isn't the khichdi cooked?\" asked Akbar.\n",
    "\n",
    "Birbal replied, \"The fire is too far from the pot, just like the judge was too far removed from the reality of the poor man's situation.\"\n",
    "\n",
    "Akbar immediately understood the point Birbal was making. He realized that the judge had made an unfair decision without considering the circumstances of the man's life.\n",
    "\n",
    "The Emperor then pardoned the man and rewarded him with warm clothes and some money. The judge was also reprimanded for his lack of empathy.\n",
    "\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords are-: ['akbar', 'birbal', 'man', 'pot', 'khichdi']\n",
      "Entities are-: ['birbal', 'the winter', 'that evening', 'akbar', 'emperor akbar', 'winter day']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'On a cold winter day, a poor man came to the court of Emperor Akbar, shivering in his thin clothes. He complained that he had been fined unjustly by a judge for not wearing warm clothes in the winter, despite the fact that he couldn\\'t afford them. Akbar wanted to know if the man was telling the truth. So, he turned to Birbal and asked, \"Is it really that cold outside?\" He then asked a servant to place it where Akbar could see it from his palace. The next morning, Birbal invited Akbar to his home to show him the khichdi. asked Akbar. Birbal replied, \"The fire is too far from the pot, just like the judge was too far removed from the reality of the poor man\\'s situation.\" The Emperor then pardoned the man and rewarded him with warm clothes and some money.'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dynamic_detailed_summarize_premium(bir,2,'story')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords are-: ['emperor', 'turban', 'birbal', 'weight', 'gold']\n",
      "Entities are-: ['birbal', 'one day', 'akbar']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"One day, a merchant approached Emperor Akbar's court claiming that he had been cheated by his partner. He said that his partner owed him a large sum of gold, equivalent to the weight of the Emperor's turban. Nobody knew the exact weight of the Emperor's turban, as it had never been weighed before. Some courtiers suggested that the turban should be dismantled and weighed, but Emperor Akbar was reluctant to do so, as it was a symbol of his royal stature. The merchant was paid according to the weight and left the court satisfied. Emperor Akbar, impressed with Birbal's wisdom, rewarded him for his clever solution, which not only resolved the merchant's issue but also preserved the sanctity of the royal turban.\""
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dynamic_detailed_summarize_premium(\"\"\"One day, a merchant approached Emperor Akbar's court claiming that he had been cheated by his partner. He said that his partner owed him a large sum of gold, equivalent to the weight of the Emperor's turban.\n",
    "\n",
    "This left the court puzzled. Nobody knew the exact weight of the Emperor's turban, as it had never been weighed before. Some courtiers suggested that the turban should be dismantled and weighed, but Emperor Akbar was reluctant to do so, as it was a symbol of his royal stature.\n",
    "\n",
    "Seeing the dilemma, Birbal stepped forward with a solution. He asked for a large bowl of water and placed it on one side of a balance scale. On the other side, he placed the turban. He then started adding gold coins to the bowl of water until the scale balanced.\n",
    "\n",
    "Once the scales were balanced, Birbal declared, \"This is the weight of the Emperor's turban in gold.\"\n",
    "\n",
    "Everyone was amazed at Birbal's ingenuity. The merchant was paid according to the weight and left the court satisfied.\n",
    "\n",
    "Emperor Akbar, impressed with Birbal's wisdom, rewarded him for his clever solution, which not only resolved the merchant's issue but also preserved the sanctity of the royal turban.\n",
    "\n",
    "\"\"\",2,'story')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords are-: ['icc', 'court', 'said', 'cyberattack', 'risk']\n",
      "Entities are-: ['friday', 'icc', 'techcrunch', 'one', 'the international criminal court', 'daily', 'russian', 'court', 'netherlands', 'september', 'russia', 'last month', 'the hague', 'vladimir putin', 'sonia robla']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'The International Criminal Court (ICC), the world’s only permanent international court with a mandate to investigate and prosecute genocide, crimes against humanity and war crimes, has determined that a September cyberattack against its systems was attempted espionage. The ICC, which holds sensitive information related to alleged war crimes and data about witnesses who could be at risk if their identities were exposed, said it has not yet determined whether any data had been accessed or stolen during the cyberattack.'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dynamic_detailed_summarize_premium(\"\"\"The International Criminal Court (ICC), the world’s only permanent international court with a mandate to investigate and prosecute genocide, crimes against humanity and war crimes, has determined that a September cyberattack against its systems was attempted espionage.\n",
    "\n",
    "The court, headquartered in The Hague, Netherlands, confirmed last month that hackers had infiltrated its network. In an update posted Friday, the ICC said it had since determined that this incident was a “targeted and sophisticated attack” with the “objective of espionage.”\n",
    "\n",
    "“The attack can therefore be interpreted as a serious attempt to undermine the court’s mandate,” the court said.\n",
    "\n",
    "The ICC, which holds sensitive information related to alleged war crimes and data about witnesses who could be at risk if their identities were exposed, said it has not yet determined whether any data had been accessed or stolen during the cyberattack.\n",
    "\n",
    "“Should evidence be found that specific data entrusted to the Court has been compromised, those affected would be contacted immediately and directly by the Court,” the ICC stated. The court has yet to confirm the specific nature of the cyberattack, such as whether malware was deployed.\n",
    "\n",
    "ICC spokesperson Sonia Robla did not immediately respond to TechCrunch’s questions.\n",
    "\n",
    "The court said it has not yet confirmed who was behind the cyberattack, but noted that it anticipates it will face disinformation campaigns targeting the ICC and its officials in an attempt to delegitimize its activities.\n",
    "\n",
    "In its statement, the ICC noted that the cyberattack comes at a time of “broader and heightened security concerns” for the court. After the ICC issued arrest warrants for Russian President Vladimir Putin, Russia retaliated by issuing arrest warrants for the ICC’s president, their deputy, the chief prosecutor and one of the presiding judges.\n",
    "\n",
    "At the same time, the ICC said it has undergone “daily and persistent attempts” to attack and disrupt its systems.\n",
    "\n",
    "In response to the unprecedented September cyberattack, the ICC said it is strengthening its cybersecurity safeguards, which will include reinforcing its risk management framework and ensuring it has procedures in place to protect against any potential security risk to victims and witnesses.\"\"\",1,'article')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords are-: ['openai', 'gpt', 'chatgpt', 'users', 'text']\n",
      "Entities are-: ['meta', 'first', '100 million', 'chatgpt', 'gpt-4 turbo', 'december', 'nearly a year ago', 'fortune 500', 'safari', 'microsoft', 'microsoft edge', 'google chrome', '3', 'ai', 'llama', 'january', 'weekly', 'more than 92%', 'gpt-4', 'sam altman', 'openai', 'november 6', 'gpt']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'ChatGPT, OpenAI’s text-generating AI chatbot, has taken the world by storm. While there is a more…nefarious side to ChatGPT, it’s clear that AI tools are not going away anytime soon. And with the integration of DALL-E 3, users are also able to generate both text prompts and images right in ChatGPT.'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dynamic_detailed_summarize_premium(\"\"\"ChatGPT, OpenAI’s text-generating AI chatbot, has taken the world by storm. What started as a tool to hyper-charge productivity through writing essays and code with short text prompts has evolved into a behemoth used by more than 92% of Fortune 500 companies for more wide-ranging needs. And that growth has propelled OpenAI itself into becoming one of the most-hyped companies in recent memory, even if CEO and co-founder Sam Altman’s firing and swift return raised concerns about its direction and opened the door for competitors.\n",
    "\n",
    "What does that mean for OpenAI, ChatGPT and its other ambitions? The fallout is still settling, but it might empower competitors like Meta and its LLaMA family of large language models, or help other AI startups get attention and funding as the industry watches OpenAI implode and put itself back together.\n",
    "\n",
    "While there is a more…nefarious side to ChatGPT, it’s clear that AI tools are not going away anytime soon. Since its initial launch nearly a year ago, ChatGPT has hit 100 million weekly active users, and OpenAI is heavily investing in it.\n",
    "\n",
    "Prior to the leadership chaos, on November 6, OpenAI held its first developer conference: OpenAI DevDay. During the conference, it announced a slew of updates coming to GPT, including GPT-4 Turbo (a super-charged version of GPT-4, its latest language-writing model) and a multimodal API. OpenAI also unveiled the GPT store, where users could create and monetize their own custom versions of GPT, but its launch was delayed in December and is now set to launch sometime in January. \n",
    "\n",
    "GPT-4, which can write more naturally and fluently than previous models, remains largely exclusive to paying ChatGPT users. But you can access GPT-4 for free through Microsoft’s Bing Chat in Microsoft Edge, Google Chrome and Safari web browsers. Beyond GPT-4 and OpenAI DevDay announcements, OpenAI recently connected ChatGPT to the internet for all users. And with the integration of DALL-E 3, users are also able to generate both text prompts and images right in ChatGPT. \"\"\",1,'article')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".AI_service",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}