part-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1cc1d59c-d823-49f7-97fc-7e2575e1dc6a",
   "metadata": {},
   "source": [
    "# Part 2: Let's apply spaCy a bit\n",
    "\n",
    "I would now like to teach spaCy some more by applying it to a fun example. So in this video I will explain the data that can be found [here](https://github.com/mikeckennedy/talk-python-transcripts). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7007ec5-19aa-4d47-9137-ef5b2590e8d5",
   "metadata": {},
   "source": [
    "## A bit of data cleaning. \n",
    "\n",
    "It's a bit of regex, as well as some `str`-methods. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "bb48cc93-d579-4fdf-bf88-454edb11ef06",
   "metadata": {},
   "outputs": [],
   "source": [
    "import srsly \n",
    "from re import compile\n",
    "from pathlib import Path \n",
    "\n",
    "regex = compile(\"([0-9][0-9]:[0-9][0-9]:[0-9][0-9])\")\n",
    "\n",
    "def episode_lines(path):\n",
    "    i = 0\n",
    "    for line in Path(path).read_text().split(\"\\n\"):\n",
    "        if regex.match(line):\n",
    "            without_time = regex.sub(\"\", line[8:])\n",
    "            without_name = without_time[without_time.find(\":\") + 1: ]\n",
    "            speaker = without_time[:without_time.find(\":\")].strip() if (\":\" in without_time) else \"\"\n",
    "            i += 1\n",
    "            yield {\n",
    "                \"text\": without_name.strip().replace(\"\\xa0\", \" \"), \n",
    "                \"meta\": {\n",
    "                    \"speaker\": speaker,\n",
    "                    \"file\": str(path),\n",
    "                    \"turn\": i\n",
    "                }\n",
    "            }\n",
    "\n",
    "def all_episode_lines(turn_limit=None):\n",
    "    for path in reversed(sorted(Path(\"transcripts\").glob(\"*.txt\"))):\n",
    "        for line in episode_lines(path):\n",
    "            if turn_limit:\n",
    "                turn = line['meta']['turn']\n",
    "                if turn_limit:\n",
    "                    if turn in turn_limit:\n",
    "                        yield line\n",
    "                else:\n",
    "                    yield line\n",
    "            else:\n",
    "                yield line\n",
    "\n",
    "\n",
    "lines = all_episode_lines()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15cb9487-55b5-476f-af56-e2b9fc21474e",
   "metadata": {},
   "source": [
    "Let's first just iterate over some of the data to see spaCy in action. The results won't be perfect, but it's nice to see what you can get out of the box. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "d8ba84a1-cbcc-4b8c-a161-3dfffcde3eba",
   "metadata": {},
   "outputs": [],
   "source": [
    "lines = all_episode_lines(turn_limit=[25, 30, 31, 32])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "46799c19-546f-445f-91e5-7e7b0bd27e8a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': \"So one of the things you've been up to in addition to courses is writing books, Django in action, almost released. Is that right? What's the status?\",\n",
       " 'meta': {'speaker': '',\n",
       "  'file': 'transcripts/437-htmx-for-django-developers-and-all-of-us.txt',\n",
       "  'turn': 25}}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1957e335-36d9-44ce-8707-5f307ed6e4ea",
   "metadata": {},
   "source": [
    "You might look at my code and wonder ... \"why generators\"? If you're a data scientist you might be used to thinking in dataframes ... so why does it make sense to use generators here? \n",
    "\n",
    "In short, it just turns out to be slightly more convenient. Text can contain many nested items that we're interested in like entities and sentences. That nested structure already makes it somewhat inconvient to use a flat data structure like a table. So we're more likely to use a sequence of dictionaries. \n",
    "\n",
    "You could also use a list for that, but if you're dealing with _huge_ quantities of text ... then a lazy approach might make more sense. \n",
    "\n",
    "The code from before ... note how that only keeps one file open at a time? Stuff like that is _really_ nice. And this is why spaCy assumes generators."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcf9e633-77b4-41a6-906e-5ef322935c39",
   "metadata": {},
   "source": [
    "## Toying on Real Data \n",
    "\n",
    "So let's run our `en_core_web_md` model against some examples of the transcripts. Just to get a feel of happens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "7d1c7b01-d955-40ba-b1d1-b3f7c06238bf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">Exactly. And speaking of projects, we've been working on \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    one\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " of the things that kind of inspired me to have you on the show to just dive into \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Django\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    HTMX\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " is you actually did a talk Python course, \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    HTMX\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Django\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
       "</mark>\n",
       " modern Python web apps hold the \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    JavaScript\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       ", which is awesome. That's a really fun, \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    just under two hours\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">TIME</span>\n",
       "</mark>\n",
       " course that really shows people like how to integrate \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    HTMX\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " into \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Django\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       ".</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc = nlp(next(lines)['text'])\n",
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22c650aa-e123-4695-95e8-650085af3f19",
   "metadata": {},
   "source": [
    "### Discuss \n",
    "\n",
    "The results are not perfect, but it's not bad when you consider that the spaCy model isn't trained on data that knows about twitter and programming tools. I personally think that it's interesting that `Django` is often detected as a person and sometimes as a product. It's not the worst mistake if you think about it ... \n",
    "\n",
    "So while we agree that it's not perfect, it might be fun to see how it fares on the entire dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63c201b2-cdd7-403c-8fb2-36d4dba65896",
   "metadata": {},
   "source": [
    "## Speed \n",
    "\n",
    "If you're going to use spaCy over a bunch of examples, then you may want to use `nlp.pipe` instead of `nlp`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "8cb932c7-2786-424a-8664-7a5e85d4f7ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "import itertools as it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "52b5b596-5bb5-4479-b2bd-cff312412820",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 8.09 s, sys: 15.2 ms, total: 8.11 s\n",
      "Wall time: 8.21 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "lines = all_episode_lines()\n",
    "\n",
    "subset = it.islice(lines, 1000)\n",
    "[nlp(line['text']).ents for line in subset];"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "61b21472-0886-4e93-9184-1655b4b81cf5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.01 s, sys: 187 ms, total: 3.2 s\n",
      "Wall time: 3.34 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "lines = all_episode_lines()\n",
    "\n",
    "subset = (ex['text'] for ex in it.islice(lines, 1000))\n",
    "[doc.ents for doc in nlp.pipe(subset)];"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "24672771-0e17-49ce-bfd8-89b3c1ab44fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.03 s, sys: 123 ms, total: 3.15 s\n",
      "Wall time: 3.41 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "lines = all_episode_lines()\n",
    "\n",
    "subset = ((ex['text'], ex) for ex in it.islice(lines, 1000))\n",
    "[doc.ents for doc, ex in nlp.pipe(subset, as_tuples=True)];"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "032dc1a0-8594-431a-a355-eae0e0daeac3",
   "metadata": {},
   "source": [
    "There's also another speedup that we can do. We can also choose to load the spaCy model with only the components that we _really_ need. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "93da97df-1759-495d-8b98-ea028919ab9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load(\"en_core_web_md\", enable=[\"ner\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e89a0a4-1f2b-41eb-9043-9dee7625eb20",
   "metadata": {},
   "source": [
    "This will enable only the `ner` component in the pipeline and will disable anything else that we may not need. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "c7a5181a-0d5f-4ed0-a05e-1b7fcfe28b47",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.2 s, sys: 175 ms, total: 1.37 s\n",
      "Wall time: 1.42 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "lines = all_episode_lines()\n",
    "\n",
    "subset = ((ex['text'], ex) for ex in it.islice(lines, 1000))\n",
    "[doc.ents for doc, ex in nlp.pipe(subset, as_tuples=True)];"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08cfbda4-0de1-4442-8c5a-5a897f417e19",
   "metadata": {},
   "source": [
    "So again we see that it's a bunch quicker. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31bd0ff8-fb4c-4ee4-98d2-4038466369c4",
   "metadata": {},
   "source": [
    "## Hacking on Python Packages\n",
    "\n",
    "Let's now check if spaCy can detect Python packages by having it detect `PRODUCT` entities. It's not going to be a perfect mapping, but you might be able to imagine how spaCy might mistake a Python package for a product given how it is used in a sentence linguistically. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "c6c98824-6fdd-4c40-9d92-8d5ad84b3e79",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "84396"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import itertools as it \n",
    "from collections import Counter \n",
    "\n",
    "lines = (line['text'] for line in all_episode_lines())\n",
    "n_lines = sum(1 for _ in all_episode_lines())\n",
    "n_lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "9f79557f-389a-4591-8ded-4403727bbac2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "46c69ce1-570d-48a2-b2f1-8320f7d03e5e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/84396 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 84396/84396 [02:23<00:00, 588.42it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 8s, sys: 12.7 s, total: 2min 21s\n",
      "Wall time: 2min 23s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "counter = Counter()\n",
    "for doc in nlp.pipe(tqdm(lines, total=n_lines)):\n",
    "    product_entities = [ent.text for ent in doc.ents if ent.label_ == \"PRODUCT\"]\n",
    "    counter.update(Counter(product_entities))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "9f68c37e-ca05-41f0-8cb7-dba4b2a310df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Linux', 577),\n",
       " ('Excel', 547),\n",
       " ('Windows', 517),\n",
       " ('Python', 425),\n",
       " ('Django', 307),\n",
       " ('JavaScript', 269),\n",
       " ('C++', 223),\n",
       " ('VS', 150),\n",
       " ('Docker', 141),\n",
       " ('Flask', 115),\n",
       " ('SQLAlchemy', 96),\n",
       " ('Google Play', 86),\n",
       " ('Matplotlib', 83),\n",
       " ('Emacs', 78),\n",
       " ('Pyodide', 70),\n",
       " ('Perl', 69),\n",
       " ('MATLAB', 69),\n",
       " ('VS Code', 61),\n",
       " ('Cython', 57),\n",
       " ('Apache', 47),\n",
       " ('FastAPI', 46),\n",
       " ('Celery', 45),\n",
       " ('Talkpython', 40),\n",
       " ('Pandas', 38),\n",
       " ('Twitter', 35),\n",
       " ('Reddit', 35),\n",
       " ('Fast API', 33),\n",
       " ('CS', 32),\n",
       " ('Sanic', 31),\n",
       " ('Keras', 29)]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counter.most_common(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db62092c-f14e-4184-a852-ac6f49861ddd",
   "metadata": {},
   "source": [
    "## Another Speedup \n",
    "\n",
    "Our approach can be faster if we add a few more cores. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "5f633bf7-88fb-45ad-b387-e0c72b089cc7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/84396 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 83%|████████▎ | 69632/84396 [02:46<00:43, 343.15it/s]"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "lines = (line['text'] for line in all_episode_lines())\n",
    "n_lines = sum(1 for _ in all_episode_lines())\n",
    "\n",
    "counter = Counter()\n",
    "for doc in nlp.pipe(tqdm(lines, total=n_lines), n_process=8):\n",
    "    new_count = Counter([ent.text for ent in doc.ents if ent.label_ == \"PRODUCT\"])\n",
    "    counter.update(new_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0ee81c0-cf74-4f3d-8088-eca0255fe07f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Linux', 577),\n",
       " ('Excel', 547),\n",
       " ('Windows', 517),\n",
       " ('Python', 425),\n",
       " ('Django', 307),\n",
       " ('JavaScript', 269),\n",
       " ('C++', 223),\n",
       " ('VS', 150),\n",
       " ('Docker', 141),\n",
       " ('Flask', 115),\n",
       " ('SQLAlchemy', 96),\n",
       " ('Google Play', 86),\n",
       " ('Matplotlib', 83),\n",
       " ('Emacs', 78),\n",
       " ('Pyodide', 70),\n",
       " ('Perl', 69),\n",
       " ('MATLAB', 69),\n",
       " ('VS Code', 61),\n",
       " ('Cython', 57),\n",
       " ('Apache', 47),\n",
       " ('FastAPI', 46),\n",
       " ('Celery', 45),\n",
       " ('Talkpython', 40),\n",
       " ('Pandas', 38),\n",
       " ('Twitter', 35),\n",
       " ('Reddit', 35),\n",
       " ('Fast API', 33),\n",
       " ('CS', 32),\n",
       " ('Sanic', 31),\n",
       " ('Keras', 29)]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counter.most_common(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2d30db5-aa40-4df3-a7dd-e8104bd5ba52",
   "metadata": {},
   "source": [
    "In our case, the speedup is not linear. This can be blamed on the `counter.update(new_count)` line. This operation becomes slower over time, but it's also a part of the process that can't be parallelized. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ad643fc-f40a-410e-99f0-5aa37bbf6058",
   "metadata": {},
   "source": [
    "## Another approach \n",
    "\n",
    "The approach that we just took works ... but it's a bit hacky and odds are that it's missing out on a whole bunch of packages. But we can also resort to another approach ... after all ... we have lists of popular python packages available [to us](https://hugovk.github.io/top-pypi-packages/). The linked site is super cool, it uses the BigQuery data that has every PYPI download. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d38c4e89-1717-4d41-bbdd-5020d679cccb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-11-27 14:25:40--  https://hugovk.github.io/top-pypi-packages/top-pypi-packages.min.json\n",
      "Resolving hugovk.github.io (hugovk.github.io)... 185.199.109.153, 185.199.108.153, 185.199.111.153, ...\n",
      "Connecting to hugovk.github.io (hugovk.github.io)|185.199.109.153|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 411246 (402K) [application/json]\n",
      "Saving to: ‘top-pypi-packages.min.json.1’\n",
      "\n",
      "top-pypi-packages-3 100%[===================>] 401,61K  1,14MB/s    in 0,3s    \n",
      "\n",
      "2023-11-27 14:25:40 (1,14 MB/s) - ‘top-pypi-packages.min.json.1’ saved [411246/411246]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! wget https://hugovk.github.io/top-pypi-packages/top-pypi-packages.min.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "818174d5-3c19-4940-911d-bc19f995c339",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['boto3',\n",
       " 'urllib3',\n",
       " 'botocore',\n",
       " 'requests',\n",
       " 'typing-extensions',\n",
       " 'setuptools',\n",
       " 'charset-normalizer',\n",
       " 'certifi',\n",
       " 's3transfer',\n",
       " 'wheel']"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "package_names = [ex[\"project\"] for ex in srsly.read_json(\"top-pypi-packages.min.json\")[\"rows\"]]\n",
    "package_names[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abb5c0b4-a0ce-4246-ba61-4dd636d7529b",
   "metadata": {},
   "source": [
    "If we want to detect Python packages, maybe we can just re-use this? Note that this approach won't be perfect either ... but it may cover a bunch of ground ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50e36f09-b13a-4860-97c2-92fe80222301",
   "metadata": {},
   "source": [
    "## Custom Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "5ce28547-642d-4924-bbe3-620a03d351c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 4 boto core\n"
     ]
    }
   ],
   "source": [
    "from spacy.matcher import Matcher\n",
    "nlp = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "matcher = Matcher(nlp.vocab)\n",
    "pattern = [{\"LOWER\": \"boto\"}, {\"LOWER\": \"core\"}]\n",
    "matcher.add(\"pypackage\", [pattern])\n",
    "\n",
    "doc = nlp(\"We use boto core a lot in our company.\")\n",
    "matches = matcher(doc)\n",
    "for match_id, start, end in matches:\n",
    "    string_id = nlp.vocab.strings[match_id]  # Get string representation\n",
    "    span = doc[start:end]  # The matched span\n",
    "    print(start, end, span.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "59ccb04f-dedd-4f5e-b824-2cc2bb3fc7b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7 8 Go\n"
     ]
    }
   ],
   "source": [
    "from spacy.matcher import Matcher\n",
    "nlp = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "matcher = Matcher(nlp.vocab)\n",
    "pattern = [{\"LOWER\": \"go\", \"POS\": {\"NOT_IN\": [\"VERB\"]}}]\n",
    "matcher.add(\"pypackage\", [pattern])\n",
    "\n",
    "doc = nlp(\"I sometimes also write some code in Go.\")\n",
    "matches = matcher(doc)\n",
    "for match_id, start, end in matches:\n",
    "    string_id = nlp.vocab.strings[match_id]  # Get string representation\n",
    "    span = doc[start:end]  # The matched span\n",
    "    print(start, end, span.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "de0dde4d-dd30-48d3-bab5-064e17c3b9b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 4 boto core\n"
     ]
    }
   ],
   "source": [
    "from spacy.matcher import PhraseMatcher\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "matcher = PhraseMatcher(nlp.vocab)\n",
    "terms = [\"boto core\", \"pandas\"]\n",
    "patterns = [nlp.make_doc(text) for text in terms]\n",
    "matcher.add(\"pypackage\", patterns)\n",
    "\n",
    "doc = nlp(\"We use boto core a lot in our company.\")\n",
    "matches = matcher(doc)\n",
    "for match_id, start, end in matches:\n",
    "    string_id = nlp.vocab.strings[match_id]  # Get string representation\n",
    "    span = doc[start:end]  # The matched span\n",
    "    print(start, end, span.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5ff692b-0153-409c-8e55-e92767517e6c",
   "metadata": {},
   "source": [
    "We can also extend this, so that the `Doc` will actually have the right span as an entity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "4a50e7cb-006e-43aa-ac0c-0ca8f3494cdb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">We use \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    boto core\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">pypackage</span>\n",
       "</mark>\n",
       " a lot in our company.</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from spacy.tokens import Span \n",
    "from spacy.matcher import PhraseMatcher\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_md\")\n",
    "\n",
    "matcher = PhraseMatcher(nlp.vocab)\n",
    "terms = [\"boto core\", \"pandas\"]\n",
    "patterns = [nlp.make_doc(text) for text in terms]\n",
    "matcher.add(\"pypackage\", patterns)\n",
    "\n",
    "doc = nlp(\"We use boto core a lot in our company.\")\n",
    "matches = matcher(doc)\n",
    "ents = []\n",
    "for match_id, start, end in matches:\n",
    "    string_id = nlp.vocab.strings[match_id]\n",
    "    ents.append(Span(doc, start, end, string_id))\n",
    "\n",
    "doc.set_ents(ents)\n",
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ea64189e-63ea-4b65-82df-c60a5ce8d758",
   "metadata": {},
   "source": [
    "I wanted to show this feature because it's very flexible. You can set entities yourself with custom code all you like ... but in our case ... we could also just re-use existing spaCy components. \n",
    "\n",
    "## Entity Component\n",
    "\n",
    "Besides statistical tools, spaCy also allows you to write rule-based solutions on top of it's data structures. \n",
    "\n",
    "Explain this image: https://spacy.io/usage/processing-pipelines\n",
    "\n",
    "You can add an entity ruler that does token based matching. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "0b465588-3a59-4ed0-8ee1-c58396091b5b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">I sometimes also write some code in \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Go\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">proglang</span>\n",
       "</mark>\n",
       ".</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "nlp = spacy.blank(\"en\")\n",
    "ruler = nlp.add_pipe(\"entity_ruler\")\n",
    "patterns = [{\"label\": \"proglang\", \"pattern\": [{\"LOWER\": \"go\", \"POS\": {\"NOT_IN\": [\"VERB\"]}}]}]\n",
    "ruler.add_patterns(patterns)\n",
    "\n",
    "doc = nlp(\"I sometimes also write some code in Go.\")\n",
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60b5afc5-3aba-49ac-9d5c-d0dee19eb206",
   "metadata": {},
   "source": [
    "But phrase based matching can also be done directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "20c93456-08ed-4590-85c2-777be55c82ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">I used to use \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    pandas\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">pypackage</span>\n",
       "</mark>\n",
       " a lot but nowadays I'm doing \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    polars\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">pypackage</span>\n",
       "</mark>\n",
       ".</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "nlp = spacy.blank(\"en\")\n",
    "ruler = nlp.add_pipe(\"entity_ruler\")\n",
    "patterns = [{\"label\": \"pypackage\", \"pattern\": pkg} for pkg in package_names]\n",
    "ruler.add_patterns(patterns)\n",
    "\n",
    "doc = nlp(\"I used to use pandas a lot but nowadays I'm doing polars.\")\n",
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae75ceee-e83a-4fbc-a768-63c5b2044790",
   "metadata": {},
   "source": [
    "Neat!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "0eea1148-b5d2-4589-a9b8-054dfb2a500b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method Language.to_disk of <spacy.lang.en.English object at 0x7f8545c63340>>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# If we wanted to re-use this model \n",
    "nlp.to_disk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16bbaf43-29a2-4d84-8a2f-132a238823ca",
   "metadata": {},
   "source": [
    "## Rerunning our model\n",
    "\n",
    "Let's now apply our rule based model to see if we capture something else."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "0e66daa3-c8e2-484d-8be7-b5dbae6d8afe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['pypackage']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[e.label_ for e in nlp(\"fastapi\").ents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "c0778fb8-a511-4984-bc34-e20b82e0c918",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 84396/84396 [00:42<00:00, 1990.55it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 49.4 s, sys: 457 ms, total: 49.9 s\n",
      "Wall time: 50.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "lines = (line['text'] for line in all_episode_lines())\n",
    "# lines = it.islice(lines, 10000)\n",
    "n_lines = sum(1 for _ in all_episode_lines())\n",
    "\n",
    "counter = Counter()\n",
    "for doc in nlp.pipe(tqdm(lines, total=n_lines), n_process=8):\n",
    "    new_count = Counter([ent.text for ent in doc.ents if ent.label_ == \"pypackage\"])\n",
    "    counter.update(new_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "11482cda-c605-4f47-8755-6b01d0b5cba2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('data', 9745),\n",
       " ('sure', 4607),\n",
       " ('first', 3755),\n",
       " ('us', 3628),\n",
       " ('build', 2886),\n",
       " ('bunch', 2176),\n",
       " ('install', 1493),\n",
       " ('times', 1464),\n",
       " ('ago', 1299),\n",
       " ('control', 791),\n",
       " ('click', 731),\n",
       " ('pip', 694),\n",
       " ('future', 657),\n",
       " ('portion', 629),\n",
       " ('pick', 608),\n",
       " ('requests', 607),\n",
       " ('six', 575),\n",
       " ('email', 569),\n",
       " ('notebook', 559),\n",
       " ('path', 514),\n",
       " ('image', 507),\n",
       " ('dependencies', 501),\n",
       " ('style', 484),\n",
       " ('update', 440),\n",
       " ('moment', 439),\n",
       " ('eight', 438),\n",
       " ('databases', 412),\n",
       " ('public', 376),\n",
       " ('workflow', 373),\n",
       " ('pandas', 357),\n",
       " ('typing', 346),\n",
       " ('connect', 340),\n",
       " ('events', 316),\n",
       " ('flask', 312),\n",
       " ('packaging', 300),\n",
       " ('coverage', 294),\n",
       " ('result', 290),\n",
       " ('black', 278),\n",
       " ('pattern', 273),\n",
       " ('higher', 268),\n",
       " ('names', 258),\n",
       " ('waiting', 251),\n",
       " ('mode', 243),\n",
       " ('binary', 239),\n",
       " ('distributed', 238),\n",
       " ('progress', 234),\n",
       " ('segments', 216),\n",
       " ('rules', 215),\n",
       " ('schema', 210),\n",
       " ('tables', 204),\n",
       " ('rich', 199),\n",
       " ('asyncio', 199),\n",
       " ('constantly', 185),\n",
       " ('art', 181),\n",
       " ('conda', 178),\n",
       " ('secure', 177),\n",
       " ('wheel', 174),\n",
       " ('fire', 168),\n",
       " ('markdown', 151),\n",
       " ('decorator', 146),\n",
       " ('statistics', 144),\n",
       " ('ta', 140),\n",
       " ('dash', 138),\n",
       " ('patch', 136),\n",
       " ('item', 133),\n",
       " ('parse', 132),\n",
       " ('lazy', 130),\n",
       " ('logging', 129),\n",
       " ('properties', 128),\n",
       " ('secret', 128),\n",
       " ('mock', 124),\n",
       " ('spin', 123),\n",
       " ('schedule', 122),\n",
       " ('face', 121),\n",
       " ('returns', 120),\n",
       " ('pipe', 111),\n",
       " ('channels', 109),\n",
       " ('paste', 107),\n",
       " ('secrets', 104),\n",
       " ('identify', 102),\n",
       " ('flaky', 102),\n",
       " ('panel', 100),\n",
       " ('transaction', 99),\n",
       " ('poetry', 97),\n",
       " ('rust', 97),\n",
       " ('config', 94),\n",
       " ('bytecode', 94),\n",
       " ('outcome', 94),\n",
       " ('records', 91),\n",
       " ('spaces', 91),\n",
       " ('keyboard', 89),\n",
       " ('holidays', 89),\n",
       " ('pytest', 85),\n",
       " ('emails', 84),\n",
       " ('corner', 82),\n",
       " ('docker', 82),\n",
       " ('py', 78),\n",
       " ('sockets', 77),\n",
       " ('limits', 77),\n",
       " ('matplotlib', 77),\n",
       " ('hatch', 76),\n",
       " ('translate', 75),\n",
       " ('bookstore', 75),\n",
       " ('transform', 73),\n",
       " ('distribute', 72),\n",
       " ('behave', 69),\n",
       " ('textual', 68),\n",
       " ('discord', 67),\n",
       " ('vector', 67),\n",
       " ('dask', 67),\n",
       " ('username', 66),\n",
       " ('dataset', 66),\n",
       " ('responses', 65),\n",
       " ('multiprocessing', 65),\n",
       " ('safety', 63),\n",
       " ('slack', 63),\n",
       " ('hypothesis', 62),\n",
       " ('hacking', 61),\n",
       " ('s3', 61),\n",
       " ('lib', 60),\n",
       " ('installer', 59),\n",
       " ('datasets', 58),\n",
       " ('executing', 58),\n",
       " ('distance', 58),\n",
       " ('evaluate', 57),\n",
       " ('scikit-learn', 54),\n",
       " ('arrow', 53),\n",
       " ('construct', 53),\n",
       " ('pyramid', 52),\n",
       " ('selection', 52),\n",
       " ('promise', 51),\n",
       " ('cookies', 51),\n",
       " ('routes', 49),\n",
       " ('deprecated', 49),\n",
       " ('retry', 49),\n",
       " ('threaded', 48),\n",
       " ('pydantic', 47),\n",
       " ('priority', 47),\n",
       " ('fixture', 45),\n",
       " ('fixtures', 43),\n",
       " ('formulas', 43),\n",
       " ('incremental', 43),\n",
       " ('ruff', 41),\n",
       " ('inject', 40),\n",
       " ('blob', 40),\n",
       " ('pipx', 39),\n",
       " ('particle', 39),\n",
       " ('pony', 39),\n",
       " ('implicit', 38),\n",
       " ('interpret', 38)]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counter.most_common(150)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7970457-f36e-4403-8fe9-a6b2610276e8",
   "metadata": {},
   "source": [
    "A few things to note:\n",
    "\n",
    "- This approach is _much_ faster the `en_core_web_md` model. That's because we're just doing string matching.\n",
    "- This approach seems to match a bunch of Python packages. But it's not perfect either! It seems to match a _bunch_ of things that don't feel like packages. But we can check to confirm that they actually are ... "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaafdedd-d998-4347-b340-e6cfa05ff44c",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}