topicnet/demos/RTL-Wiki-Preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RTL Wiki. Preprocessing\n",
    "\n",
    "Example of a text data preprocessing before topic modeling (applied to RTL-Wiki dataset).\n",
    "\n",
    "Alternatively, you can skip this step and use our `topicnet.dataset_manager`.\n",
    "\n",
    "The dataset was introduced in [1] and later recreated in [2]. You can download it from https://hobbitdata.informatik.uni-leipzig.de/homes/mroeder/palmetto/datasets/rtl-wiki.tar.gz.\n",
    "\n",
    "References:\n",
    "\n",
    "* [1] \"Reading Tea Leaves: How Humans Interpret Topic Models\" (NIPS 2009).\n",
    "* [2] \"Exploring the Space of Topic Coherence Measures\" (WSDM 2015).\n",
    "* Related notebook (with preprocessing for other dataset): [20NG-Preprocessing.ipynb](./20NG-Preprocessing.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contents<a id=\"contents\"></a>\n",
    "\n",
    "* [Data loading](#data-loading)\n",
    "* [Data filtering](#data-filtering)\n",
    "* [Text preprocessing (lemmatization, stopwords filtering, N-grams extraction)](#text-preprocessing)\n",
    "* [Vowpal Wabbit](#vowpal-wabbit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import os\n",
    "import re\n",
    "import string\n",
    "from collections import Counter\n",
    "\n",
    "try:\n",
    "    from tqdm.notebook import tqdm\n",
    "except ModuleNotFoundError:\n",
    "    # TODO: remove after tqdm v5.0.0 released\n",
    "    from tqdm import tqdm_notebook as tqdm\n",
    "\n",
    "\n",
    "import nltk\n",
    "import pandas as pd\n",
    "from bs4 import BeautifulSoup\n",
    "from nltk.collocations import (\n",
    "    BigramAssocMeasures,\n",
    "    BigramCollocationFinder,\n",
    ")\n",
    "from nltk.corpus import (\n",
    "    stopwords,\n",
    "    wordnet,\n",
    ")\n",
    "from nltk.stem import WordNetLemmatizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data loading<a id=\"data-loading\"></a>\n",
    "\n",
    "<div style=\"text-align: right\">Back to <a href=#contents>Contents</a></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2022-11-17 20:10:42--  https://hobbitdata.informatik.uni-leipzig.de/homes/mroeder/palmetto/datasets/rtl-wiki.tar.gz\n",
      "Resolving hobbitdata.informatik.uni-leipzig.de (hobbitdata.informatik.uni-leipzig.de)... 139.18.121.22\n",
      "Connecting to hobbitdata.informatik.uni-leipzig.de (hobbitdata.informatik.uni-leipzig.de)|139.18.121.22|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 176728881 (169M) [application/octet-stream]\n",
      "Saving to: ‘rtl-wiki.tar.gz’\n",
      "\n",
      "rtl-wiki.tar.gz     100%[===================>] 168.54M  2.14MB/s    in 69s     \n",
      "\n",
      "2022-11-17 20:11:56 (2.46 MB/s) - ‘rtl-wiki.tar.gz’ saved [176728881/176728881]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# download corpus and unpack it:\n",
    "\n",
    "! wget https://hobbitdata.informatik.uni-leipzig.de/homes/mroeder/palmetto/datasets/rtl-wiki.tar.gz -O rtl-wiki.tar.gz\n",
    "! tar xzf rtl-wiki.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The corpus is a sample of 10000 articles from English Wikipedia in a MediaWiki markup format.\n",
    "\n",
    "Hence, we need to strip specific wiki formatting. We advise using a `mwparserfromhell` fork optimized to deal with the English Wikipedia.\n",
    "\n",
    "```bash\n",
    "git clone --branch images_and_interwiki https://github.com/bt2901/mwparserfromhell.git\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'mwparserfromhell'...\n",
      "remote: Enumerating objects: 34, done.\u001b[K\n",
      "remote: Counting objects: 100% (34/34), done.\u001b[K\n",
      "remote: Compressing objects: 100% (25/25), done.\u001b[K\n",
      "remote: Total 5756 (delta 15), reused 19 (delta 9), pack-reused 5722\n",
      "Receiving objects: 100% (5756/5756), 1.75 MiB | 2.78 MiB/s, done.\n",
      "Resolving deltas: 100% (3987/3987), done.\n"
     ]
    }
   ],
   "source": [
    "! git clone --branch images_and_interwiki https://github.com/bt2901/mwparserfromhell.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mwparserfromhell import mwparserfromhell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data filtering<a id=\"data-filtering\"></a>\n",
    "\n",
    "<div style=\"text-align: right\">Back to <a href=#contents>Contents</a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Wikipedia dataset is too heterogenous. Building a good topic model here requires a lot of topics or a lot of documents.\n",
    "\n",
    "To make collection more focused, we will filter out everything which isn't about people. We will use the following criteria to distinguish people and not-people:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# all infoboxes related to persons, according to  https://en.wikipedia.org/wiki/Wikipedia:List_of_infoboxes\n",
    "person_infoboxes = {'infobox magic: the gathering player', 'infobox architect', 'infobox mountaineer', 'infobox scientist', 'infobox chess biography', 'infobox racing driver', 'infobox saint', 'infobox snooker player', 'infobox figure skater', 'infobox theological work', 'infobox gaelic athletic association player', 'infobox professional wrestler', 'infobox noble', 'infobox pelotari', 'infobox native american leader', 'infobox pretender', 'infobox amateur wrestler', 'infobox college football player', 'infobox buddha', 'infobox cfl biography', 'infobox playboy playmate', 'infobox cyclist', 'infobox martial artist', 'infobox motorcycle rider', 'infobox motocross rider', 'infobox bandy biography', 'infobox video game player', 'infobox dancer', 'infobox nahua officeholder', 'infobox criminal', 'infobox squash player', 'infobox go player', 'infobox bullfighting career', 'infobox engineering career', 'infobox pirate', 'infobox latter day saint biography', 'infobox sumo wrestler', 'infobox youtube personality', 'infobox national hockey league coach', 'infobox rebbe', 'infobox football official', 'infobox aviator', 'infobox pharaoh', 'infobox classical composer', 'infobox fbi ten most wanted', 'infobox chef', 'infobox engineer', 'infobox nascar driver', 'infobox medical person', 'infobox jewish leader', 'infobox horseracing personality', 'infobox poker player', 'infobox economist', 'infobox peer', 'infobox war on terror detainee', 'infobox philosopher', 'infobox professional bowler', 'infobox champ car driver', 'infobox golfer', 'infobox le mans driver', 'infobox alpine ski racer', 'infobox boxer (amateur)', 'infobox bodybuilder', 'infobox college coach', 'infobox speedway rider', 'infobox skier', 'infobox medical details', 'infobox field hockey player', 'infobox badminton player', 'infobox sports announcer details', 'infobox academic', 'infobox f1 driver', 'infobox ncaa athlete', 'infobox biathlete', 'infobox comics creator', 'infobox rugby league biography', 'infobox fencer', 'infobox theologian', 'infobox religious biography', 'infobox egyptian dignitary', 'infobox curler', 'infobox racing driver series section', 'infobox afl biography', 'infobox speed skater', 'infobox climber', 'infobox rugby biography', 'infobox clergy', 'infobox equestrian', 'infobox member of the knesset', 'infobox pageant titleholder', 'infobox lacrosse player', 'infobox tennis biography', 'infobox gymnast', 'infobox sport wrestler', 'infobox sports announcer', 'infobox surfer', 'infobox darts player', 'infobox christian leader', 'infobox presenter', 'infobox gunpowder plotter', 'infobox table tennis player', 'infobox sailor', 'infobox astronaut', 'infobox handball biography', 'infobox volleyball biography', 'infobox spy', 'infobox wrc driver', 'infobox police officer', 'infobox swimmer', 'infobox netball biography', 'infobox model', 'infobox comedian', 'infobox boxer'}\n",
    "\n",
    "# is page included in a category with demography information?\n",
    "demography_re = re.compile(\"([0-9]+ (deaths|births))|(living people)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "dir_name = \"persons\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "! mkdir $dir_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6fce07b982a846168621e66832d36bbc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=7838), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "for filename in tqdm(glob.glob(\"documents/*.html\")):\n",
    "    doc_id = filename.partition(\"/\")[-1]\n",
    "    doc_id = doc_id.rpartition(\".\")[0] + \".txt\"\n",
    "    is_about_person = False\n",
    "\n",
    "    with open(filename, \"r\") as f:\n",
    "        soup = BeautifulSoup(\"\".join(f.readlines()))\n",
    "        text = soup.findAll('textarea', id=\"wpTextbox1\")[0].contents[0]\n",
    "        text = text.replace(\"&amp;\", \"&\").replace('&lt;', '<').replace('&gt;', '>')\n",
    "        wikicode = mwparserfromhell.parse(text)\n",
    "\n",
    "        if dir_name == \"persons\":\n",
    "            for node in wikicode.nodes:\n",
    "                entry_type = str(type(node))\n",
    "\n",
    "                if \"Wikilink\" in entry_type:\n",
    "                    special_link_name, _, cat_name = node.title.lower().strip().partition(\":\")\n",
    "                    if special_link_name == \"category\":\n",
    "                        if demography_re.match(cat_name):\n",
    "                            is_about_person = True\n",
    "\n",
    "                if \"Template\" in entry_type:\n",
    "                    name = str(node.name).lower().strip()\n",
    "                    if name in person_infoboxes:\n",
    "                        is_about_person = True\n",
    "\n",
    "            should_be_saved = is_about_person\n",
    "        else:\n",
    "            # Replace False value for True if you want all of RTL Wiki Dataset\n",
    "            should_be_saved = False\n",
    "\n",
    "        if should_be_saved:\n",
    "            with open(f\"{dir_name}/{doc_id}\", \"w\") as f2:\n",
    "                stripped_text = wikicode.strip_code()\n",
    "                f2.write(stripped_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a folder `persons` which contains 1201 document. Let's take a look at the one of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eusebius of Caesarea\r\n",
      "Eusebius of Caesarea (c. 263 – c. 339) (often called Eusebius Pamphili, \"Eusebius [the friend] of Pamphilus\") became the bishop of Caesarea Palaestina, the capital of Iudaea province, c 314. He is often referred to as the Father of Church History because of his work in recording the history of the early Christian church, especially Chronicle and Ecclesiastical History Wetterau, Bruce. World history. New York: Henry Holt and company. 1994. . \r\n",
      "\r\n",
      "Biography\r\n",
      "\r\n",
      "His  date and place of birth are unknown and little is known of his youth, however it is estimated that he was born in 265 Encyclopedia of the Early Church, Published in 1992, English Version, page 299 . He became acquainted with the presbyter Dorotheus in Antioch and probably received exegetical instruction from him. In 296 he was in Palestine and saw Constantine who visited the country with Diocletian. He was in Caesarea when Agapius was bishop and became friendly with Pamphilus of Caesarea, with whom he seems to have studied the text of the Bible, with the aid of Origen's Hexapla and commentaries collected by Pamphilus, in an attempt to prepare a correct version.\r\n",
      "\r\n",
      "In 307, Pamphilus was imprisoned, but Eusebius continued their project.  The resulting defence of Origen, in which they had collaborated, was finished by Eusebius after the death of Pamphilus and sent to the martyrs in the mines of Phaeno located in modern Jordan.  Eusebius then seems to have gone to Tyre and later to Egypt, where he first suffered persecution.\r\n",
      "\r\n",
      "Eusebius is next heard of as bishop of Caesarea Maritima.  He succeeded Agapius, whose time of office is not certain, but Eusebius must have become bishop soon after 313.  Nothing is known about the early years of his tenure.  When the Council of Nicaea met in 325, Eusebius was prominent in its transactions.  He was not naturally a spiritual leader or theologian, but as a very learned man and a famous author who enjoyed the special favour of the emperor, he came to the fore among the members of the council (traditionally given as 318 attendees). He presented a confession to the council which was rejected in favour of a more specifically anti-Arian creed from Palestine. This later creed became the basis of the council's major theological statement, the Nicene Creed. Bruce L. Shelley, Church History in Plain Language, (2nd ed. Dallas, TX: Word Publishing, 1995.), p.102. \r\n"
     ]
    }
   ],
   "source": [
    "! head $dir_name/Eusebius.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text preprocessing (lemmatization, stopwords filtering, N-grams extraction)<a id=\"text-preprocessing\"></a>\n",
    "\n",
    "<div style=\"text-align: right\">Back to <a href=#contents>Contents</a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to lemmatize texts, remove stopwords and extract informative ngramms.\n",
    "\n",
    "There's no one \"correct\" way to do it, but the reasonable baseline is using well-known `nltk` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.data.path.append(\n",
    "    os.path.join(os.path.expanduser(\"~\"), 'nltk_data')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = glob.glob(dir_name + '/*.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = []\n",
    "\n",
    "for path in files:\n",
    "    entry = {}\n",
    "    entry['id'] = path.split('/')[-1].rpartition(\".\")[0]\n",
    "\n",
    "    with open(path, 'r') as f:\n",
    "        entry['raw_text'] = \" \".join(line.strip() for line in f.readlines())\n",
    "\n",
    "    data.append(entry)\n",
    "\n",
    "wiki_texts = pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1201/1201 [01:37<00:00, 12.29it/s]\n"
     ]
    }
   ],
   "source": [
    "tokenized_text = []\n",
    "\n",
    "for text in tqdm(wiki_texts['raw_text'].values):\n",
    "    tokens = nltk.wordpunct_tokenize(text.lower())\n",
    "    tokenized_text.append(nltk.pos_tag(tokens))\n",
    "\n",
    "wiki_texts['tokenized'] = tokenized_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def nltk2wn_tag(nltk_tag):\n",
    "    if nltk_tag.startswith('J'):\n",
    "        return wordnet.ADJ\n",
    "    elif nltk_tag.startswith('V'):\n",
    "        return wordnet.VERB\n",
    "    elif nltk_tag.startswith('N'):\n",
    "        return wordnet.NOUN\n",
    "    elif nltk_tag.startswith('R'):\n",
    "        return wordnet.ADV\n",
    "    else:          \n",
    "        return ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop = set(stopwords.words('english'))\n",
    "\n",
    "lemmatized_text = []\n",
    "wnl = WordNetLemmatizer()\n",
    "\n",
    "for text in wiki_texts['tokenized'].values:\n",
    "    lemmatized = [wnl.lemmatize(word,nltk2wn_tag(pos))\n",
    "                  if nltk2wn_tag(pos) != ''\n",
    "                  else wnl.lemmatize(word)\n",
    "                  for word, pos in text ]\n",
    "    lemmatized = [word for word in lemmatized \n",
    "                  if word not in stop and word.isalpha()]\n",
    "    lemmatized_text.append(lemmatized)\n",
    "\n",
    "wiki_texts['lemmatized'] = lemmatized_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ngrams are a powerful feature, and BigARTM is able to take advantage of it (the technical term is 'multimodal topic modeling': our topic model could model a lot of different features linked to a specific document, not just words)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "bigram_measures = BigramAssocMeasures()\n",
    "finder = BigramCollocationFinder.from_documents(wiki_texts['lemmatized'])\n",
    "\n",
    "finder.apply_freq_filter(5)\n",
    "set_dict = set(finder.nbest(bigram_measures.pmi,32100)[100:])\n",
    "documents = wiki_texts['lemmatized']\n",
    "bigrams = []\n",
    "\n",
    "for doc in documents:\n",
    "    entry = ['_'.join([word_first, word_second])\n",
    "             for word_first, word_second in zip(doc[:-1],doc[1:])\n",
    "             if (word_first, word_second) in set_dict]\n",
    "    bigrams.append(entry)\n",
    "\n",
    "wiki_texts['bigram'] = bigrams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vowpal Wabbit<a id=\"vowpal-wabbit\"></a>\n",
    "\n",
    "<div style=\"text-align: right\">Back to <a href=#contents>Contents</a></div>\n",
    "\n",
    "* More info about different text formats (including Vowpal Wabbit): [bigartm.formats.html](https://bigartm.readthedocs.io/en/v0.8.1/formats.html)\n",
    "* Example of a toy collection in Vowpal Wabbit format: [sample_vw.txt](https://github.com/machine-intelligence-laboratory/OptimalNumberOfTopics/blob/master/sample/vw.txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def vowpalize_sequence(sequence):\n",
    "    word_2_frequency = Counter(sequence)\n",
    "\n",
    "    del word_2_frequency['']\n",
    "\n",
    "    vw_string = ''\n",
    "\n",
    "    for word in word_2_frequency:\n",
    "        vw_string += word + \":\" + str(word_2_frequency[word]) + ' '\n",
    "\n",
    "    return vw_string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "vw_text = []\n",
    "\n",
    "for index, data in wiki_texts.iterrows():\n",
    "    vw_string = ''    \n",
    "    doc_id = data.id\n",
    "    lemmatized = '@lemmatized ' + vowpalize_sequence(data.lemmatized)\n",
    "    bigram = '@bigram ' + vowpalize_sequence(data.bigram)\n",
    "    vw_string = ' |'.join([doc_id, lemmatized, bigram])\n",
    "    vw_text.append(vw_string)\n",
    "\n",
    "wiki_texts['vw_text'] = vw_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vowpal Wabbit (\"wv\") is a text format which is a good fit for multimodal topic modeling. Here, we elected to store dataset in a Bag-of-Words format (for performance reasons), but VW could store everything as a sequence of words as well.\n",
    "\n",
    "It looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'İsmet_İnönü |@lemmatized mustafa:2 smet:7 nönü:18 september:1 december:2 turkish:25 army:8 general:5 tsk:1 genel:1 kurmay:1 baskanlari:1 prime:5 minister:5 second:4 president:6 republic:4 turkey:19 widely:1 refer:1 milli:2 şef:2 national:5 chief:5 title:2 bestow:1 upon:1 elect:2 family:3 early:5 life:2 bear:2 zmir:2 originally:1 malatya:3 mixed:1 kurdish:1 heritage:1 young:2 turk:2 child:1 borderland:1 erik:1 jan:1 zürcher:1 universiteit:1 leiden:1 ismet:2 inonu:1 making:1 statesman:1 metin:1 heper:1 brill:1 academic:1 publisher:1 father:2 hacı:1 reşid:1 bey:1 member:2 ottoman:5 bureaucracy:1 examine:1 magistrate:1 mother:1 cevriye:1 hanım:1 daughter:1 russo:1 war:19 refugee:1 bulgaria:1 due:3 assignment:3 move:1 one:1 city:2 another:2 thus:1 complete:1 primary:1 education:1 siva:1 half:1 know:3 pasha:3 change:1 name:2 atatürk:8 decree:1 countryman:1 surname:2 decide:1 take:2 central:1 anatolian:1 town:1 command:1 force:3 great:2 battle:6 first:8 victory:3 play:1 important:1 role:1 independence:4 son:1 erdal:1 wigner:1 medal:1 winner:1 mathematical:1 physicist:1 former:2 deputy:1 well:3 leader:6 social:3 democracy:2 party:12 democratic:5 populist:1 honorary:1 people:1 military:7 career:2 graduate:1 academy:1 gunnery:1 officer:2 receive:1 join:3 committee:1 union:3 progress:1 win:3 suppress:1 two:1 major:2 revolt:1 struggle:1 empire:1 rumelia:1 later:5 yemen:1 whose:1 yahya:1 muhammad:1 hamid:1 ed:1 din:1 serve:5 balkan:2 bulgarian:1 front:5 world:5 miralay:1 colonel:1 eastern:1 syria:1 appoint:4 commander:2 western:3 work:1 together:1 kemal:2 caucasus:1 lose:6 megiddo:1 edmund:1 allenby:1 last:2 day:2 go:1 anatolia:2 nationalist:1 movement:1 position:2 remain:1 promote:1 brigadier:1 successfully:1 defeat:1 greek:2 also:3 grand:1 assembly:1 ankara:3 replace:1 fevzi:1 paşa:1 staff:1 advance:1 july:1 result:1 afyon:1 kütahya:1 eskişehir:1 negotiator:1 delegation:1 treaty:1 lausanne:1 become:1 famous:1 resolve:1 stubbornness:1 defend:1 demand:2 concede:1 little:1 side:4 negotiating:1 table:1 cause:3 peace:2 conference:2 long:2 expect:1 partially:1 deaf:1 simply:1 turn:1 hearing:1 aid:2 british:2 foreign:3 secretary:2 lord:1 curzon:1 launch:1 lengthy:1 speech:1 oppose:1 recognition:1 pact:1 would:5 restate:1 say:2 thing:1 political:2 several:1 term:1 maintain:2 system:2 put:1 place:2 act:1 every:1 crisis:2 rebellion:1 sheikh:1 attempted:1 assassination:1 restore:1 country:5 try:1 manage:1 economy:1 heavy:1 handed:1 government:4 intervention:1 especially:1 economic:5 implement:1 plan:2 inspire:1 five:1 year:3 soviet:4 much:1 private:2 property:1 control:2 effort:1 land:1 still:5 state:3 resemble:1 defunct:1 desire:2 liberal:1 inönü:12 leadership:2 celal:1 bayar:1 founder:1 bank:1 türkiye:1 ş:1 bankası:1 period:2 churchill:6 secretly:2 meet:3 inside:2 train:2 wagon:2 yenice:1 station:1 kilometer:1 outside:1 adana:2 january:2 death:1 view:1 appropriate:1 candidate:1 succeed:1 enjoy:1 official:2 e:1 ii:2 break:1 presidency:3 ally:5 axis:3 pressure:2 bring:1 german:2 send:1 franz:1 von:1 papen:1 winston:2 near:1 franklin:1 roosevelt:4 cairo:1 think:1 continue:2 neutrality:2 interest:1 block:1 reach:1 strategic:1 oil:1 reserve:1 middle:1 east:1 end:1 evaluate:1 possible:1 participation:1 decently:1 size:1 air:1 throughout:1 want:2 open:2 new:1 hand:1 believe:1 attack:1 risky:1 eventual:1 failure:1 disastrous:1 effect:1 hardship:1 suffer:1 decade:1 incessant:1 determine:1 keep:2 could:2 building:1 recover:1 loss:1 lack:1 modern:1 weapon:1 infrastructure:1 enter:2 fight:1 along:1 possibly:1 within:2 border:1 assurance:1 financial:1 guarantee:1 united:2 kingdom:1 stand:1 beside:1 event:1 invasion:2 strait:2 fear:1 stalin:1 unconcealed:1 eventually:1 give:2 principle:1 relation:1 nato:1 condition:1 describe:1 depression:1 evidence:1 negative:1 growth:1 http:1 www:1 ggdc:1 net:1 maddison:1 horizontal:1 xl:1 may:2 explain:1 election:8 yet:1 overcome:1 strong:1 opposition:4 switch:1 multi:2 allow:1 previously:1 mention:1 hold:2 probably:2 history:2 absolute:1 ruling:1 power:3 achievement:1 february:1 germany:1 japan:1 tomb:2 anıtkabir:2 international:1 transform:1 reside:1 infamous:1 vote:2 cast:2 onlooker:1 secret:1 police:1 able:1 observe:1 voter:1 ballot:1 tally:1 behind:1 closed:1 door:1 free:1 preside:1 peaceful:1 transfer:1 adnan:1 menderes:1 ten:1 return:1 coup:1 etat:1 allegedly:1 conspire:1 although:1 imprison:1 majority:1 form:1 coalition:1 süleyman:1 demirel:1 race:1 bülent:1 ecevit:1 standard:1 time:1 highly:1 educate:1 man:1 speak:1 arabic:1 english:1 french:1 addition:1 native:1 die:1 inter:1 next:1 mausoleum:1 massive:1 construct:1 legacy:1 university:1 stadium:1 istanbul:1 home:2 beşiktaş:1 football:1 club:1 medium:1 image:2 ogg:1 sound:1 file:1 message:2 november:1 text:2 pdf:1 see:1 pembe:1 köşk:1 çankaya:1 köşkü:1 reference:1  |@bigram smet_nönü:5 september_december:1 turkish_army:3 army_general:1 prime_minister:5 second_president:2 president_republic:2 republic_turkey:3 bestow_upon:1 elect_president:1 early_life:1 life_bear:1 family_originally:1 academic_publisher:1 turkish_war:4 family_move:1 change_name:1 decide_take:1 great_battle:1 first_battle:1 second_battle:1 play_important:1 important_role:1 war_independence:3 former_deputy:1 deputy_prime:1 social_democracy:1 social_democratic:2 people_party:1 military_career:1 military_academy:1 receive_first:1 ottoman_army:1 army_join:1 win_first:1 military_victory:1 two_major:1 ottoman_empire:1 serve_military:1 military_officer:1 balkan_war:1 world_war:4 war_serve:1 eastern_front:1 later_appoint:2 appoint_commander:2 western_front:1 work_together:1 lose_battle:2 last_day:1 promote_brigadier:1 brigadier_general:1 also_member:1 grand_national:1 national_assembly:1 chief_staff:1 appoint_chief:1 become_famous:1 peace_conference:1 last_long:1 british_foreign:2 foreign_secretary:2 secretary_lord:1 say_thing:1 political_career:1 later_serve:1 serve_prime:1 put_place:1 every_major:1 attempted_assassination:1 government_intervention:1 economic_crisis:1 five_year:1 year_plan:1 soviet_union:2 take_much:1 private_property:1 government_control:1 economic_system:1 founder_first:1 minister_national:1 official_title:1 war_ii:2 ii_break:1 first_year:1 franz_von:1 winston_churchill:2 later_meet:1 franklin_roosevelt:1 would_serve:1 middle_east:1 army_air:1 air_force:1 throughout_war:1 open_new:1 still_believe:1 know_well:1 country_suffer:1 enter_war:2 war_fight:1 united_state:1 state_united:1 united_kingdom:1 would_stand:1 soviet_invasion:2 foreign_relation:1 economic_growth:1 http_www:1 may_explain:1 could_win:1 strong_opposition:1 within_party:1 previously_mention:1 election_hold:2 hold_first:1 first_place:1 world_history:1 leader_opposition:2 opposition_party:1 country_world:1 democratic_state:1 election_vote:1 secret_police:1 behind_closed:1 closed_door:1 party_official:1 party_lose:1 lose_first:1 free_election:1 transfer_power:1 democratic_party:1 ten_year:1 return_power:1 military_coup:1 coup_etat:1 form_coalition:1 coalition_government:1 general_election:1 party_leadership:1 standard_time:1 english_french:1 french_german:1 football_club:1 see_also:1 '"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wiki_texts['vw_text'].head().values[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "wiki_texts[['id','raw_text', 'vw_text']].to_csv('./wiki_data.csv')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}