Topic Classfication - Theory.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. First, automatically identify the topics withing a corpus of textual data by using unsupervised machine learning.\n",
    "2. Apply a supervised classifcation algorithm to assign topic labels to each textual document by using the result of the previous step as target labels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Steps:\n",
    "1. Import Texts [Retrieve Data from Database]\n",
    "2. Clean Texts [Remove Fixed Patterns]\n",
    "3. Translate [If needed]\n",
    "4. Preprocess Texts [(a) Tokenization (b) Stop Words Removal]\n",
    "5. Bigrams [Identify word pairs within data]\n",
    "6. Document Term Matrix [(a) Bag of Words (b) Vectorization]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Remove Parts of the text repeated. \n",
    "2. Preprocess texts using tokenization, lemmatisation, stop-words and digit removal. \n",
    "3. Add n-grams to dataset to guide topic model with the assumption that short sequences of words treated as single entities, or tokens usually contain useful information to identify the topics of a sentence. Only bigrams turned out to be meaningful in this context - indeed they significantly improved the topic model performance. \n",
    "4. Use count vectorization to transform data into a numeric term-document-matrix, having documents as rows, single tokens as columns and the corresponding frequencies as values (frequency of the selected token in a given chat). The Bag of Words approach does not take into account words order, which should not play a crucial role in topic identification. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LDA - Latent Dirichet Allocation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Takes document-term-matrix as input to infer probabilistic distribution on:\n",
    "    1. A set of latent (i.e. unknown) topics across the documents. \n",
    "    2. The words in corpus vocabulary (set of all words used in dataset) by looking at topics in document in which the words are contained and other topic assignments for that particular word across corpus)\n",
    "\n",
    "LDA outputs the k topics (where k is given to model as parameter) in form of high-dimensional vectors where each component represents the weight for a particular word in vocabulary . By looking at terms with heighest weights it's possible to manually give a name to the k topics, improving human interpretability of output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LDA also provides a topic distribution for each document in dataset as a sparse vector (few components with high weights, all the rest with 0 weight) making it easier to interpret high-dimensional topic vectors and extract relevant topics for each text. \n",
    "\n",
    "So, the LDA model provides topic weights for each document it is trained on. Now, the transition to a supervised approach becomes straight forward. The vector component within the heighest weight is picked and the corresponding topic is used as target label for given document."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Generative Probabilistic Model of a collection of composites made up of parts. It uses NLP and topic modelling. \n",
    "\n",
    "In terms of topic modelling, the composites are document and the parts are words or phrases .\n",
    "\n",
    "LDA consists of 2 tables (matrices)\n",
    "1. First table describes probability or chance of selecting a particular part when sampling of a particular topic (category).\n",
    "2. Second table describes chance of selecting a particular topic when sampling a particular document or composite."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LDA algorithm assumes composites were generated like so:\n",
    "1. Pick your unique set of parts. \n",
    "2. Pick how many composites you want. \n",
    "3. Pick how many parts you want per composite (sample from a Poisson Distribution)\n",
    "4. Pick how may categories/topics you want. \n",
    "5. Pick a number between non-zero and positive infinity and call it alpha. \n",
    "6. Pick a number between non-zero and positive infinity and call it beta. \n",
    "7. Build Parts VS Topics table. For each column, draw a sample from Dirichlet Distribution (which is a distribution of distributions) using beta as input. Each sample will fill out each column in table, sum to one, and give probability of each part per topic (column).\n",
    "8. Build Composites VS Topics table. For each column, draw a sample from Dirichlet Distribution using alpha as input. Each sample will fill out each row in the table, sum to one, and give probability of each topic (column) per composite. \n",
    "9. Build actual composites. For each composite:\n",
    "    - Look up its row in Composites VS Topic table.\n",
    "    - Sample a topic based on probabilities in row. \n",
    "    - Go to Parts VS Topics table. \n",
    "    - Look up topics sampled. \n",
    "    - Sample a part based on probabilitiesin column.\n",
    "    - Repeat from step 2 until you have reached how many parts this composite was set to have. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dirichlet Distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Dirichlet Distribution takes a number (called alpha) for eac h topic/category. \n",
    "\n",
    "At low alpha values (less than one), most of the topic distribution samples are in corners (near the topics). For really low alpha values, it is likely you will end up sampling (1,0,0) or (0,1,0) or (0,0,1). This would mean that a document would only ever have one topic. This is the case when there are only 3 possible topics. \n",
    "\n",
    "At alpha valu equal to one, any space in surface of triangle is uniformly distributed. You could equally likely end up with a sample favoring only one topic, a sample that gives an even mixture of all topics or something in between. \n",
    "\n",
    "For alpha values greate than one, samples start to congregate in center of triangle. This means that as alpha gets bigger, samples will more likely be uniform, represent an even mixture of topics.\n",
    "\n",
    "The alpha controls the mixture of topics for any given document. Turn it down and documents will likely have less of a mixture of topics. Turn it up and documents will have more of a mixture of topics. \n",
    "\n",
    "The beta hyperparameter controls distribution of words per topic. Turn it down and topics will likely have less words. Turn it up and topics will likely have more words.\n",
    "\n",
    "Ideally, one composite should be made up of only a few topics and one parts to belong to only some of the topics. For this alpha and beta should be set to below 1. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why LDA is better?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you view number of topics as number of clusters and probabilities as proportion of cluster membership then using LDA is a way of soft clustering your composites and parts. \n",
    "\n",
    "In contrast, K-Means where each entity can only belong to one cluster (hard-clustering). LDA allows \"fuzzy\" membership. This provides a more nuanced way of recommending similar items finding duplicates or discovering user profiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}