evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparison of Word2Vec, GloVe and FastText for measuring semantic similarity between pairs of sentences\n",
    "## Pablo Valdunciel \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the *KeyedVectors* model from the [*gesim*](https://radimrehurek.com/gensim/index.html) library to load the pre-trained vectors/embeddings of the different models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import KeyedVectors "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the pre-trained vectors with Word2Vec, GloVe and FastText models. The pre-trained vectors of each model are:\n",
    "\n",
    "|  Model | Source | Vocabulary size | Vector dimension | Language |\n",
    "|:--------:|:-----------------------:|:-----------:|:-------------------------:|:------:|\n",
    "| **Word2Vec** |     [GoogleNews-vectors-negative300.bin.gz](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit)                      |   3 million          |            300            | EN |\n",
    "|   **GloVe**  |[Common  Crawl  (840B  tokens,2.2M vocab, cased, 300d vectors)](http://nlp.stanford.edu/data/glove.840B.300d.zip)|     2,2 million        |            300            | EN |\n",
    "| **FastText** |  [rawl-300d-2M.vec.zip:  2  million  word  vectors  trained  on  Common  Crawl  (600Btokens)](https://dl.fbaipublicfiles.com/fasttext/vectors-english/crawl-300d-2M-subword.zip)                       |  2 million           |            300            | EN |\n",
    "\n",
    "In the case of GloVe vectors, the function [*glove2word2vec*](https://radimrehurek.com/gensim/scripts/glove2word2vec.html) from the *gensim* library has been used to convert the file to Word2Vec format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH_WORD2VEC = './data/embedding/word2vec/GoogleNews-vectors-negative300.bin'\n",
    "PATH_GLOVE = './data/embedding/glove/glove.840B.300d.w2v.txt'\n",
    "PATH_FASTTEXT = './data/embedding/fasttext/crawl-300d-2M.vec'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the three vector sets (this might take a few minutes to complete)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2vec = KeyedVectors.load_word2vec_format(PATH_WORD2VEC, binary=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "glove = KeyedVectors.load_word2vec_format(PATH_GLOVE, binary=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "fasttext = KeyedVectors.load_word2vec_format(PATH_FASTTEXT, binary=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the functions of the *load.py* module, we load the **test** sets of the STS12, STS13, STS14, STS15, STS16 and SICK-R tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Corpus | # of instances|                                        Source                                         |\n",
    "|:------:|:----------:|:------------------------------------------------------------------------------------------:|\n",
    "|  STS12 | 3108 |                      [SemEval 2012 - STS](https://www.cs.york.ac.uk/semeval-2012/task6/)                       |\n",
    "|  STS13 | 1500 |                               [SemEval 2013 - STS](http://ixa2.si.ehu.es/sts/)                                |\n",
    "|  STS14 | 3750 |               [SemEval 2014 - STS](http://alt.qcri.org/semeval2014/task10/index.php?id=sts-en)                |\n",
    "|  STS15 | 3000| [SemEval 2015 - STS](http://alt.qcri.org/semeval2015/task2/index.php?id=semantic-textual-similarity-for-english) |\n",
    "|  STS16 |  1186|                         [SemEval 2016 - STS](http://alt.qcri.org/semeval2016/task1/)                           |\n",
    "| SICK-R | 9927 |[SICK Relatedness](https://zenodo.org/record/2787612#.Xlf59XVKhhE) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All functions in the *load.py* module perform sentence preprocessing according to the parameters you specify. By default, sentence preprocessing is based on *tokenisation/segmentation* of sentences. In addition, there are the following possibilities:\n",
    "\n",
    "- `lowercase`: make all words lowercase.\n",
    "- `stop_words`: remove words that have almost no semantic meaning, such as determiners, preprositions, etc.\n",
    "- `punctuation`: remove punctuation symbols.\n",
    "- `only_ascci`: remove words that do not consist of ASCII characters.\n",
    "- `lemmatization`: replace words with their lexemes.\n",
    "\n",
    "Text preprocessing is implemented in the *preprocess* function in the *utils.py* module. The *preprocess* function makes use of the [spaCy](https://spacy.io/) library to perform the preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from load import load_sts_12, load_sts_13, load_sts_14, load_sts_15, load_sts_16, load_SICK\n",
    "from load import load_frequencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, only the *tokenisation/segmentation* of the sentences and a conversion to lowercase is performed to achieve uniformity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH_STS = './data/datasets/STS'\n",
    "PATH_SICK = './data/datasets/SICK'\n",
    "PATH_FREQUENCIES = './data/frequencies.tsv'\n",
    "PREPROCESSING =  {'lowercase':  True, \n",
    "                  'stop_words': False, \n",
    "                  'punctuation': False, \n",
    "                  'only_ascii': False, \n",
    "                  'lemmatization': False\n",
    "                 }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load also the frequencies of the words in the corpus in order to be able to apply the SIF average."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "freqs = load_frequencies(PATH_FREQUENCIES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "***** TASK: STS12 *****\n",
      "\n",
      "Preprocessing -MSRpar-\n",
      "-MSRpar- preprocessed correctly\n",
      "Preprocessing -MSRvid-\n",
      "-MSRvid- preprocessed correctly\n",
      "Preprocessing -SMTeuroparl-\n",
      "-SMTeuroparl- preprocessed correctly\n",
      "Preprocessing -surprise.OnWN-\n",
      "-surprise.OnWN- preprocessed correctly\n",
      "Preprocessing -surprise.SMTnews-\n",
      "-surprise.SMTnews- preprocessed correctly\n",
      "\n",
      "\n",
      "***** TASK: STS13 (-SMT) ***\n",
      "\n",
      "Preprocessing -FNWN-\n",
      "-FNWN- preprocessed correctly\n",
      "Preprocessing -headlines-\n",
      "-headlines- preprocessed correctly\n",
      "Preprocessing -OnWN-\n",
      "-OnWN- preprocessed correctly\n",
      "\n",
      "\n",
      "***** TASK: STS14 *****\n",
      "\n",
      "Preprocessing -deft-forum-\n",
      "-deft-forum- preprocessed correctly\n",
      "Preprocessing -deft-news-\n",
      "-deft-news- preprocessed correctly\n",
      "Preprocessing -headlines-\n",
      "-headlines- preprocessed correctly\n",
      "Preprocessing -images-\n",
      "-images- preprocessed correctly\n",
      "Preprocessing -OnWN-\n",
      "-OnWN- preprocessed correctly\n",
      "Preprocessing -tweet-news-\n",
      "-tweet-news- preprocessed correctly\n",
      "\n",
      "\n",
      "***** TASK: STS15 *****\n",
      "\n",
      "Preprocessing -answers-forums-\n",
      "-answers-forums- preprocessed correctly\n",
      "Preprocessing -answers-students-\n",
      "-answers-students- preprocessed correctly\n",
      "Preprocessing -belief-\n",
      "-belief- preprocessed correctly\n",
      "Preprocessing -headlines-\n",
      "-headlines- preprocessed correctly\n",
      "Preprocessing -images-\n",
      "-images- preprocessed correctly\n",
      "\n",
      "\n",
      "***** TASK: STS16 *****\n",
      "\n",
      "Preprocessing -answer-answer-\n",
      "-answer-answer- preprocessed correctly\n",
      "Preprocessing -headlines-\n",
      "-headlines- preprocessed correctly\n",
      "Preprocessing -plagiarism-\n",
      "-plagiarism- preprocessed correctly\n",
      "Preprocessing -postediting-\n",
      "-postediting- preprocessed correctly\n",
      "Preprocessing -question-question-\n",
      "-question-question- preprocessed correctly\n",
      "\n",
      "\n",
      "***** Task: SICK-Relatedness*****\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sts12 = load_sts_12(PATH_STS, PREPROCESSING, verbose=True)\n",
    "sts13 = load_sts_13(PATH_STS, PREPROCESSING, verbose=True)\n",
    "sts14 = load_sts_14(PATH_STS, PREPROCESSING, verbose=True)\n",
    "sts15 = load_sts_15(PATH_STS, PREPROCESSING, verbose=True)\n",
    "sts16 = load_sts_16(PATH_STS, PREPROCESSING, verbose=True)\n",
    "sick = load_SICK(PATH_SICK, PREPROCESSING, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The methods for calculating the semantic similarity between two sentences are found in the *methods.py* module and are as follows:\n",
    "\n",
    "- `avg_cosine`: the vector of a sentence is obtained by averaging the vectors of the words in that sentence. The similarity between two vectors is calculated using cosine similarity.\n",
    "- `sif_cosine`: the vector of a sentence is obtained by taking a weighted average (*Smooth Inverse Frequency*, SIF) of the vectors of the words in that sentence. The similarity between two vectors is calculated using cosine similarity.\n",
    "- `wmd`: the similarity between two sentences is calculated as the inverse of the *Word Mover's Distance* between them. The *KeyedVectors* model of *gensim* incorporates the calculation of this distance ([see method](https://radimrehurek.com/gensim/models/keyedvectors.html#gensim.models.keyedvectors.FastTextKeyedVectors.wmdistance)). This distance is not bounded and will take the value *infinite* when none of the words of any of the two sentences is present in the model, so it has been bounded in the interval [0, 100]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "from methods import avg_cosine, wmd, sif_cosine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "METHODS = [\n",
    "    (\"W2V + AVG\", partial(avg_cosine, model=word2vec)),\n",
    "    (\"W2V + SIF\", partial(sif_cosine, model=word2vec, frequencies=freqs, a=0.001)),\n",
    "    (\"W2V + WMD\", partial(wmd, model=word2vec)),   \n",
    "    (\"GLOVE + AVG\", partial(avg_cosine, model=glove)),\n",
    "    (\"GLOVE + SIF\", partial(sif_cosine, model=glove, frequencies=freqs, a=0.001)),\n",
    "    (\"GLOVE + WMD\", partial(wmd, model=glove)),    \n",
    "    (\"FT + AVG\", partial(avg_cosine, model=fasttext)),\n",
    "    (\"FT + SIF\", partial(sif_cosine, model=fasttext, frequencies=freqs, a=0.001)),\n",
    "    (\"FT + WMD\", partial(wmd, model=fasttext))    \n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `utils` module contains the `evaluate` function that calculates the Pearson and Spearman correlation coefficients between the *Gold Standard* scores of a task and the similarity measures provided by one or more methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import evaluate\n",
    "from scipy.stats import rankdata, chi2\n",
    "from Orange.evaluation import compute_CD, graph_ranks\n",
    "import numpy as np\n",
    "import pprint\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "N = 6  # Number of corpus\n",
    "K = 9  # Number of methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "sts12_pearson, sts12_spearman = evaluate(sts12, METHODS)\n",
    "sts13_pearson, sts13_spearman = evaluate(sts13, METHODS)\n",
    "sts14_pearson, sts14_spearman = evaluate(sts14, METHODS)\n",
    "sts15_pearson, sts15_spearman = evaluate(sts15, METHODS)\n",
    "sts16_pearson, sts16_spearman = evaluate(sts16, METHODS)\n",
    "sick_pearson, sick_spearman = evaluate(sick, METHODS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pearson correlation coefficients\n",
    "\n",
    "In the following table we can observe the Pearson correlation coefficients obtained by each of the methods on each corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pearson_corr = np.array([ np.array(list(sts12_pearson.values())), \n",
    "                          np.array(list(sts13_pearson.values())),\n",
    "                          np.array(list(sts14_pearson.values())),\n",
    "                          np.array(list(sts15_pearson.values())),\n",
    "                          np.array(list(sts16_pearson.values())),\n",
    "                          np.array(list(sick_pearson.values()))\n",
    "                        ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Corpus | W2V  +  AVG | W2V  +  SIF | W2V  +  WMD | GLOVE  +  AVG | GLOVE + SIF | GLOVE  +  WMD | FT  +  AVG | FT  +  SIF | FT  +  WMD |\n",
    "|:---------------:|:-----------:|:-----------:|:-----------:|:-------------:|:-----------:|:-------------:|:----------:|:----------:|:----------:|\n",
    "|      STS12      |    0,5577   |    0,5670   |    0,4735   |     0,5503    |    0,5887   |     0,5511    |   0,6006   |   0,6213   |   0,5535   |\n",
    "|      STS13      |    0,6402   |    0,7227   |    0,5212   |     0,5431    |    0,7004   |     0,4865    |   0,6396   |   0,7431   |   0,5043   |\n",
    "|      STS14      |    0,6868   |    0,7279   |    0,6122   |     0,5625    |    0,7069   |     0,5803    |   0,6664   |   0,7356   |   0,5921   |\n",
    "|      STS15      |    0,7049   |    0,7490   |    0,6839   |     0,6006    |    0,7319   |     0,6796    |   0,6999   |   0,7613   |   0,6888   |\n",
    "|      STS16      |    0,6391   |    0,7191   |    0,6408   |     0,5025    |    0,6847   |     0,6133    |   0,6326   |   0,7315   |   0,6312   |\n",
    "|      SICK-R     |    0,7012   |    0,7318   |    0,6125   |     0,6517    |    0,7215   |     0,5995    |   0,6980   |   0,7400   |   0,6045   |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rankings \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the average rankin for each corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "ranks = np.array([np.array(rankdata((-1)*pearson_corr[task])) for task in range(pearson_corr.shape[0])])\n",
    "avg_ranks = np.average(ranks, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Corpus \\ Método | W2V  +  AVG | W2V  +  SIF | W2V  +  WMD | GLOVE  +  AVG | GLOVE + SIF | GLOVE  +  WMD | FT  +  AVG | FT  +  SIF | FT  +  WMD |\n",
    "|:---------------:|:-----------:|:-----------:|:-----------:|:-------------:|:-----------:|:-------------:|:----------:|:----------:|:----------:|\n",
    "|      STS12      |      5      |      4      |      9      |       8       |      3      |       7       |      2     |      1     |      6     |\n",
    "|      STS13      |      4      |      2      |      7      |       6       |      3      |       9       |      5     |      1     |      8     |\n",
    "|      STS14      |      4      |      2      |      6      |       9       |      3      |       8       |      5     |      1     |      7     |\n",
    "|      STS15      |      4      |      2      |      7      |       9       |      3      |       8       |      5     |      1     |      6     |\n",
    "|      STS16      |      5      |      2      |      4      |       9       |      3      |       8       |      6     |      1     |      7     |\n",
    "|      SICK-R     |      4      |      2      |      7      |       6       |      3      |       9       |      5     |      1     |      8     |\n",
    "|       |       |      |      |          |        |           |       |         |       |\n",
    "|     Average    |     4,33    |     2,33    |     6,67    |      7,83     |     3,00    |      8,17     |    4,67    |    1,00    |    7,00    |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Friedman test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hypothesis test:\n",
    "\n",
    "- H<sub>0</sub>: all the metods are equivalent\n",
    "- H<sub>1</sub>: not all the methods are equivalent\n",
    "\n",
    "To solve this hypothesis test, the Friedman test on the average rankings is performed. Even without performing the test, these average rankings already provide a good comparison of the different methods.\n",
    "\n",
    "\n",
    "    \n",
    "| Position | Average ranking |    Method   |\n",
    "|:--------:|:----------------:|:-----------:|\n",
    "|     1    |         1        |   FT + SIF  |\n",
    "|     2    |       2,33       |  W2V + SIF  |\n",
    "|     3    |         3        | GLOVE + SIF |\n",
    "|     4    |       4,33       |  W2V + AVG  |\n",
    "|     5    |       4,67       |   FT + AVG  |\n",
    "|     6    |       6,67       |  W2V + WMD  |\n",
    "|     7    |         7        |   FT + WMD  |\n",
    "|     8    |       7,83       | GLOVE + AVG |\n",
    "|     9    |       8,17       | GLOVE + WMD |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "p-value = 0.000028\n"
     ]
    }
   ],
   "source": [
    "sqr_avg_ranks = np.array(list(map(lambda r: r**2, avg_ranks)))\n",
    "friedman = (12*N) / (K*(K+1)) * (sum(sqr_avg_ranks) - (K*(K+1)**2/4))\n",
    "p_value = 1 - chi2.cdf(friedman, K-1)\n",
    "\n",
    "print(\"p-value = {:4f}\".format(p_value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The p-value obtained for the Friedman test is **0.000028**, so the null hypothesis, H<sub>0</sub>, on the equivalence of the methods is rejected at the usual confidence levels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Nemenyi test\n",
    "\n",
    "Having rejected the null hypothesis H<sub>0</sub> of method equivalence, the Nemenyi *post-hoc* test is performed to determine between which pairs of methods there are significant differences. Considering a confidence level of 95%, $\\alpha = 0.05$, and knowing that the number of methods is *K=9* and the number of data sets is *N=6*, the calculated critical Nemenyi distance is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Critical Nemenyi distance (alpha = 0.05) =  4.904265743437033\n"
     ]
    }
   ],
   "source": [
    "cd_nemenyi = compute_CD(avg_ranks, N, alpha='0.05', test='nemenyi') \n",
    "print(\"Nemenyi critical distance (alpha = 0.05) = \", cd_nemenyi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We plot the significant differences based on the critical distance obtained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAq0AAAC3CAYAAAA4jmd/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de1RV5b7/8Tcuw1C8sVNSQQkSRFiwBEJZaoldLONsAq/kTtI8ptvqiNsz9rYa51fjnG6OVOwMR+WpIVstlrFLLLyVeryxUEpEidKT2tpmmKVFoago8fuDwRziBUUuawKf1xiOwZrzWXN+n7lW9nHyzOfxqKqqqkJERERExMTaubsAEREREZHrUWgVEREREdNTaBURERER01NoFRERERHTU2gVEREREdNTaBURERER01NoFRERERHTU2gVEREREdNTaBURERER01NoFRERERHTU2gVEREREdNTaBVpAX744QcmTpxIUFAQ0dHRjB49mv/7v//Dy8uLQYMGERoaSmxsLBkZGe4uVUREpEm0d3cBIlK3qqoqkpKSSE1NxeFwALBv3z5OnDhBUFAQe/fuBeDIkSMkJydTVVXFlClT3FmyiIhIo9OdVhGT+9///V9uueUWZsyYYWyLjIzE39+/VrvAwEAWLlzIG2+80dwlioiINDmFVhGT+/LLL4mOjr6htlFRURw4cKCJKxIREWl+Cq0irUhVVZW7SxAREWkSCq0iJhcWFsaePXtuqO3evXsJDQ1t4opERESan0KriMmNHDmS8+fPs3TpUmPb/v37+e6772q1c7lczJ07l6effrq5SxQREWlyHlX6faKI6ZWUlDB79mz27NnDrbfeSkBAAOnp6URERDBgwADOnTtH586d+fOf/8zjjz/u7nJFREQanUKriIiIiJiehgc0genTp7u7BBGRRqO/00TEDBRam0BJSYm7SxARaTT6O01EzEChVURERERMT6FVRERERExPD2I1gb59+xIREeHuMkREGsX+/fs5evSou8sQkTauvbsLaI0iIiLIyclxdxkiIo0iISHB3SWIiGh4gIiIiIiYn0JrE+jdu7e7SxARaTT6O01EzEBjWkVERETE9HSnVURERERMT6FVRERERExPoVVERERETE+hVURERERMT6FVRERERExPoVVERERETE+hVURERERMT6G1EU2dOpWePXsSHh7u7lKa3XfffUd8fDwDBw4kLCyMxYsXu7ukZnPu3DliY2OJjIwkLCyM//f//p+7S2p2lZWVDBo0qM0t9xkQEIDVasVmsxETE+PucppVaWkpY8eOZcCAAYSGhpKXl+fukprFwYMHsdlsxp8uXbqQnp7u7rKazaJFiwgLCyM8PJyUlBTOnTvn7pKazeLFiwkPDycsLKxNfeZmosUFGtH27dvx9vZm8uTJfPnll+4up1kdP36c48ePExUVRVlZGdHR0WRnZzNw4EB3l9bkqqqqOHPmDN7e3ly4cIFhw4axePFihgwZ4u7Sms3ChQv54osv+O2338jJyXF3Oc0mICCAL774gttuu83dpTS71NRUhg8fzrRp06ioqKC8vJxu3bq5u6xmVVlZSZ8+fdi9ezf9+vVzdzlN7vvvv2fYsGF89dVXeHl5MX78eEaPHs3jjz/u7tKa3JdffsnEiRPJz8/H09OTBx98kLfeeos777zT3aW1KbrT2ojuvvtufHx83F2GW/Tq1YuoqCgAOnfuTGhoKN9//72bq2oeHh4eeHt7A3DhwgUuXLiAh4eHm6tqPseOHWPt2rVMmzbN3aVIM/n111/Zvn07TzzxBACenp5tLrACbN68maCgoDYRWGtcvHiRs2fPcvHiRcrLy9vMEr9ff/01gwcPpmPHjrRv35577rmHjz76yN1ltTkKrdLoXC4Xe/fuZfDgwe4updlUVlZis9no2bMn999/f5vq++zZs5k/fz7t2rW9v048PDx44IEHiI6OZunSpe4up9l8++239OjRgylTpjBo0CCmTZvGmTNn3F1Ws3M4HKSkpLi7jGbTp08f5s6dS9++fenVqxddu3blgQcecHdZzSI8PJwdO3Zw6tQpysvLWbduHd999527y2pz2t7/ZaRJnT59mjFjxpCenk6XLl3cXU6zsVgsFBYWcuzYMfLz89vM8JCcnBx69uxJdHS0u0txi507d1JQUMD69etZsmQJ27dvd3dJzeLixYsUFBQwc+ZM9u7dS6dOnXj11VfdXVazqqio4OOPP2bcuHHuLqXZ/PLLL6xZs4Zvv/2WkpISzpw5w8qVK91dVrMIDQ3lr3/9Kw888AAPPvggNpsNi8Xi7rLaHIVWaTQXLlxgzJgxTJo0ieTkZHeX4xbdunUjPj6eDRs2uLuUZpGbm8vHH39MQEAAEydOZMuWLfzpT39yd1nNpk+fPgD07NmTpKQk8vPz3VxR8/Dz88PPz8/4jcLYsWMpKChwc1XNa/369URFReHr6+vuUprNpk2buOOOO+jRowe33HILycnJOJ1Od5fVbJ544gn27NnD9u3b6d69O8HBwe4uqc1RaJVGUVVVxRNPPEFoaChz5sxxdznN6qeffqK0tBSAs2fP8tlnnzFgwAA3V9U8XnnlFY4dO4bL5cLhcDBy5Mg2c+flzJkzlJWVGT9/+umnbWbmkNtvvx1/f38OHjwIVI/tbAsPXV4qMzOzTQ0NAOjbty+7du2ivLycqqoqNm/eTGhoqLvLajY//vgjAEePHuWjjz7i0UcfdXNFbU97dxfQmqSkpLB161ZOnjyJn58fL774ovGgQmuXm5vLihUrjOl/AF5++WVGjx7t5sqa3vHjx0lNTaWyspLff/+d8ePHt7mpn9qiEydOkJSUBFT/uvzRRx/lwQcfdHNVzee///u/mTRpEhUVFQQGBrJs2TJ3l9Rszpw5w2effcbbb7/t7lKa1eDBgxk7dixRUVG0b9+eQYMGMX36dHeX1WzGjBnDqVOnuOWWW1iyZEmbfPjQ3TTllYiIiIiYnoYHiIiIiIjpKbSKiIiIiOkptIqIiIiI6Sm0ioiIiIjpKbQ2gbb0NOXl1Pe2SX1vm9T3tkl9F3dRaG0CJSUl7i7BbdT3tkl9b5vU97ZJfRd3UWhthdryvwTV97ZJfW+b1Pe2qS33va3TPK1NoFu3bgwbNsxt59+/fz8RERGt5tz1OaY7+75161ZGjBjhlnODPvcbaevuOptCa/vcm+L8TVFna/zvvSmuZ2v7b+7w4cN8/fXXbjm3aEWsJjFs2DBycnLcXUarkZCQ0CKuZ0ups6Voiuupz6ht0nfpxug6idlpeICIiIiImJ5Cq4iIiIiYnkKriIiIiJieQquIiIiImJ5Cq4iIiIiYnkKriIiIiJieQquIiIiImJ5Cq4iIiIiYnkKriIiIiJieQquIiIiImJ5Cq4iIiIiYnkKriIiIiJhee3cX0BrMnj2bwsJC43VRUREjRoxwX0GtTEu5ni2lzpaiKa7n1Y5ps9lIT09v1POIiEjjU2htBIWFhWzbtq3WtstfS8O0lOvZUupsKZrieuozEhFpmRRaG4HNZqv1uqioCKvV6qZqWp+Wcj1bSp0tRVNcz6sd8/L/fkVExJwUWhvB5b9aTEhIICcnx03VtD4t5Xq2lDpbiqa4nvqMRERaLj2IJSIiIiKmp9AqIiIiIqan0CoiIiIipqcxrWI6LXUKsZZSZ0vRXFNeSeun6dNEWgeFVjGdljyFWEups6XQlFfSWPRdEmn5FFrFdFrqFGItpc6WormmvJLWT9OnibQOCq1iOi11CrGWUmdLoSmvpLHouyTSOuhBLBERERExPYVWERERETE9hVYRERERMT2NaRW5SS11aq6WQlNeSWPRd+nGXN4nTeElZqPQKnKTWvLUXC2FpimSxqLv0o1pjX2S1kOhVeQmtdSpuVoKTXkljUXfpRtzeZ80hZeYjUKryE1qqVNztRSapkgai75LN6Y19klaFz2IJSIiIiKmp9AqIiIiIqan0CoiIiIipqcxrSJiCs0xhVhrnKZIrk/fpRvTHNdJ02hJQyi0iogpNNcUYprSp23Sd+nG6DqJmSm0iogpNMcUYq1xmiK5Pn2XbkxzXCdNoyUNodAqIqbQHFOIaUqftknfpRuj6yRmpwexRERERMT0FFpFRERExPQUWkVERETE9BRaRURERMT0FFpFRERExPQUWkVERETE9BRaRURERMT03BpaLRYLNpvN+PP2228bP3t7exMSEoLNZmPy5MkNOs+uXbsYPHgwNpuN0NBQXnjhBQAyMjJ46qmnAHjhhRfo06ePcf6//e1vDe2eiIiIiOHEiRM8+uijBAYGEh0dTVxcHKtXrwZg69atJCQkXPGeiooKZs+ezZ133kn//v1JTEzk2LFjAMTHx7Nx48Za7dPT05k5cyYulwsvL69aOWv58uUNqn/QoEHGctsXL17E29ublStXGvujo6MpKCggIyMDDw8PNm3aZOzLzs7Gw8ODf/zjHwCMGDGCkJAQIiIiGDBgAE899RSlpaV1nt+tiwt4eXnVWmsc4MknnwSqO/P6668TExNT5zEyMjJwuVxGEL2a1NRUPvjgAyIjI6msrOTgwYNXbZeWlsbcuXPr1wkRERGR66iqquKRRx4hNTWV999/H4B//vOffPzxx3W+79lnn6WsrIyDBw9isVhYtmwZycnJ7N69m5SUFBwOB6NGjTLaOxwO5s+fD0BQUNAVOasuAQEBuFyua+4fOnQoTqcTm83Gvn37CA4Oxul08qc//YkzZ85w+PBhIiMj2b9/P1arFYfDwX333QdAZmYmkZGRtY733nvvERMTQ0VFBfPmzSMxMbHOZX/bxPCAH3/8kV69egHVd3cHDhzo5opERESkLdmyZQuenp7MmDHD2NavXz+efvrpa76nvLycZcuWsWjRIiwWCwBTpkyhQ4cObNmyhbFjx7J27VoqKioAcLlclJSUMHz48Cbpg91ux+l0AuB0OpkxY4YRivPz84mOjjbqHD58OPn5+Vy4cIHTp09z6NChay7j6+npyfz58zl69Cj79u275vndGlrPnj1r3LJOSkpqsvOkpaUREhJCUlISb7/9NufOnbtqu0WLFhn1XH67XURERORmFRcXExUVVa/3HDp0iL59+9KlS5da22NiYiguLsbHx4fY2FjWr18PVN9lHT9+PB4eHgAcPny41vCAHTt2NKgPNXdaoTq03n333XTo0IGysjKcTid2u91o6+HhwX333cfGjRtZs2YNf/zjH+s8tsViITIykgMHDlyzjemGB9yIU6dOce+99wLw888/U1FRQXZ2NgArVqzAarXWav8f//EfTJo0iU8//ZT333+fzMxMtm7desVxNTxAREREmsOsWbPYuXMnnp6efP755zd9nJohAomJiTgcDt59911j340MD5g1axa5ubkAlJSUGHdDx40bx3PPPVerbb9+/aioqOCHH37gwIEDhISEcNddd7F7926cTucVd40nTpzIG2+8wa+//sqCBQt4+eWX66ylqqqqzv1uDa036w9/+IPxIdzImFao/uBmzpzJv/7rv9KjRw9OnTrVoBqmT59OSUnJVfd5eXk16NgiIiKtgZeX11UfLmqNevfuzdKlS6+5PywsjA8//NB4vWTJEk6ePFnnsztBQUEcPXqUsrIyOnfubGzfs2ePcV0TExNJS0ujoKCA8vJyoqOj61X3kiVLjJ8DAgKuG3LtdjtZWVn06tULDw8PhgwZQm5uLvn5+cTFxdVqGxsbS1FRER07diQ4OLjO41ZWVlJUVERoaOg127TI0Fpfa9euZfTo0Xh4ePDNN99gsVjo1q1bg45Z1xdTREREICsry90lmMbIkSN59tlnefPNN5k5cyZQPWa1Lp06dSI1NZU5c+bw1ltvYbFYWL58OeXl5YwcORIAb29v4uPjmTp1KikpKU3eD7vdTnp6Oo8//jgAcXFx/Pu//zu33347Xbt2vaL9q6++yq233lrnMS9cuMBzzz2Hv78/ERER12zXJkLrihUrSEtLo2PHjrRv35733nvPGCgsIiIi0tQ8PDzIzs4mLS2N+fPn06NHDzp16sRrr71mtNm8eTN+fn7G66ysLF555RXmzp1LcHAw7dq1Y8CAAaxevdoYtwrVQwSSkpJwOBy1zlkzprXG1KlTeeaZZxrUj6FDh5KWlmbcVe3VqxeVlZW1xrNe6qGHHrrmsSZNmkSHDh04f/489913H2vWrKnz3B5V1xtAIOJmCQkJ5OTkuLuM62opdbYUTXE99Rm1Tfou3ZjW2CdpXdrElFciIiIi0rIptIqIiIiI6Sm0ioiIiIjpKbSKiIiIiOkptIqIiIiI6Sm0ioiIiIjpNVpoTUtLIz093Xg9atQopk2bZrz+y1/+wsKFCyksLCQuLo6wsDAiIiJYtWoVAC+++CLz5s2rdczCwsI6V0a4EeXl5UyaNAmr1Up4eDjDhg3j9OnTQPWEvAAulwsvL69a6/NWVFQ06LwiIiIilzpx4gSPPvoogYGBREdHExcXx+rVqwHYunXrVVcPq6ioYPbs2dx5553079+fxMREjh07BkB8fDwbN26s1T49PZ2ZM2deNdssX768Ufoxe/Zs+vTpw++//w5U5yg/Pz/jdQ2bzcbu3bsBWLlyJREREYSFhREZGcm0adMoLS2t13kbLbQOHToUp9MJwO+//87JkycpLi429judTux2Ox07dmT58uUUFxezYcMGZs+eTWlpKSkpKUaAreFwOK67ukNAQECd+xcvXoyvry9FRUV8+eWXvPvuu9xyyy1XtKtZn7fmj6en5w32XERERKRuVVVVPPLII9x9990cOXKEPXv24HA4jAB6Lc8++yxlZWUcPHiQb775hkceeYTk5GSqqqpISUm5YkGBS7PT5dlm8uTJdZ7repkKqjPe6tWr8ff3Z9u2bcb7+vbty44dO4x2Bw4coKysjMGDB7NhwwYWLVrE+vXrKS4upqCgALvdzokTJ657vks1Wmi12+3k5eUBUFxcTHh4OJ07d+aXX37h/PnzfP3110RFRREcHEz//v2B6nV6e/bsyU8//URwcDDdu3c3EjnABx980OAlyY4fP06fPn2M1yEhIXTo0KFBxxQRERGpjy1btuDp6cmMGTOMbf369ePpp5++5nvKy8tZtmwZixYtMlbynDJlCh06dGDLli2MHTuWtWvXGr8ddrlclJSUMHz48Cbrx9atWwkLC2PmzJlkZmYa2y8P0A6Hg4kTJwLw0ksv8frrrxt5zGKxMHXqVEJCQup17kYLrb1796Z9+/YcPXoUp9NJXFwcgwcPJi8vjy+++AKr1XrF3cv8/HwqKioICgoCand4165d+Pj4GAH3Zk2dOpXXXnuNuLg4nn/+eb755purtqtZ6sxmszFr1qwGnVNERETkUsXFxURFRdXrPYcOHaJv37506dKl1vaYmBiKi4vx8fEhNjaW9evXA9VBcfz48cYSr5dmG5vNVutO6M3KzMw0lo1du3YtFy5cAGD8+PFkZ2dz8eJFAFatWmXceLyZvl9N+wYf4RJ2ux2n04nT6WTOnDl8//33OJ1OunbtytChQ2u1PX78OI899hh///vfadeuOjtPmDABu93OggUL6hwaMGvWLHJzcwEoKSkx1tUdN24czz33XK22NpuNI0eO8Omnn7Jp0ybuuusu8vLyrhgrW3ML/UZNnz6dkpKSG24vN8/Ly8vdJYiItHpeXl5XHVMpN6Z3794sXbr0htvPmjWLnTt34unpyeeff37T56254ZeYmIjD4eDdd9819t1ItqlPpqqoqGDdunUsXLiQzp07M3jwYDZu3EhCQgK+vr6Eh4ezefNmfH19ad++PeHh4Vecr6ioiMcee4yysjJefvllJkyYcMN9bdTQWjOutaioiPDwcPz9/VmwYAFdunRhypQpRrvffvuNhx9+mJdeeokhQ4YY2/39/bnjjjvYtm0bH374oTHc4HJLliwxfg4ICLjuB+Lt7U1ycjLJycm0a9eOdevWNfgBr/p8MUVERMwuKyvL3SW0amFhYXz44YfG6yVLlnDy5EliYmKu+Z6goCCOHj1KWVkZnTt3Nrbv2bPH+AdGYmIiaWlpFBQUUF5eTnR0dL3qqk+m2rhxI6WlpVitVqB6+MKl/9ipCdC+vr61bjyGhYVRUFBAfHw8VquVwsJCnnrqKc6ePVuvWht1yiu73U5OTg4+Pj5YLBZ8fHwoLS0lLy8Pu90OVKf0pKQkJk+ezNixY684RkpKCmlpaQQGBuLn59fgmnJzc/nll1+Mc3/11Vf069evwccVERERuVEjR47k3LlzvPnmm8a28vLyOt/TqVMnUlNTmTNnDpWVlQAsX76c8vJyRo4cCVTfmIuPj2fq1KkNfg7oejIzM3nnnXdwuVy4XC6+/fZbPvvsM6MfycnJrFu3jlWrVhnjWQHmzZvH3Llzaz10Vt/ACo0cWq1WKydPnqx199RqtdK1a1duu+02oPrhqu3bt5ORkWGMsbg01Y8bN47i4uJGu/CHDx/mnnvuwWq1MmjQIGJiYhgzZkyjHFtERETkRnh4eJCdnc22bdu44447iI2NJTU1lddee81os3nzZvz8/Iw/eXl5vPLKK9x6663Gg+xZWVmsXr3aGLcK1Tf89u3bd0V2unxM6xtvvHHT9ZeXl7NhwwYefvhhY1unTp0YNmwYn3zyCQDdunUjLi4OX19fAgMDjXajR4/mmWee4aGHHmLgwIHY7XYsFgujRo2qVw0eVVVVVTfdAxExJCQkkJOT4+4yWo2muJ76jNomfZdEWgetiCUiIiIipqfQKiIiIiKmp9AqIiIiIqan0CoiIiIipqfQKiIiIiKmd93QeuLECR599FECAwOJjo4mLi6O1atXA9Xrz15t9YyKigpmz57NnXfeSf/+/UlMTDTm5oqPj2fjxo212qenpzNz5kxcLhdeXl61pmdYvnx5gzpYXl7OpEmTsFqthIeHM2zYME6fPg1Uz20GXPW8Nev4ioiIiDQGi8VSK2u8/fbbxs/e3t6EhIRgs9mYPHnyTZ9j3759xqpWUD23qpeXl7HcalFREREREQCMGDGCvn37culEUo888sgV+WjQoEGEhoYSGxtLRkbGTdfWUHWuiFVVVcUjjzxCamoq77//PgD//Oc/+fjjj+s86LPPPktZWRkHDx7EYrGwbNkykpOT2b17t7FawqVzczkcDubPnw/UfznVgIAAXC7XNfcvXrwYX19fioqKADh48CC33HLLFe3qe14RERGR+vDy8roiazz55JNAdYB8/fXX61whCyAjIwOXy8ULL7xw1f1Wq7XWKlpOp5PQ0FD27t1LbGwsTqfTWPAJqudWzc3NZdiwYZSWlnL8+PFaxwsKCmLv3r0AHDlyhOTkZKqqqmqtdNpc6rzTumXLFjw9PZkxY4axrV+/fjz99NPXfE95eTnLli1j0aJFWCwWAKZMmUKHDh3YsmULY8eOZe3atcadTJfLRUlJCcOHD2+M/lzh+PHj9OnTx3gdEhJChw4dmuRcIiIiIu7Url07YmJi2L17N1C95OusWbNwOp0AOJ1Ohg4darSfOHEiDocDgI8++ojk5ORrHjswMJCFCxc2aJGChqgztBYXFxMVFVWvAx46dIi+ffvSpUuXWttjYmIoLi7Gx8eH2NhY1q9fD1TfZR0/fryxssPlqzfs2LGjXue/3NSpU3nttdeIi4vj+eef55tvvrlqu0vPO2vWrAadU0RERORyZ8+eNbJGUlJSk51n6NChOJ1Ozpw5Q7t27RgxYkSt0HrpndZ7772X7du3U1lZicPhYMKECXUeOyoqigMHDjRZ7XWpc3jA5WbNmsXOnTvx9PTk888/v+mT1gwRSExMxOFw8O677xr7buTX9LNmzSI3NxeAkpISY+zGuHHjeO6552q1tdlsHDlyhE8//ZRNmzZx1113kZeXR2hoaK12Gh4gIiIiTelqwwNuxKlTp7j33nsB+Pnnn6moqCA7OxuAFStWYLVaa7W32+0sWLCA4cOHc9dddxEUFMShQ4f46aefOH36NEFBQUZbi8XCsGHDcDgcnD17loCAgDprcedCqnWG1rCwMD788EPj9ZIlSzh58mSd4y2CgoJqjaWosWfPHuOhrcTERNLS0igoKKC8vJzo6Oh6Fb1kyRLj54CAgOt+Aby9vUlOTiY5OZl27dqxbt26K0JrfU2fPp2SkpIGHUNaFy8vL3eXICLNxMvL66oPIkvb1bt3b5YuXdokx/7DH/5gZJ3rjWkFGDJkCJ9//jm5ubnExcUB4Ofnh8PhMF5fauLEiSQlJdV5zBp79+5tcIa6WXWG1pEjR/Lss8/y5ptvMnPmTKB6zGpdOnXqRGpqKnPmzOGtt97CYrGwfPlyysvLGTlyJFAdIuPj45k6dSopKSmN1JWry83NZeDAgXTv3p2Kigq++uorRowY0eDjNtUXU0REzC8rK8vdJYhcU+fOnfH392fZsmVs3boVgLi4ONLT0/nzn/98Rfvhw4czb96862Yyl8vF3Llz63y2qSnVOabVw8OD7Oxstm3bxh133EFsbCypqam89tprRpvNmzfj5+dn/MnLy+OVV17h1ltvJTg4mP79+5OVlcXq1auNcatQPURg3759V1ygy8e0NnSw7+HDh7nnnnuwWq0MGjSImJgYxowZ06BjioiIiJjZ0KFDOX/+PP7+/kB1aD1y5Eit8aw1PDw8mDt3LrfddtsV+w4fPmxMeTV+/HieeeYZt8wcAOBR5c7BCSIi15CQkEBOTo7pjynmp89dpHXQilgiIiIiYnoKrSIiIiJiegqtIiIiImJ6Cq0iIiIiYnoKrSIiIiJiegqtIiIiImJ69Q6taWlppKenG69HjRrFtGnTjNd/+ctfWLhwIYWFhcTFxREWFkZERASrVq0C4MUXX2TevHm1jllYWNhoqyvMnj2bPn368PvvvwPVE+H6+fkZr2vYbDZ2794NwMqVK4mIiCAsLIzIyEimTZtGaWlpo9QjIiIiYtb8NGjQIGO1rYsXL+Lt7c3KlSuN/dHR0RQUFJCRkYGHhwebNm0y9mVnZ+Ph4cE//vEPAEaMGEFISAgREREMGDCAp556qlHzVL1D69ChQ3E6nQD8/vvvnDx5kuLiYvk9b0AAAAi0SURBVGO/0+nEbrfTsWNHli9fTnFxMRs2bGD27NmUlpaSkpJifAA1HA7HdVdhuN5auDX1rF69Gn9/f7Zt22a8r2/fvuzYscNod+DAAcrKyhg8eDAbNmxg0aJFrF+/nuLiYgoKCrDb7Zw4ceJGL4mIiIhIncyany6ta9++fQQHBxuvz5w5w+HDh4mMjATAarXicDiM92ZmZhr7arz33nvs37+f/fv306FDBxITE+s8f33UO7Ta7Xby8vIAKC4uJjw8nM6dO/PLL79w/vx5vv76a6KioozVsKB6Pd6ePXvy008/ERwcTPfu3Y27nAAffPBBoyznunXrVsLCwpg5cyaZmZnG9pSUlFoX2eFwMHHiRABeeuklXn/9dfr06QOAxWJh6tSphISENLgeERERETBvfrLb7UZIdTqdzJgxw7jzmp+fT3R0NBaLBahe7jU/P58LFy5w+vRpDh06hM1mu+pxPT09mT9/PkePHmXfvn0NqrFGvUNr7969ad++PUePHsXpdBIXF8fgwYPJy8vjiy++wGq14unpWes9+fn5VFRUEBQUBNQOkbt27cLHx8f4gBoiMzOTlJQUkpKSWLt2LRcuXABg/PjxZGdnc/HiRQBWrVplfMjFxcVERUU1+NwiIiIi12LW/HTpnVan08ndd99Nhw4dKCsrM+7+1vDw8OC+++5j48aNrFmzhj/+8Y91HttisRAZGcmBAwcaVGON9jfzpppU7nQ6mTNnDt9//z1Op5OuXbsydOjQWm2PHz/OY489xt///nfatavOyBMmTMBut7NgwYI6b23PmjWL3NxcAEpKSow0P27cOJ577rlabSsqKli3bh0LFy6kc+fODB48mI0bN5KQkICvry/h4eFs3rwZX19f2rdvT3h4+BXnKyoq4rHHHqOsrIyXX36ZCRMmXPMaTJ8+nZKSkhu/aCJSL15eXu4uQVoJLy8vEhIS3F2GtHK9e/dm6dKldbYxY37q168fFRUV/PDDDxw4cICQkBDuuusudu/ejdPp5Omnn67VfuLEibzxxhv8+uuvLFiwgJdffrnOPldVVdW5vz5uKrTWpPKioiLCw8Px9/dnwYIFdOnShSlTphjtfvvtNx5++GFeeuklhgwZYmz39/fnjjvuYNu2bXz44YfG7fLLLVmyxPg5ICDAuF19NRs3bqS0tBSr1QpAeXl5rb+oav514uvrW+tDDgsLo6CggPj4eKxWK4WFhTz11FOcPXu2zmtwvS+miIiYQ1ZWlrtLEAHMmZ+gOkxnZWXRq1cvPDw8GDJkCLm5ueTn5xMXF1erbWxsLEVFRXTs2JHg4OA6j1tZWUlRUVGjPWx/U1Ne2e12cnJy8PHxwWKx4OPjQ2lpKXl5ecZt5IqKCpKSkpg8eTJjx4694hgpKSmkpaURGBiIn59fw3pB9dCAd955B5fLhcvl4ttvv+Wzzz6jvLwcgOTkZNatW8eqVauM8awA8+bNY+7cuRw7dszYdr3AKiIiIlJfZsxPNXWlp6cbATUuLo7ly5dz++2307Vr1yvav/rqq9e9w3rhwgXmzZuHv78/ERERjVLnTYVWq9XKyZMna6V/q9VK165due2224DqwcHbt28nIyMDm82GzWarlfTHjRtHcXFxozyAVV5ezoYNG3j44YeNbZ06dWLYsGF88sknAHTr1o24uDh8fX0JDAw02o0ePZpnnnmGhx56iIEDB2K327FYLIwaNarBdYmIiIjUMFt+qjF06FCOHDlihNZevXpRWVlZazzrpR566CHi4+Ovum/SpElEREQQHh7OmTNnWLNmTaPV6VHVmIMNRERMLCEhgZycHHeXISIiN0ErYomIiIiI6Sm0ioiIiIjpKbSKiIiIiOkptIqIiIiI6Sm0ioiIiIjpNXlotVgsxpQNNpuNt99+2/jZ29ubkJAQbDYbkydPbvC5Ll68SI8ePfjb3/5mbHvxxReZN29erXaFhYXGRLenT59m5syZBAUFERUVRXR0NP/zP//T4FpEREREpPE0eWj18vKisLDQ+PPkk08aP8fExPDee+9RWFjI8uXLr3mMjIwMXnjhheue67PPPiM4OJisrCxj2bCUlBRWrVpVq92lS59NmzaN7t27880331BQUMCGDRv4+eefb77DIiIiItLoWtXwgMzMTP7t3/6Nvn37GkubBQcH0717d3bv3m20++CDD0hJSeHw4cPk5+fzX//1X8a6vj169OCvf/2rW+oXERERkatr8tB69uxZYzhAUlJSk53n3LlzbNq0iX/5l38hJSWFzMxMY19KSgoOhwOAXbt24ePjQ//+/SkuLiYyMtIIrCIiIiJiTu2b+gQ1wwPq69SpU9x7770A/Pzzz1RUVJCdnQ3AihUrsFqttdrn5OQQHx+Pl5cXY8aM4T//8z9JT0/HYrEwYcIE7HY7CxYsqDU04HIvvfQSWVlZ/Pjjj5SUlNRZ3/Tp06/bRkTMxcvLy90liIjITWryZVy9vb05ffr0VfeNGDGC119/nZiYmDqPkZGRgcvlqnNc65gxY9i5c6fxP6Uff/yRNWvWcP/99wNw99138+KLLzJ58mTy8vLw8/Pj0KFDPPDAAxw6dKjW3da6ahYRERGR5tcqfi/+22+/sWPHDo4ePYrL5cLlcrFkyZIrhgikpaURGBiIn58fAHfeeScxMTE8//zzVFZWAtXDDJo4x4uIiIhIPbWK0Lp69WpGjhxJhw4djG2JiYl88sknnD9/HoBx48ZRXFx8xdCAd955h1OnThkB9v7772f+/PnNWr+IiIiI1K3JhweIiIiIiDRUq7jTKiIiIiKtm0KriIiIiJieQquIiIiImJ5Cq4iIiIiYnkKriIiIiJieQquIiIiImJ5Cq4iIiIiYnkKriIiIiJieQquIiIiImJ5Cq4iIiIiYnkKriIiIiJieQquIiIiImJ5Cq4iIiIiYnkKriIiIiJieQquIiIiImJ5Cq4iIiIiYnkKriIiIiJje/wd0/FTx3VcP2wAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 648x169.2 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "graph_ranks(avg_ranks, list(sts12_pearson.keys()), cd=cd_nemenyi, width=9, textspace=1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonferroni-Dunn test\n",
    "\n",
    "Since the *FT + SIF* method has obtained the first position for all data sets, it seems reasonable to use the Bonferroni-Dunn test to compare this method with the others. The Bonferroni-Dunn test is also based on the calculation of a critical distance whose expression is very similar to the Nemenyi critical distance. Considering a confidence level of 95%, α= 0.05, and knowing that the number of methods is *K=9* and the number of data sets is *N=6*, the calculated Bonferroni-Dunn critical distance is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bonferroni-Dunn critical distance (alpha = 0.05) = 4.3070\n"
     ]
    }
   ],
   "source": [
    "cd_bonferroni_dunn = compute_CD(avg_ranks, N, alpha='0.05', test='bonferroni-dunn') \n",
    "print(\"Bonferroni-Dunn critical distance (alpha = 0.05) = \", cd_bonferroni_dunn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAq0AAACwCAYAAAAli1fHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de1BTZ/4/8HcMglG8sVWqXERQECEQLoIJaIVebC27CN6IbqVS18piu+C6s6t29ltn17Y6XrA7TKvbjizaEkst2OKFVl1vBKUVUZpW10uz1GJttWWLRA0ivz8Yzs8IRm7JOZD3a4aZJOfJcz7nJOrbw3OeR9bU1NQEIiIiIiIJ6yN2AURERERED8PQSkRERESSx9BKRERERJLH0EpEREREksfQSkRERESSx9BKRERERJLH0EpEREREksfQSkRERESSx9BKRERERJLH0EpEREREksfQSkRERESSx9BKZCPff/89UlJS4Ofnh4iICEybNg3/+c9/oFAoEBYWhsDAQERFRSE3N1fsUomIiCTPSewCiHqjpqYmJCUlITU1FTqdDgBw+vRpXL16FX5+fjh16hQA4NKlS0hOTkZTUxMWLFggZslERESSxiutRDbw73//G3379sXixYuF10JDQ+Hl5WXRztfXFxs2bMCbb75p7xKJiIh6FIZWIhv48ssvERER0a624eHhOHv2rI0rIiIi6tkYWolE1tTUJHYJREREksfQSmQDQUFBOHnyZLvanjp1CoGBgTauiIiIqGdjaCWygfj4eNy+fRtbtmwRXjtz5gy+/fZbi3ZGoxHLli3DSy+9ZO8SiYiIehRZE383SWQTNTU1yMzMxMmTJ9GvXz/4+PggOzsbISEhGDduHG7duoWBAwfi97//PZ5//nmxyyUiIpI0hlYiIiIikjwOD7CBRYsWiV0CEfVQ/PuDiKhtDK02UFNTI3YJRNRD8e8PIqK2MbQSERERkeQxtBIRERGR5PFGLBvw9vZGSEiI2GUQUQ905swZVFdXi10GEZHkOIldQG8UEhKC4uJiscsgoh4oISFB7BKIiCSJwwOIiIiISPIYWm1g5MiRYpdARD0U//4gImobx7QSERERkeTxSisRERERSR5DKxERERFJHkMrEREREUkeQysRERERSR5DKxERERFJHkMrEREREUkeQysRERERSR5DazdKS0vD8OHDERwcLHYpdvftt98iLi4O48ePR1BQEDZt2iR2SXZz69YtREVFITQ0FEFBQfi///s/sUuyu8bGRoSFhTncEqQ+Pj5QKpVQqVSIjIwUuxy7qq2txcyZMzFu3DgEBgairKxM7JLs4ty5c1CpVMLPoEGDkJ2dLXZZdrNx40YEBQUhODgYWq0Wt27dErsku9m0aROCg4MRFBTkUJ+5lHBxgW505MgRuLq6Yv78+fjyyy/FLseurly5gitXriA8PBx1dXWIiIhAUVERxo8fL3ZpNtfU1IT6+nq4urqioaEBsbGx2LRpEyZOnCh2aXazYcMGfPHFF/jll19QXFwsdjl24+Pjgy+++AKPPPKI2KXYXWpqKiZNmoSFCxfCbDbDZDJhyJAhYpdlV42NjfDw8MCJEycwatQoscuxue+++w6xsbH46quvoFAoMHv2bEybNg3PP/+82KXZ3JdffomUlBSUl5fD2dkZTz/9NN5++22MGTNG7NIcCq+0dqPJkyfDzc1N7DJEMWLECISHhwMABg4ciMDAQHz33XciV2UfMpkMrq6uAICGhgY0NDRAJpOJXJX9XL58Gbt378bChQvFLoXs5H//+x+OHDmCF154AQDg7OzscIEVAA4cOAA/Pz+HCKwt7ty5g5s3b+LOnTswmUwOs+zw119/jejoaPTv3x9OTk547LHH8NFHH4ldlsNhaKVuZzQacerUKURHR4tdit00NjZCpVJh+PDhePLJJx3q2DMzM7F27Vr06eN4f53IZDI89dRTiIiIwJYtW8Qux26++eYbDBs2DAsWLEBYWBgWLlyI+vp6scuyO51OB61WK3YZduPh4YFly5bB29sbI0aMwODBg/HUU0+JXZZdBAcH4+jRo7h+/TpMJhP27NmDb7/9VuyyHI7j/StDNnXjxg3MmDED2dnZGDRokNjl2I1cLkdlZSUuX76M8vJyhxkeUlxcjOHDhyMiIkLsUkRx7NgxVFRUYO/evcjJycGRI0fELsku7ty5g4qKCqSnp+PUqVMYMGAA3njjDbHLsiuz2YyPP/4Ys2bNErsUu/n555+xa9cufPPNN6ipqUF9fT22b98udll2ERgYiD//+c946qmn8PTTT0OlUkEul4tdlsNhaKVu09DQgBkzZmDevHlITk4WuxxRDBkyBHFxcdi3b5/YpdhFaWkpPv74Y/j4+CAlJQUHDx7Eb3/7W7HLshsPDw8AwPDhw5GUlITy8nKRK7IPT09PeHp6Cr9RmDlzJioqKkSuyr727t2L8PBwuLu7i12K3ezfvx+jR4/GsGHD0LdvXyQnJ0Ov14tdlt288MILOHnyJI4cOYKhQ4fC399f7JIcDkMrdYumpia88MILCAwMxNKlS8Uux65+/PFH1NbWAgBu3ryJzz77DOPGjRO5Kvt4/fXXcfnyZRiNRuh0OsTHxzvMlZf6+nrU1dUJjz/99FOHmTnk0UcfhZeXF86dOwegeWynI9x0ea/8/HyHGhoAAN7e3jh+/DhMJhOamppw4MABBAYGil2W3fzwww8AgOrqanz00UeYO3euyBU5HiexC+hNtFotDh06hGvXrsHT0xOrVq0SblTo7UpLS7Ft2zZh+h8AeO211zBt2jSRK7O9K1euIDU1FY2Njbh79y5mz57tcFM/OaKrV68iKSkJQPOvy+fOnYunn35a5Krs5x//+AfmzZsHs9kMX19fbN26VeyS7Ka+vh6fffYZNm/eLHYpdhUdHY2ZM2ciPDwcTk5OCAsLw6JFi8Quy25mzJiB69evo2/fvsjJyXHImw/FximviIiIiEjyODyAiIiIiCSPoZWIiIiIJI+hlYiIiIgkj6GViIiIiCSPodUGHOluyvvx2B0Tj90x8dgdE4+dxMLQagM1NTVilyAaHrtj4rE7Jh67Y+Kxk1gYWnshR/6fII/dMfHYHROP3TE58rE7OobWXsiR/yfIY3dMPHbHxGN3TI587I6OiwvYwJAhQxAbGyva/s+cOYOQkJBes++O9CnmsR86dAhTpkwRZd8AP/f2tBW7TlvobZ+7LfZvizp74593W5zP3vZn7uLFi/j6669F2TdxGVebiI2NRXFxsdhl9BoJCQk94nz2lDp7ClucT35GjonfpfbheSKp4/AAIiIiIpI8hlYiIiIikjyGViIiIiKSPIZWIiIiIpI8hlYiIiIikjyGViIiIiKSPIZWIiIiIpI8hlYiIiIikjyGViIiIiKSPIZWIiIiIpI8hlYiIiIikjwnsQvoDTIzM1FZWSk8r6qqwpQpU8QrqJfpKeezp9TZU9jifLbVp0qlQnZ2drfuh4iIuh9DazeorKzE4cOHLV67/zl1TU85nz2lzp7CFueTnxERUc/E0NoNVCqVxfOqqioolUqRqul9esr57Cl19hS2OJ9t9Xn/n18iIpImhtZucP+vFhMSElBcXCxSNb1PTzmfPaXOnsIW55OfERFRz8UbsYiIiIhI8hhaiYiIiEjyGFqJiIiISPI4ppWokzjVmW3Za8or6v04fRpR78DQStRJnOrM9jjlFXUXfpeIej6GVqJO4lRntmWvKa+o9+P0aUS9A0MrUSdxqjPb4pRX1F34XSLqHXgjFhERERFJHkMrEREREUkeQysRERERSR7HtBKRJNhjCjFOeeWY+F1qH3ucJ04LRl3B0EpEkmCvKcQ4TZFj4nepfXieSMoYWolIEuwxhRinvHJM/C61jz3OE6cFo65gaCUiSbDHFGKcpsgx8bvUPjxPJHW8EYuIiIiIJI+hlYiIiIgkj6GViIiIiCSPoZWIiIiIJI+hlYiIiIgkj6GViIiIiCSPoZWIiIiIJE/U0CqXy6FSqYSfzZs3C49dXV0REBAAlUqF+fPnd2k/x48fR3R0NFQqFQIDA/Hqq68CAHJzc7FkyRIAwKuvvgoPDw9h/3/5y1+6enhEREREgqtXr2Lu3Lnw9fVFREQE1Go1CgsLAQCHDh1CQkJCq/eYzWZkZmZizJgxGDt2LBITE3H58mUAQFxcHEpKSizaZ2dnIz09HUajEQqFwiJn5eXldan+sLAwYbntO3fuwNXVFdu3bxe2R0REoKKiArm5uZDJZNi/f7+wraioCDKZDB9++CEAYMqUKQgICEBISAjGjRuHJUuWoLa21ur+RV1cQKFQWKw1DgAvvvgigOaDWbduHSIjI632kZubC6PRKATRtqSmpuKDDz5AaGgoGhsbce7cuTbbZWVlYdmyZR07CCIiIqKHaGpqwvTp05Gamor3338fAPDf//4XH3/8sdX3rVixAnV1dTh37hzkcjm2bt2K5ORknDhxAlqtFjqdDlOnThXa63Q6rF27FgDg5+fXKmdZ4+PjA6PR+MDtMTEx0Ov1UKlUOH36NPz9/aHX6/Hb3/4W9fX1uHjxIkJDQ3HmzBkolUrodDo88cQTAID8/HyEhoZa9Pfee+8hMjISZrMZy5cvR2JiotVlfx1ieMAPP/yAESNGAGi+ujt+/HiRKyIiIiJHcvDgQTg7O2Px4sXCa6NGjcJLL730wPeYTCZs3boVGzduhFwuBwAsWLAALi4uOHjwIGbOnIndu3fDbDYDAIxGI2pqajBp0iSbHINGo4FerwcA6PV6LF68WAjF5eXliIiIEOqcNGkSysvL0dDQgBs3buDChQsPXMbX2dkZa9euRXV1NU6fPv3A/YsaWm/evClcsk5KSrLZfrKyshAQEICkpCRs3rwZt27darPdxo0bhXruv9xORERE1FkGgwHh4eEdes+FCxfg7e2NQYMGWbweGRkJg8EANzc3REVFYe/evQCar7LOnj0bMpkMAHDx4kWL4QFHjx7t0jG0XGkFmkPr5MmT4eLigrq6Ouj1emg0GqGtTCbDE088gZKSEuzatQu/+c1vrPYtl8sRGhqKs2fPPrCN5IYHtMf169fx+OOPAwB++uknmM1mFBUVAQC2bdsGpVJp0f6vf/0r5s2bh08//RTvv/8+8vPzcejQoVb9cngAERER2UNGRgaOHTsGZ2dnfP75553up2WIQGJiInQ6Hd59911hW3uGB2RkZKC0tBQAUFNTI1wNnTVrFlauXGnRdtSoUTCbzfj+++9x9uxZBAQEYMKECThx4gT0en2rq8YpKSl488038b///Q/r16/Ha6+9ZrWWpqYmq9tFDa2d9atf/Ur4ENozphVo/uDS09Pxu9/9DsOGDcP169e7VMOiRYtQU1PT5jaFQtGlvomIiHoDhULR5s1FvdHIkSOxZcuWB24PCgrCzp07hec5OTm4du2a1Xt3/Pz8UF1djbq6OgwcOFB4/eTJk8J5TUxMRFZWFioqKmAymRAREdGhunNycoTHPj4+Dw25Go0GBQUFGDFiBGQyGSZOnIjS0lKUl5dDrVZbtI2KikJVVRX69+8Pf39/q/02NjaiqqoKgYGBD2zTI0NrR+3evRvTpk2DTCbD+fPnIZfLMWTIkC71ae2LSUREREBBQYHYJUhGfHw8VqxYgbfeegvp6ekAmsesWjNgwACkpqZi6dKlePvttyGXy5GXlweTyYT4+HgAgKurK+Li4pCWlgatVmvz49BoNMjOzsbzzz8PAFCr1fjTn/6ERx99FIMHD27V/o033kC/fv2s9tnQ0ICVK1fCy8sLISEhD2znEKF127ZtyMrKQv/+/eHk5IT33ntPGChMREREZGsymQxFRUXIysrC2rVrMWzYMAwYMABr1qwR2hw4cACenp7C84KCArz++utYtmwZ/P390adPH4wbNw6FhYXCuFWgeYhAUlISdDqdxT5bxrS2SEtLw8svv9yl44iJiUFWVpZwVXXEiBFobGy0GM96r2eeeeaBfc2bNw8uLi64ffs2nnjiCezatcvqvmVNDxtAQCSyhIQEFBcXi13GQ/WUOnsKW5xPfkaOid+l9umNx0S9i0NMeUVEREREPRtDKxERERFJHkMrEREREUkeQysRERERSR5DKxERERFJHkMrEREREUlet4XWrKwsZGdnC8+nTp2KhQsXCs//+Mc/YsOGDaisrIRarUZQUBBCQkKwY8cOAMCqVauwfPlyiz4rKyutrozQHiaTCfPmzYNSqURwcDBiY2Nx48YNAM0T8gKA0WiEQqGwWJ/XbDZ3ab9ERERE97p69Srmzp0LX19fREREQK1Wo7CwEABw6NChNlcPM5vNyMzMxJgxYzB27FgkJibi8uXLAIC4uDiUlJRYtM/OzkZ6enqb2SYvL69bjiMzMxMeHh64e/cugOYc5enpKTxvoVKpcOLECQDA9u3bERISgqCgIISGhmLhwoWora3t0H67LbTGxMRAr9cDAO7evYtr167BYDAI2/V6PTQaDfr374+8vDwYDAbs27cPmZmZqK2thVarFQJsC51O99DVHXx8fKxu37RpE9zd3VFVVYUvv/wS7777Lvr27duqXcv6vC0/zs7O7TxyIiIiIuuampowffp0TJ48GZcuXcLJkyeh0+mEAPogK1asQF1dHc6dO4fz589j+vTpSE5ORlNTE7RabasFBe7NTvdnm/nz51vd18MyFdCc8QoLC+Hl5YXDhw8L7/P29sbRo0eFdmfPnkVdXR2io6Oxb98+bNy4EXv37oXBYEBFRQU0Gg2uXr360P3dq9tCq0ajQVlZGQDAYDAgODgYAwcOxM8//4zbt2/j66+/Rnh4OPz9/TF27FgAzev0Dh8+HD/++CP8/f0xdOhQIZEDwAcffNDlJcmuXLkCDw8P4XlAQABcXFy61CcRERFRRxw8eBDOzs5YvHix8NqoUaPw0ksvPfA9JpMJW7duxcaNG4WVPBcsWAAXFxccPHgQM2fOxO7du4XfDhuNRtTU1GDSpEk2O45Dhw4hKCgI6enpyM/PF16/P0DrdDqkpKQAAFavXo1169YJeUwulyMtLQ0BAQEd2ne3hdaRI0fCyckJ1dXV0Ov1UKvViI6ORllZGb744gsolcpWVy/Ly8thNpvh5+cHwPKAjx8/Djc3NyHgdlZaWhrWrFkDtVqNV155BefPn2+zXctSZyqVChkZGV3aJxEREdG9DAYDwsPDO/SeCxcuwNvbG4MGDbJ4PTIyEgaDAW5uboiKisLevXsBNAfF2bNnC0u83pttVCqVxZXQzsrPzxeWjd29ezcaGhoAALNnz0ZRURHu3LkDANixY4dw4bEzx94Wpy73cA+NRgO9Xg+9Xo+lS5fiu+++g16vx+DBgxETE2PR9sqVK3juuefwr3/9C336NGfnOXPmQKPRYP369VaHBmRkZKC0tBQAUFNTI6yrO2vWLKxcudKirUqlwqVLl/Dpp59i//79mDBhAsrKylqNlW25hN5eixYtQk1NTbvbU+cpFAqxSyAi6vUUCkWbYyqpfUaOHIktW7a0u31GRgaOHTsGZ2dnfP75553eb8sFv8TEROh0Orz77rvCtvZkm45kKrPZjD179mDDhg0YOHAgoqOjUVJSgoSEBLi7uyM4OBgHDhyAu7s7nJycEBwc3Gp/VVVVeO6551BXV4fXXnsNc+bMafexdmtobRnXWlVVheDgYHh5eWH9+vUYNGgQFixYILT75Zdf8Oyzz2L16tWYOHGi8LqXlxdGjx6Nw4cPY+fOncJwg/vl5OQIj318fB76gbi6uiI5ORnJycno06cP9uzZ0+UbvDryxSQiIpK6goICsUvo1YKCgrBz507heU5ODq5du4bIyMgHvsfPzw/V1dWoq6vDwIEDhddPnjwp/AcjMTERWVlZqKiogMlkQkRERIfq6kimKikpQW1tLZRKJYDm4Qv3/menJUC7u7tbXHgMCgpCRUUF4uLioFQqUVlZiSVLluDmzZsdqrVbp7zSaDQoLi6Gm5sb5HI53NzcUFtbi7KyMmg0GgDNKT0pKQnz58/HzJkzW/Wh1WqRlZUFX19feHp6drmm0tJS/Pzzz8K+v/rqK4waNarL/RIRERG1V3x8PG7duoW33npLeM1kMll9z4ABA5CamoqlS5eisbERAJCXlweTyYT4+HgAzRfm4uLikJaW1uX7gB4mPz8f77zzDoxGI4xGI7755ht89tlnwnEkJydjz5492LFjhzCeFQCWL1+OZcuWWdx01tHACnRzaFUqlbh27ZrF1VOlUonBgwfjkUceAdB8c9WRI0eQm5srjLG4N9XPmjULBoOh2078xYsX8dhjj0GpVCIsLAyRkZGYMWNGt/RNRERE1B4ymQxFRUU4fPgwRo8ejaioKKSmpmLNmjVCmwMHDsDT01P4KSsrw+uvv45+/foJN7IXFBSgsLBQGLcKNF/wO336dKvsdP+Y1jfffLPT9ZtMJuzbtw/PPvus8NqAAQMQGxuLTz75BAAwZMgQqNVquLu7w9fXV2g3bdo0vPzyy3jmmWcwfvx4aDQayOVyTJ06tUM1yJqampo6fQREJEhISEBxcbHYZfQatjif/IwcE79LRL0DV8QiIiIiIsljaCUiIiIiyWNoJSIiIiLJY2glIiIiIsljaCUiIiIiyXtoaL169Srmzp0LX19fREREQK1Wo7CwEEDz+rNtrZ5hNpuRmZmJMWPGYOzYsUhMTBTm5oqLi0NJSYlF++zsbKSnp8NoNEKhUFhMz5CXl9elAzSZTJg3bx6USiWCg4MRGxuLGzduAGie2wxAm/ttWceXiIiIqDvI5XKLrLF582bhsaurKwICAqBSqTB//vxO7+P06dPCqlZA89yqCoVCWG61qqoKISEhAIApU6bA29sb904kNX369Fb5KCwsDIGBgYiKikJubm6na+sqqytiNTU1Yfr06UhNTcX7778PAPjvf/+Ljz/+2GqnK1asQF1dHc6dOwe5XI6tW7ciOTkZJ06cEFZLuHduLp1Oh7Vr1wLo+HKqPj4+MBqND9y+adMmuLu7o6qqCgBw7tw59O3bt1W7ju6XiIiIqCMUCkWrrPHiiy8CaA6Q69ats7pCFgDk5ubCaDTi1VdfbXO7Uqm0WEVLr9cjMDAQp06dQlRUFPR6vbDgE9A8t2ppaSliY2NRW1uLK1euWPTn5+eHU6dOAQAuXbqE5ORkNDU1Wax0ai9Wr7QePHgQzs7OWLx4sfDaqFGj8NJLLz3wPSaTCVu3bsXGjRshl8sBAAsWLICLiwsOHjyImTNnYvfu3cKVTKPRiJqaGkyaNKk7jqeVK1euwMPDQ3geEBAAFxcXm+yLiIiISEx9+vRBZGQkTpw4AaB5ydeMjAzo9XoAgF6vR0xMjNA+JSUFOp0OAPDRRx8hOTn5gX37+vpiw4YNXVqkoCushlaDwYDw8PAOdXjhwgV4e3tj0KBBFq9HRkbCYDDAzc0NUVFR2Lt3L4Dmq6yzZ88WVna4f/WGo0ePdmj/90tLS8OaNWugVqvxyiuv4Pz58222u3e/GRkZXdonERER0f1u3rwpZI2kpCSb7ScmJgZ6vR719fXo06cPpkyZYhFa773S+vjjj+PIkSNobGyETqfDnDlzrPYdHh6Os2fP2qx2a6wOD7hfRkYGjh07BmdnZ3z++eed3mnLEIHExETodDq8++67wrb2/Jo+IyMDpaWlAICamhph7MasWbOwcuVKi7YqlQqXLl3Cp59+iv3792PChAkoKytDYGCgRTsODyAiIiJbamt4QHtcv34djz/+OADgp59+gtlsRlFREQBg27ZtUCqVFu01Gg3Wr1+PSZMmYcKECfDz88OFCxfw448/4saNG/Dz8xPayuVyxMbGQqfT4ebNm/Dx8bFai5gLqVoNrUFBQdi5c6fwPCcnB9euXbM63sLPz89iLEWLkydPCjdtJSYmIisrCxUVFTCZTIiIiOhQ0Tk5OcJjHx+fh34BXF1dkZycjOTkZPTp0wd79uxpFVo7atGiRaipqelSH9S7KBQKsUsgIjtRKBRt3ohMjmvkyJHYsmWLTfr+1a9+JWSdh41pBYCJEyfi888/R2lpKdRqNQDA09MTOp1OeH6vlJQUJCUlWe2zxalTp7qcoTrLamiNj4/HihUr8NZbbyE9PR1A85hVawYMGIDU1FQsXboUb7/9NuRyOfLy8mAymRAfHw+gOUTGxcUhLS0NWq22mw6lbaWlpRg/fjyGDh0Ks9mMr776ClOmTOlyv7b6YhIRkfQVFBSIXQLRAw0cOBBeXl7YunUrDh06BABQq9XIzs7G73//+1btJ02ahOXLlz80kxmNRixbtszqvU22ZHVMq0wmQ1FREQ4fPozRo0cjKioKqampWLNmjdDmwIED8PT0FH7Kysrw+uuvo1+/fvD398fYsWNRUFCAwsJCYdwq0DxE4PTp061O0P1jWrs62PfixYt47LHHoFQqERYWhsjISMyYMaNLfRIRERFJWUxMDG7fvg0vLy8AzaH10qVLFuNZW8hkMixbtgyPPPJIq20XL14UpryaPXs2Xn75ZVFmDgAAWZOYgxOIiB4gISEBxcXFku+TpI+fO1HvwBWxiIiIiEjyGFqJiIiISPIYWomIiIhI8hhaiYiIiEjyGFqJiIiISPI6HFqzsrKQnZ0tPJ86dSoWLlwoPP/jH/+IDRs2oLKyEmq1GkFBQQgJCcGOHTsAAKtWrcLy5cst+qysrOy2iWozMzPh4eGBu3fvAmieU8zT01N43kKlUgnr8m7fvh0hISEICgpCaGgoFi5ciNra2m6ph4iIiEiq+SksLExYuODOnTtwdXXF9u3bhe0RERGoqKhAbm4uZDIZ9u/fL2wrKiqCTCbDhx9+CACYMmUKAgICEBISgnHjxmHJkiXdmqc6HFpb1rMFgLt37+LatWswGAzC9pY1bfv374+8vDwYDAbs27cPmZmZqK2thVarFT6AFjqd7qET2j5sWbGWegoLC+Hl5YXDhw8L7/P29sbRo0eFdmfPnkVdXR2io6Oxb98+bNy4EXv37oXBYEBFRQU0Gg2uXr3a3lNCREREZJVU89O9dZ0+fRr+/v7C8/r6ely8eBGhoaEAAKVSCZ1OJ7w3P9D9QT0AAAgGSURBVD9f2Nbivffew5kzZ3DmzBm4uLggMTHR6v47osOhVaPRoKysDABgMBgQHByMgQMH4ueff8bt27fx9ddfIzw8XFhYAGhe2mz48OH48ccf4e/vj6FDhwpXOQHggw8+6JaVsQ4dOoSgoCCkp6cjPz9feF2r1VqcZJ1Oh5SUFADA6tWrsW7dOnh4eABoXoM3LS0NAQEBXa6HiIiICJBuftJoNEJI1ev1WLx4sXDltby8HBEREZDL5QCaV84qLy9HQ0MDbty4gQsXLkClUrXZr7OzM9auXYvq6mqcPn26SzW26HBoHTlyJJycnFBdXQ29Xg+1Wo3o6GiUlZXhiy++gFKphLOzs8V7ysvLYTab4efnB8AyRB4/fhxubm7CB9QV+fn50Gq1SEpKwu7du9HQ0AAAmD17NoqKinDnzh0AwI4dO4QP2WAwIDw8vMv7JiIiInoQqeane6+06vV6TJ48GS4uLqirqxOu/raQyWR44oknUFJSgl27duE3v/mN1b7lcjlCQ0Nx9uzZLtXYolM3YrWk8paTrlarhecxMTEWba9cuYLnnnsOW7duRZ8+zbubM2cOPvzwQ9y9e9fqpe2MjAxhOdeamhrh8erVq1u1NZvN2LNnD6ZPn45BgwYhOjoaJSUlAAB3d3cEBwfjwIEDqKyshJOTE4KDg1v1UVVVBZVKBT8/v1aX4ImIiIi6Qor5adSoUTCbzfj+++9x9uxZBAQEYMKECThx4kSbdaWkpECn07VraAIAdOfCq06deVNLKq+qqkJwcDC8vLywfv16DBo0yGI92l9++QXPPvssVq9ejYkTJwqve3l5YfTo0Th8+DB27twpXC6/X05OjvDYx8dHuFzdlpKSEtTW1kKpVAIATCYTFAoFEhISAPz//524u7tbnOSgoCBUVFQgLi4OSqUSlZWVWLJkCW7evGn1HCxatAg1NTVW2xBR5ykUCrFLoF7i3n8LiGxl5MiR2LJli9U2UsxPQHOYLigowIgRIyCTyTBx4kSUlpaivLwcarXaom1UVBSqqqrQv39/+Pv7W+23sbERVVVV3XazfadCq0ajwbp16+Dr6wu5XA43NzfU1tbCYDDgn//8J4DmK59JSUmYP38+Zs6c2aoPrVaLrKws+Pr6wtPTs2tHgeahAe+8844QSOvr6zF69GiYTCb0798fycnJWL58Ofr3748DBw4I71u+fDmWLVuGXbt2CXU8LLACeOgXk4iIpKGgoEDsEogASDM/tdSVnZ2N559/HgCgVqvxpz/9CY8++igGDx7cqv0bb7yBfv36We2zoaEBK1euhJeXF0JCQrqlzk4ND1Aqlbh27ZpF+lcqlRg8eDAeeeQRAM2Dg48cOYLc3FzhsvS9SX/WrFkwGAzdcgOWyWTCvn378OyzzwqvDRgwALGxsfjkk08AAEOGDIFarYa7uzt8fX2FdtOmTcPLL7+MZ555BuPHj4dGo4FcLsfUqVO7XBcRERFRC6nlpxYxMTG4dOmScFV1xIgRaGxstBjPeq9nnnkGcXFxbW6bN28eQkJCEBwcjPr6euzatavb6pQ1dedgAyIiCUtISEBxcbHYZRARUSdwRSwiIiIikjyGViIiIiKSPIZWIiIiIpI8hlYiIiIikjyGViIiIiKSPIZWIiIiIpI8m4dWuVwuzDOmUqmwefNm4bGrqysCAgKgUqkwf/78Lu/rzp07GDZsGP7yl78Ir61atQrLly+3aFdZWSmsznDjxg2kp6fDz88P4eHhiIiIECb4JSIiIiJpsHloVSgUqKysFH5efPFF4XFkZCTee+89VFZWIi8v74F95Obm4tVXX33ovj777DP4+/ujoKBAWOtWq9Vix44dFu3uXS934cKFGDp0KM6fP4+Kigrs27cPP/30U+cPmIiIiIi6Xa8aHpCfn48//OEP8Pb2Ftbj9ff3x9ChQ3HixAmh3QcffACtVouLFy+ivLwcf//739GnT/OpGDZsGP785z+LUj8RERERtc3mofXmzZvCcICkpCSb7efWrVvYv38/fv3rX0Or1SI/P1/YptVqodPpAADHjx+Hm5sbxo4dC4PBgNDQUCGwEhEREZE0Odl6By3DAzrq+vXrePzxxwEAP/30E8xmM4qKigAA27Ztg1KptGhfXFyMuLg4KBQKzJgxA3/729+QnZ0NuVyOOXPmQKPRYP369RZDA+63evVqFBQU4IcffkBNTY3V+hYtWvTQNkQkLQqFQuwSiIiok2RNLYM/bcTV1RU3btxoc9uUKVOwbt06REZGWu0jNzcXRqPR6rjWGTNm4NixY8I/Sj/88AN27dqFJ598EgAwefJkrFq1CvPnz0dZWRk8PT1x4cIFPPXUU7hw4YLF1VZrNRMRERGR/fWK34v/8ssvOHr0KKqrq2E0GmE0GpGTk9NqiEBWVhZ8fX3h6ekJABgzZgwiIyPxyiuvoLGxEUDzMAMb53giIiIi6qBeEVoLCwsRHx8PFxcX4bXExER88sknuH37NgBg1qxZMBgMrYYGvPPOO7h+/boQYJ988kmsXbvWrvUTERERkXU2Hx5ARERERNRVveJKKxERERH1bgytRERERCR5DK1EREREJHkMrUREREQkeQytRERERCR5DK1EREREJHkMrUREREQkeQytRERERCR5DK1EREREJHkMrUREREQkeQytRERERCR5DK1EREREJHkMrUREREQkeQytRERERCR5DK1EREREJHkMrUREREQkeQytRERERCR5/w8ppACNj3vS4QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 648x162 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "graph_ranks(avg_ranks, list(sts12_pearson.keys()), cd=cd_bonferroni_dunn, width=9, textspace=1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The critical distance of the Bonferroni-Dunn test is **4.3070**. Selecting the *FT + SIF* method as the control method, it is found that this method is significantly better than *W2V + WMD*, *FT + WMD*, *GLOVE + WMD* and *GLOVE + AVG*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Combining FastText's pre-trained word vectors by performing a SIF average is the most accurate method for measuring semantic similarity between pairs of documents.\n",
    "   \n",
    "\n",
    "2. Among the evaluated pre-trained vector sets, it can be observed that the GloVe set performs worse than the Word2Vec or FastText sets, regardless of the method used to calculate semantic similarity.\n",
    "\n",
    "    \n",
    "3. Although performing an unweighted average of the word vectors is a good starting point to obtain a vector of a sentence, applying a SIF (Smooth Inverse Frequency) weighting is a better alternative. \n",
    "   \n",
    "\n",
    "\n",
    "4. Among the possibilities to measure similarity, the inverse of the distance *Word Mover's Distance* as a measure gives the worst results."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}