LSTM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SENTIMENT ANALYSIS (IN ARABIC) ###\n",
    "\n",
    "#### TODO ####\n",
    "  - Try with a CNN, per: https://arxiv.org/pdf/1702.01923.pdf\n",
    "  - Better clean input, remove high seq # outliers (Twitter dataset) (from 176 to closer to 20/30)\n",
    "    - https://www.kaggle.com/lystdo/lstm-with-word2vec-embeddings/code (https://www.kaggle.com/currie32/quora-question-pairs/the-importance-of-cleaning-text)\n",
    "  - GRU vs. LSTM (https://datascience.stackexchange.com/questions/14581/when-to-use-gru-over-lstm)\n",
    "  - t-SNE implementation (even though it won't make much sense to non-Arabic reader)\n",
    "  - Grid Search, across Hyperparams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from keras.models import Model, Sequential\n",
    "from keras.layers import Dense, Input, Dropout, LSTM, Activation\n",
    "from keras.layers import Bidirectional, GlobalMaxPool1D\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "from keras.initializers import glorot_uniform\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__[fastText](https://fasttext.cc/)__:\n",
    " - 2016, Mikolov [1] (same person who created word2vec)\n",
    " - Learning is \"fast\"\n",
    " - Pre-trained word vectors via Facebook Research (https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md)\n",
    "  - Pre-trained word vectors for 294 languages, trained on Wikipedia using fastText. These vectors in dimension 300 were obtained using the skip-gram model described in __[Bojanowski et al. (2016)](https://arxiv.org/abs/1607.04606)__ with default parameters.\n",
    " \n",
    " \n",
    " 1. P. Bojanowski*, E. Grave*, A. Joulin, T. Mikolov, __[Enriching Word Vectors with Subword Information](https://arxiv.org/abs/1607.04606)__\n",
    " \n",
    "__[GloVe](http://nlp.stanford.edu/projects/glove/)__:\n",
    " - 2014, Stanford NLP\n",
    " \n",
    "__[word2vec](https://code.google.com/archive/p/word2vec/)__:\n",
    " - 2013, Mikolov\n",
    "\n",
    "__[How is GloVe different from word2vec?](https://www.quora.com/How-is-GloVe-different-from-word2vec)__: TL;DR\n",
    " - Perform roughtly the same function, but are built differently (Word2Vec is a \"predictive\" Feed-Fwd NN w/ SGD)\n",
    " - GloVe is result of Stanford NLP wanting to build on the Word2Vec concept of \"semantic analogies\" and yet retain \"intrinsic statistical properties of the corpus\" (i.e., \"word occurrence statistics\")\n",
    "\n",
    "__[GloVe vs word2vec revisited.](http://dsnotes.com/post/glove-enwiki/)__: TL;DR\n",
    " - Results are similar enough (maybe GloVe is slightly more accurate)\n",
    " - Trade-off: GloVe uses A LOT of RAM/Memory (on order of 15x more!); Word2Vec is 2x slower"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Credits ####\n",
    "\n",
    " - Sentiment Analysis for Arabic Text (tweets, reviews, and standard Arabic) using word2vec\n",
    "  - https://github.com/iamaziz/ar-embeddings\n",
    "  - A. Altowayan and L. Tao \"Word Embeddings for Arabic Sentiment Analysis\", IEEE BigData 2016 Workshop\n",
    " - Twitter Data set for Arabic Sentiment Analysis Data Set\n",
    "  - https://archive.ics.uci.edu/ml/datasets/Twitter+Data+set+for+Arabic+Sentiment+Analysis\n",
    "  - Abdulla N. A., Mahyoub N. A., Shehab M., Al-Ayyoub M.,Arabic Sentiment Analysis: Corpus-based and Lexicon-based,IEEE conference on Applied Electrical Engineering and Computing Technologies (AEECT 2013),December 3-12, 2013, Amman, Jordan. (Accepted for Publication).\n",
    " - arabic-stop-words; 750 words in a text file\n",
    "  - https://github.com/mohataher/arabic-stop-words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gensim.models import KeyedVectors\n",
    "def load_w2v(filepath,binary):\n",
    "    return KeyedVectors.load_word2vec_format(filepath, binary=binary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "610977\n"
     ]
    }
   ],
   "source": [
    "# word2vec (via Aziz Alto), : https://drive.google.com/open?id=0ByiDbCx0i9pEQV9ZUEFIb0hwMmM\n",
    "# 185 MB; vocab=159175; takes ~5-10 secs to load\n",
    "#w2v = load_w2v(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/arabic-news.bin\", binary=True)\n",
    "# --\n",
    "# fastText (via Facebook Research), Wikipedia: https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.ar.vec\n",
    "# 1.5 GB; vocab=610977; takes ~3 mins to load\n",
    "w2v = load_w2v(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/wiki.ar.vec\", binary=False)\n",
    "print(len(w2v.vocab))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "MAX_SEQUENCE_LENGTH = 85 #176 # based on our inputs; TODO: remove outliers? dynamically calculate!\n",
    "MAX_NB_WORDS = len(w2v.vocab)\n",
    "EMBEDDING_DIM = 300 # w2v, fastText; GloVe=50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(300,)\n",
      "[('،حسن', 0.6795670986175537), ('أبوحسن', 0.6792263388633728), ('#حسن', 0.6771581172943115), ('وحسن', 0.6517007350921631), ('لحسن', 0.6505534648895264), ('\\u200fحسن', 0.6446623206138611), ('كحسن', 0.6202388405799866), ('عبدالحسن', 0.6161733865737915), ('بوحسن', 0.6157495379447937), ('د/حسن', 0.6062546968460083)]\n"
     ]
    }
   ],
   "source": [
    "sample = w2v[\"حسن\"]\n",
    "print(sample.shape)\n",
    "#print(sample)\n",
    "print(w2v.most_similar(\"حسن\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import glob\n",
    "import codecs\n",
    "from keras.preprocessing.text import text_to_word_sequence\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "def get_stop_words():\n",
    "    path = \"/home/zmccoy/sandbox/sentiment-analysis-arabic/stop-words-list.txt\"\n",
    "    stop_words = []\n",
    "    with codecs.open(path, \"r\", encoding=\"utf-8\", errors=\"ignore\") as myfile:\n",
    "        stop_words = myfile.readlines()\n",
    "    stop_words = [word.strip() for word in stop_words]\n",
    "    return stop_words\n",
    "\n",
    "def get_label(path):\n",
    "    filename = path[path.rfind('/'):]\n",
    "    if filename.startswith(\"/pos\"):\n",
    "        return np.array([1, 0])\n",
    "    return np.array([0, 1])\n",
    "\n",
    "def get_text(path):\n",
    "    text = \"\"\n",
    "    text_words = []\n",
    "    # use codecs.open to ingnore non utf-8 encoded chars; instead of \"with open(path, 'r') as myfile:\"\n",
    "    with codecs.open(path, \"r\", encoding=\"utf-8\", errors=\"ignore\") as myfile:\n",
    "        text = myfile.read()\n",
    "    words = text.split(\" \")\n",
    "    stop_words = get_stop_words()\n",
    "    for word in words:\n",
    "        if word not in stop_words:\n",
    "            text_words.append(word)\n",
    "    return ' '.join(text_words)\n",
    "\n",
    "def get_text_sequences(texts):\n",
    "    tokenizer = Tokenizer(num_words=MAX_NB_WORDS)\n",
    "    tokenizer.fit_on_texts(texts)\n",
    "    sequences = tokenizer.texts_to_sequences(texts)\n",
    "    word_index = tokenizer.word_index\n",
    "    #print('Found %s unique tokens.' % len(word_index))\n",
    "    data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)\n",
    "    return data, word_index\n",
    "\n",
    "def get_data_encoded(path):\n",
    "    vectors = []\n",
    "    padded_vectors = []\n",
    "    unknown_words = []\n",
    "    labels = []\n",
    "    all_paths = glob.glob(os.path.join(path,'*.txt'))\n",
    "    np.random.shuffle(all_paths)\n",
    "    for path in all_paths:\n",
    "        # use codecs.open to ingnore non utf-8 encoded chars; instead of \"with open(path, 'r') as myfile:\"\n",
    "        with codecs.open(path, \"r\", encoding='utf-8', errors='ignore') as myfile:\n",
    "            text = myfile.read()\n",
    "            word_sequences = text_to_word_sequence(text,\n",
    "                                        filters='ا\\n\\ufeff\\xa0!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t',\n",
    "                                        #filters='!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n',\n",
    "                                        lower=True,\n",
    "                                        split=\" \")\n",
    "            vector = []\n",
    "            for word in word_sequences:\n",
    "                stop_words = get_stop_words()\n",
    "                if word not in stop_words:\n",
    "                    try:\n",
    "                        vector.append(w2v[word])\n",
    "                    except KeyError:\n",
    "                        # TODO: incrementally update w2v with OOVs (out-of-vocabularies)\n",
    "                        #print(\"' \" + str(word) + \"' not in vocabulary\")\n",
    "                        unknown_words.append(word)\n",
    "            vectors.append(vector)\n",
    "            label = get_label(path)\n",
    "            labels.append(label)\n",
    "    padded_vectors = sequence.pad_sequences(vectors, maxlen=MAX_SEQUENCE_LENGTH)\n",
    "    return np.array(padded_vectors), np.array(labels), unknown_words\n",
    "    \n",
    "def get_data(path):\n",
    "    texts = []\n",
    "    labels = []\n",
    "    all_paths = glob.glob(os.path.join(path,'*.txt'))\n",
    "    np.random.shuffle(all_paths)\n",
    "    for path in all_paths: \n",
    "        texts.append(get_text(path))\n",
    "        labels.append(get_label(path))\n",
    "    sequences, word_index = get_text_sequences(texts)\n",
    "    return sequences, np.array(labels), word_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['،', 'ء', 'ءَ', 'آ', 'آب', 'آذار', 'آض', 'آل', 'آمينَ', 'آناء', 'آنفا', 'آه', 'آهاً', 'آهٍ', 'آهِ', 'أ', 'أبدا', 'أبريل', 'أبو', 'أبٌ', 'أجل', 'أجمع', 'أحد', 'أخبر', 'أخذ', 'أخو', 'أخٌ', 'أربع', 'أربعاء', 'أربعة', 'أربعمئة', 'أربعمائة', 'أرى', 'أسكن', 'أصبح', 'أصلا', 'أضحى', 'أطعم', 'أعطى', 'أعلم', 'أغسطس', 'أفريل', 'أفعل به', 'أفٍّ', 'أقبل', 'أكتوبر', 'أل', 'ألا', 'ألف', 'ألفى', 'أم', 'أما', 'أمام', 'أمامك', 'أمامكَ', 'أمد', 'أمس', 'أمسى', 'أمّا', 'أن', 'أنا', 'أنبأ', 'أنت', 'أنتم', 'أنتما', 'أنتن', 'أنتِ', 'أنشأ', 'أنه', 'أنًّ', 'أنّى', 'أهلا', 'أو', 'أوت', 'أوشك', 'أول', 'أولئك', 'أولاء', 'أولالك', 'أوّهْ', 'أى', 'أي', 'أيا', 'أيار', 'أيضا', 'أيلول', 'أين', 'أيّ', 'أيّان', 'أُفٍّ', 'ؤ', 'إحدى', 'إذ', 'إذا', 'إذاً', 'إذما', 'إذن', 'إزاء', 'إلى', 'إلي', 'إليكم', 'إليكما', 'إليكنّ', 'إليكَ', 'إلَيْكَ', 'إلّا', 'إمّا', 'إن', 'إنَّ', 'إى', 'إياك', 'إياكم', 'إياكما', 'إياكن', 'إيانا', 'إياه', 'إياها', 'إياهم', 'إياهما', 'إياهن', 'إياي', 'إيهٍ', 'ئ', 'ا', 'ا?', 'ا?ى', 'االا', 'االتى', 'ابتدأ', 'ابين', 'اتخذ', 'اثر', 'اثنا', 'اثنان', 'اثني', 'اثنين', 'اجل', 'احد', 'اخرى', 'اخلولق', 'اذا', 'اربعة', 'اربعون', 'اربعين', 'ارتدّ', 'استحال', 'اصبح', 'اضحى', 'اطار', 'اعادة', 'اعلنت', 'اف', 'اكثر', 'اكد', 'الآن', 'الألاء', 'الألى', 'الا', 'الاخيرة', 'الان', 'الاول', 'الاولى', 'التى', 'التي', 'الثاني', 'الثانية', 'الحالي', 'الذاتي', 'الذى', 'الذي', 'الذين', 'السابق', 'الف', 'اللاتي', 'اللتان', 'اللتيا', 'اللتين', 'اللذان', 'اللذين', 'اللواتي', 'الماضي', 'المقبل', 'الوقت', 'الى', 'الي', 'اليه', 'اليها', 'اليوم', 'اما', 'امام', 'امس', 'امسى', 'ان', 'انبرى', 'انقلب', 'انه', 'انها', 'او', 'اول', 'اي', 'ايار', 'ايام', 'ايضا', 'ب', 'بؤسا', 'بإن', 'بئس', 'باء', 'بات', 'باسم', 'بان', 'بخٍ', 'بد', 'بدلا', 'برس', 'بسبب', 'بسّ', 'بشكل', 'بضع', 'بطآن', 'بعد', 'بعدا', 'بعض', 'بغتة', 'بل', 'بلى', 'بن', 'به', 'بها', 'بهذا', 'بيد', 'بين', 'بَسْ', 'بَلْهَ', 'ة', 'ت', 'تاء', 'تارة', 'تاسع', 'تانِ', 'تانِك', 'تبدّل', 'تجاه', 'تحت', 'تحوّل', 'تخذ', 'ترك', 'تسع', 'تسعة', 'تسعمئة', 'تسعمائة', 'تسعون', 'تسعين', 'تشرين', 'تعسا', 'تعلَّم', 'تفعلان', 'تفعلون', 'تفعلين', 'تكون', 'تلقاء', 'تلك', 'تم', 'تموز', 'تينك', 'تَيْنِ', 'تِه', 'تِي', 'ث', 'ثاء', 'ثالث', 'ثامن', 'ثان', 'ثاني', 'ثلاث', 'ثلاثاء', 'ثلاثة', 'ثلاثمئة', 'ثلاثمائة', 'ثلاثون', 'ثلاثين', 'ثم', 'ثمان', 'ثمانمئة', 'ثمانون', 'ثماني', 'ثمانية', 'ثمانين', 'ثمنمئة', 'ثمَّ', 'ثمّ', 'ثمّة', 'ج', 'جانفي', 'جدا', 'جعل', 'جلل', 'جمعة', 'جميع', 'جنيه', 'جوان', 'جويلية', 'جير', 'جيم', 'ح', 'حاء', 'حادي', 'حار', 'حاشا', 'حاليا', 'حاي', 'حبذا', 'حبيب', 'حتى', 'حجا', 'حدَث', 'حرى', 'حزيران', 'حسب', 'حقا', 'حمدا', 'حمو', 'حمٌ', 'حوالى', 'حول', 'حيث', 'حيثما', 'حين', 'حيَّ', 'حَذارِ', 'خ', 'خاء', 'خاصة', 'خال', 'خامس', 'خبَّر', 'خلا', 'خلافا', 'خلال', 'خلف', 'خمس', 'خمسة', 'خمسمئة', 'خمسمائة', 'خمسون', 'خمسين', 'خميس', 'د', 'دال', 'درهم', 'درى', 'دواليك', 'دولار', 'دون', 'دونك', 'ديسمبر', 'دينار', 'ذ', 'ذا', 'ذات', 'ذاك', 'ذال', 'ذانك', 'ذانِ', 'ذلك', 'ذهب', 'ذو', 'ذيت', 'ذينك', 'ذَيْنِ', 'ذِه', 'ذِي', 'ر', 'رأى', 'راء', 'رابع', 'راح', 'رجع', 'رزق', 'رويدك', 'ريال', 'ريث', 'رُبَّ', 'ز', 'زاي', 'زعم', 'زود', 'زيارة', 'س', 'ساء', 'سابع', 'سادس', 'سبت', 'سبتمبر', 'سبحان', 'سبع', 'سبعة', 'سبعمئة', 'سبعمائة', 'سبعون', 'سبعين', 'ست', 'ستة', 'ستكون', 'ستمئة', 'ستمائة', 'ستون', 'ستين', 'سحقا', 'سرا', 'سرعان', 'سقى', 'سمعا', 'سنة', 'سنتيم', 'سنوات', 'سوف', 'سوى', 'سين', 'ش', 'شباط', 'شبه', 'شتانَ', 'شخصا', 'شرع', 'شمال', 'شيكل', 'شين', 'شَتَّانَ', 'ص', 'صاد', 'صار', 'صباح', 'صبر', 'صبرا', 'صدقا', 'صراحة', 'صفر', 'صهٍ', 'صهْ', 'ض', 'ضاد', 'ضحوة', 'ضد', 'ضمن', 'ط', 'طاء', 'طاق', 'طالما', 'طرا', 'طفق', 'طَق', 'ظ', 'ظاء', 'ظل', 'ظلّ', 'ظنَّ', 'ع', 'عاد', 'عاشر', 'عام', 'عاما', 'عامة', 'عجبا', 'عدا', 'عدة', 'عدد', 'عدم', 'عدَّ', 'عسى', 'عشر', 'عشرة', 'عشرون', 'عشرين', 'عل', 'علق', 'علم', 'على', 'علي', 'عليك', 'عليه', 'عليها', 'علًّ', 'عن', 'عند', 'عندما', 'عنه', 'عنها', 'عوض', 'عيانا', 'عين', 'عَدَسْ', 'غ', 'غادر', 'غالبا', 'غدا', 'غداة', 'غير', 'غين', 'ـ', 'ف', 'فإن', 'فاء', 'فان', 'فانه', 'فبراير', 'فرادى', 'فضلا', 'فقد', 'فقط', 'فكان', 'فلان', 'فلس', 'فهو', 'فو', 'فوق', 'فى', 'في', 'فيفري', 'فيه', 'فيها', 'ق', 'قاطبة', 'قاف', 'قال', 'قام', 'قبل', 'قد', 'قرش', 'قطّ', 'قلما', 'قوة', 'ك', 'كأن', 'كأنّ', 'كأيّ', 'كأيّن', 'كاد', 'كاف', 'كان', 'كانت', 'كانون', 'كثيرا', 'كذا', 'كذلك', 'كرب', 'كسا', 'كل', 'كلتا', 'كلم', 'كلَّا', 'كلّما', 'كم', 'كما', 'كن', 'كى', 'كيت', 'كيف', 'كيفما', 'كِخ', 'ل', 'لأن', 'لا', 'لا سيما', 'لات', 'لازال', 'لاسيما', 'لام', 'لايزال', 'لبيك', 'لدن', 'لدى', 'لدي', 'لذلك', 'لعل', 'لعلَّ', 'لعمر', 'لقاء', 'لكن', 'لكنه', 'لكنَّ', 'للامم', 'لم', 'لما', 'لمّا', 'لن', 'له', 'لها', 'لهذا', 'لهم', 'لو', 'لوكالة', 'لولا', 'لوما', 'ليت', 'ليرة', 'ليس', 'ليسب', 'م', 'مئة', 'مئتان', 'ما', 'ما أفعله', 'ما انفك', 'ما برح', 'مائة', 'ماانفك', 'مابرح', 'مادام', 'ماذا', 'مارس', 'مازال', 'مافتئ', 'ماي', 'مايزال', 'مايو', 'متى', 'مثل', 'مذ', 'مرّة', 'مساء', 'مع', 'معاذ', 'معه', 'مقابل', 'مكانكم', 'مكانكما', 'مكانكنّ', 'مكانَك', 'مليار', 'مليم', 'مليون', 'مما', 'من', 'منذ', 'منه', 'منها', 'مه', 'مهما', 'ميم', 'ن', 'نا', 'نبَّا', 'نحن', 'نحو', 'نعم', 'نفس', 'نفسه', 'نهاية', 'نوفمبر', 'نون', 'نيسان', 'نيف', 'نَخْ', 'نَّ', 'ه', 'هؤلاء', 'ها', 'هاء', 'هاكَ', 'هبّ', 'هذا', 'هذه', 'هل', 'هللة', 'هلم', 'هلّا', 'هم', 'هما', 'همزة', 'هن', 'هنا', 'هناك', 'هنالك', 'هو', 'هي', 'هيا', 'هيهات', 'هيّا', 'هَؤلاء', 'هَاتانِ', 'هَاتَيْنِ', 'هَاتِه', 'هَاتِي', 'هَجْ', 'هَذا', 'هَذانِ', 'هَذَيْنِ', 'هَذِه', 'هَذِي', 'هَيْهات', 'و', 'و6', 'وأبو', 'وأن', 'وا', 'واحد', 'واضاف', 'واضافت', 'واكد', 'والتي', 'والذي', 'وان', 'واهاً', 'واو', 'واوضح', 'وبين', 'وثي', 'وجد', 'وراءَك', 'ورد', 'وعلى', 'وفي', 'وقال', 'وقالت', 'وقد', 'وقف', 'وكان', 'وكانت', 'ولا', 'ولايزال', 'ولكن', 'ولم', 'وله', 'وليس', 'ومع', 'ومن', 'وهب', 'وهذا', 'وهو', 'وهي', 'وَيْ', 'وُشْكَانَ', 'ى', 'ي', 'ياء', 'يفعلان', 'يفعلون', 'يكون', 'يلي', 'يمكن', 'يمين', 'ين', 'يناير', 'يوان', 'يورو', 'يوليو', 'يوم', 'يونيو', 'ّأيّان']\n"
     ]
    }
   ],
   "source": [
    "print(get_stop_words())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1800, 85)\n",
      "(1800, 2)\n",
      "6447\n",
      "(200, 85)\n",
      "(200, 2)\n",
      "1599\n",
      "7543\n"
     ]
    }
   ],
   "source": [
    "word_index_train = {}\n",
    "word_index_valid = {}\n",
    "word_index_test = {}\n",
    "X_train, y_train, word_index_train = get_data(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Twitter/Combined/train\")\n",
    "#X_train, y_train, word_index_train = get_data(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Erik/train\")\n",
    "print(X_train.shape)\n",
    "print(y_train.shape)\n",
    "print(len(word_index_train))\n",
    "X_valid, y_valid, word_index_valid = get_data(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Twitter/Combined/valid\")\n",
    "#X_valid, y_valid, word_index_valid = get_data(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Erik/valid\")\n",
    "print(X_valid.shape)\n",
    "print(y_valid.shape)\n",
    "print(len(word_index_valid))\n",
    "#X_test, y_test, word_index_test = get_data(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Erik/test\")\n",
    "#print(X_test.shape)\n",
    "#print(y_test.shape)\n",
    "#print(len(word_index_test))\n",
    "word_index = {}\n",
    "for wi in (word_index_train, word_index_valid, word_index_test): word_index.update(wi)\n",
    "print(len(word_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1800, 85, 300)\n",
      "(1800, 2)\n",
      "3958\n",
      "(200, 85, 300)\n",
      "(200, 2)\n",
      "969\n"
     ]
    }
   ],
   "source": [
    "X_train_enc, y_train, unk_train = get_data_encoded(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Twitter/Combined/train\")\n",
    "#X_train_enc, y_train, unk_train = get_data_encoded(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Erik/train\")\n",
    "print(X_train_enc.shape)\n",
    "print(y_train.shape)\n",
    "print(len(unk_train))\n",
    "X_valid_enc, y_valid, unk_valid = get_data_encoded(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Twitter/Combined/valid\")\n",
    "#X_valid_enc, y_valid, unk_valid = get_data_encoded(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Erik/valid\")\n",
    "print(X_valid_enc.shape)\n",
    "print(y_valid.shape)\n",
    "print(len(unk_valid))\n",
    "#X_test_enc, y_test, unk_test = get_data_encoded(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Twitter/Combined/test\")\n",
    "#X_test_enc, y_test, unk_test = get_data_encoded(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Erik/test\")\n",
    "#print(X_test_enc.shape)\n",
    "#print(y_test.shape)\n",
    "#print(len(unk_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7544, 300)\n"
     ]
    }
   ],
   "source": [
    "def create_embedding_matrix(word_index):\n",
    "    nb_words = min(MAX_NB_WORDS, len(word_index))+1\n",
    "    embedding_matrix = np.zeros((nb_words, EMBEDDING_DIM))\n",
    "    for word, ii in word_index.items():\n",
    "        if word in w2v.vocab:\n",
    "            embedding_matrix[ii] = w2v.word_vec(word)\n",
    "    return embedding_matrix\n",
    "\n",
    "embedding_matrix = create_embedding_matrix(word_index)\n",
    "print(embedding_matrix.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "610977\n",
      "300\n"
     ]
    }
   ],
   "source": [
    "embedding_layer = w2v.get_keras_embedding(train_embeddings=False)\n",
    "print(embedding_layer.input_dim)\n",
    "print(embedding_layer.output_dim)\n",
    "#embedding_layer.set_weights(embedding_matrix)\n",
    "#embedding_layer.get_weights()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "def get_callbacks(weights_filepath):\n",
    "    checkpoint = ModelCheckpoint(filepath=weights_filepath, monitor='val_loss', verbose=1, save_best_only=True, mode='min')\n",
    "    early_stopping = EarlyStopping(monitor='val_loss', patience=4)\n",
    "    callbacks = [checkpoint, early_stopping]\n",
    "    return callbacks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_model_pretrained_embedding(trainable=False):\n",
    "    weights_filename = \"weights/keras-lstm-pretrained-embedding.h5\"\n",
    "    if trainable is True:\n",
    "        weights_filename = \"weights/keras-lstm-pretrained-embedding-trainable.h5\"\n",
    "    #embedding_layer = Embedding(len(word_index)+1, EMBEDDING_DIM, weights=[embedding_matrix], input_length=MAX_SEQUENCE_LENGTH, trainable=trainable)\n",
    "    embedding_layer = w2v.get_keras_embedding(train_embeddings=trainable)\n",
    "    return get_model_embedding(embedding_layer), weights_filename\n",
    "    \n",
    "def get_model_from_scratch_embedding():\n",
    "    embedding_layer = Embedding(len(word_index)+1, EMBEDDING_DIM, input_length=MAX_SEQUENCE_LENGTH)\n",
    "    return get_model_embedding(embedding_layer), \"weights/keras-lstm-from-scratch-embedding.h5\"\n",
    "    \n",
    "def get_model_embedding(embedding_layer):\n",
    "    sentence_indices = Input(shape=(MAX_SEQUENCE_LENGTH,),dtype='int32')\n",
    "    embeddings = embedding_layer(sentence_indices)   \n",
    "    # Propagate the embeddings through an LSTM layer with 128-dimensional hidden state\n",
    "    # Returned output should be a batch of sequences.\n",
    "    X = LSTM(128, return_sequences=True)(embeddings)\n",
    "    X = Dropout(0.5)(X)\n",
    "    # Returned output should be a single hidden state, not a batch of sequences.\n",
    "    X = LSTM(128)(X)\n",
    "    X = Dropout(0.5)(X)\n",
    "    X = Dense(2)(X)\n",
    "    X = Activation('softmax')(X)\n",
    "    model = Model(inputs=sentence_indices,outputs=X)\n",
    "    return model  \n",
    "\n",
    "def get_model_no_embedding():\n",
    "    model = Sequential()\n",
    "    model.add(LSTM(128, return_sequences=True, input_shape=(MAX_SEQUENCE_LENGTH,EMBEDDING_DIM)))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(LSTM(128))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(2))\n",
    "    model.add(Activation('softmax'))\n",
    "    return model, \"weights/keras-lstm-no-embedding.h5\"\n",
    "\n",
    "def get_model_pretrained_embedding_bidirectional():\n",
    "    sentence_indices = Input(shape=(MAX_SEQUENCE_LENGTH,),dtype='int32')\n",
    "    embedding_layer = Embedding(len(word_index)+1, EMBEDDING_DIM, weights=[embedding_matrix], input_length=MAX_SEQUENCE_LENGTH, trainable=True)\n",
    "    embeddings = embedding_layer(sentence_indices)   \n",
    "    X = Bidirectional(LSTM(60, return_sequences=True, dropout=0.1, recurrent_dropout=0.1))(embeddings)\n",
    "    X = GlobalMaxPool1D()(X)\n",
    "    X = Dropout(0.1)(X)\n",
    "    X = Dense(50)(X)\n",
    "    X = Activation(\"relu\")(X)\n",
    "    X = Dropout(0.1)(X)\n",
    "    X = Dense(2)(X)\n",
    "    X = Activation('softmax')(X)\n",
    "    model = Model(inputs=sentence_indices,outputs=X)\n",
    "    return model, \"weights/keras-lstm-pretrained-embedding-bidirectional-trainable.h5\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 3 approaches:\n",
    "\n",
    " 1. Learn embedding from scratch - simply add an Embedding layer to your model\n",
    " 1. Fine tune learned embeddings - this involves setting word2vec / GloVe vectors as your Embedding layer's weights.\n",
    " 1. Use word word2vec / Glove word vectors as inputs to your model, instead of one-hot encoding.\n",
    " \n",
    " \n",
    " - (Bidirectional) -see: https://www.kaggle.com/sbongo/do-pretrained-embeddings-give-you-the-extra-edge\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#1\n",
    "model_from_scratch, weights_from_scratch = get_model_from_scratch_embedding()\n",
    "#2A\n",
    "model_pt_frozen, weights_pt_frozen = get_model_pretrained_embedding()\n",
    "#2B\n",
    "model_pt_trainable, weights_pt_trainable = get_model_pretrained_embedding(trainable=True)\n",
    "#3\n",
    "model_no_embedding, weights_no_embedding = get_model_no_embedding()\n",
    "# Bidirectional\n",
    "model_bidirectional, weights_bidirectional = get_model_pretrained_embedding_bidirectional()\n",
    "#model.load_weights(weights_filepath)\n",
    "#model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_from_scratch.compile(loss=\"categorical_crossentropy\", optimizer='adam', metrics=['accuracy'])\n",
    "model_pt_frozen.compile(loss=\"categorical_crossentropy\", optimizer='adam', metrics=['accuracy'])\n",
    "model_pt_trainable.compile(loss=\"categorical_crossentropy\", optimizer='adam', metrics=['accuracy'])\n",
    "model_no_embedding.compile(loss=\"categorical_crossentropy\", optimizer='adam', metrics=['accuracy'])\n",
    "model_bidirectional.compile(loss=\"categorical_crossentropy\", optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1800 samples, validate on 200 samples\n",
      "Epoch 1/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6968 - acc: 0.4905\n",
      "Epoch 00001: val_loss improved from inf to 0.69453, saving model to weights/keras-lstm-from-scratch-embedding.h5\n",
      "1800/1800 [==============================] - 26s 14ms/step - loss: 0.6967 - acc: 0.4906 - val_loss: 0.6945 - val_acc: 0.4900\n",
      "Epoch 2/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6504 - acc: 0.6540\n",
      "Epoch 00002: val_loss did not improve\n",
      "1800/1800 [==============================] - 23s 13ms/step - loss: 0.6507 - acc: 0.6533 - val_loss: 0.8813 - val_acc: 0.4500\n",
      "Epoch 3/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.2467 - acc: 0.9057\n",
      "Epoch 00003: val_loss did not improve\n",
      "1800/1800 [==============================] - 27s 15ms/step - loss: 0.2461 - acc: 0.9061 - val_loss: 1.9009 - val_acc: 0.4750\n",
      "Epoch 4/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.0728 - acc: 0.9782\n",
      "Epoch 00004: val_loss did not improve\n",
      "1800/1800 [==============================] - 35s 20ms/step - loss: 0.0725 - acc: 0.9783 - val_loss: 2.0343 - val_acc: 0.4600\n",
      "Epoch 5/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.0286 - acc: 0.9905\n",
      "Epoch 00005: val_loss did not improve\n",
      "1800/1800 [==============================] - 32s 18ms/step - loss: 0.0285 - acc: 0.9906 - val_loss: 3.0507 - val_acc: 0.4700\n",
      "Train on 1800 samples, validate on 200 samples\n",
      "Epoch 1/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.7059 - acc: 0.5089\n",
      "Epoch 00001: val_loss improved from inf to 0.69471, saving model to weights/keras-lstm-pretrained-embedding.h5\n",
      "1800/1800 [==============================] - 27s 15ms/step - loss: 0.7063 - acc: 0.5078 - val_loss: 0.6947 - val_acc: 0.5000\n",
      "Epoch 2/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6908 - acc: 0.5218\n",
      "Epoch 00002: val_loss did not improve\n",
      "1800/1800 [==============================] - 33s 18ms/step - loss: 0.6906 - acc: 0.5222 - val_loss: 0.6963 - val_acc: 0.4950\n",
      "Epoch 3/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6883 - acc: 0.5474\n",
      "Epoch 00003: val_loss did not improve\n",
      "1800/1800 [==============================] - 24s 14ms/step - loss: 0.6886 - acc: 0.5472 - val_loss: 0.7043 - val_acc: 0.4900\n",
      "Epoch 4/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6837 - acc: 0.5642\n",
      "Epoch 00004: val_loss did not improve\n",
      "1800/1800 [==============================] - 23s 13ms/step - loss: 0.6835 - acc: 0.5644 - val_loss: 0.7080 - val_acc: 0.4750\n",
      "Epoch 5/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6627 - acc: 0.5915\n",
      "Epoch 00005: val_loss did not improve\n",
      "1800/1800 [==============================] - 30s 17ms/step - loss: 0.6621 - acc: 0.5922 - val_loss: 0.7405 - val_acc: 0.4700\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/zmccoy/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor with 183293100 elements. This may consume a large amount of memory.\n",
      "  \"This may consume a large amount of memory.\" % num_elements)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1800 samples, validate on 200 samples\n",
      "Epoch 1/10\n",
      "1792/1800 [============================>.] - ETA: 1s - loss: 0.7044 - acc: 0.5106\n",
      "Epoch 00001: val_loss improved from inf to 0.69446, saving model to weights/keras-lstm-pretrained-embedding-trainable.h5\n",
      "1800/1800 [==============================] - 238s 132ms/step - loss: 0.7045 - acc: 0.5106 - val_loss: 0.6945 - val_acc: 0.4650\n",
      "Epoch 2/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6834 - acc: 0.5575\n",
      "Epoch 00002: val_loss did not improve\n",
      "1800/1800 [==============================] - 211s 117ms/step - loss: 0.6833 - acc: 0.5578 - val_loss: 0.7104 - val_acc: 0.5000\n",
      "Epoch 3/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6071 - acc: 0.6847\n",
      "Epoch 00003: val_loss did not improve\n",
      "1800/1800 [==============================] - 204s 113ms/step - loss: 0.6070 - acc: 0.6850 - val_loss: 0.8493 - val_acc: 0.4550\n",
      "Epoch 4/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.3165 - acc: 0.8577\n",
      "Epoch 00004: val_loss did not improve\n",
      "1800/1800 [==============================] - 171s 95ms/step - loss: 0.3153 - acc: 0.8583 - val_loss: 1.4280 - val_acc: 0.4750\n",
      "Epoch 5/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.1248 - acc: 0.9542\n",
      "Epoch 00005: val_loss did not improve\n",
      "1800/1800 [==============================] - 172s 96ms/step - loss: 0.1255 - acc: 0.9539 - val_loss: 2.2088 - val_acc: 0.4500\n",
      "Train on 1800 samples, validate on 200 samples\n",
      "Epoch 1/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6925 - acc: 0.4989\n",
      "Epoch 00001: val_loss improved from inf to 0.67816, saving model to weights/keras-lstm-no-embedding.h5\n",
      "1800/1800 [==============================] - 28s 16ms/step - loss: 0.6926 - acc: 0.4978 - val_loss: 0.6782 - val_acc: 0.5000\n",
      "Epoch 2/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6729 - acc: 0.5809\n",
      "Epoch 00002: val_loss did not improve\n",
      "1800/1800 [==============================] - 29s 16ms/step - loss: 0.6742 - acc: 0.5794 - val_loss: 0.7221 - val_acc: 0.5650\n",
      "Epoch 3/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6461 - acc: 0.6278\n",
      "Epoch 00003: val_loss did not improve\n",
      "1800/1800 [==============================] - 24s 13ms/step - loss: 0.6454 - acc: 0.6278 - val_loss: 0.6875 - val_acc: 0.4850\n",
      "Epoch 4/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6009 - acc: 0.6356\n",
      "Epoch 00004: val_loss did not improve\n",
      "1800/1800 [==============================] - 26s 15ms/step - loss: 0.6010 - acc: 0.6350 - val_loss: 0.7564 - val_acc: 0.5750\n",
      "Epoch 5/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.5689 - acc: 0.6652\n",
      "Epoch 00005: val_loss did not improve\n",
      "1800/1800 [==============================] - 24s 13ms/step - loss: 0.5686 - acc: 0.6650 - val_loss: 0.7912 - val_acc: 0.4800\n",
      "Train on 1800 samples, validate on 200 samples\n",
      "Epoch 1/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6962 - acc: 0.5195\n",
      "Epoch 00001: val_loss improved from inf to 0.70411, saving model to weights/keras-lstm-pretrained-embedding-bidirectional-trainable.h5\n",
      "1800/1800 [==============================] - 27s 15ms/step - loss: 0.6965 - acc: 0.5183 - val_loss: 0.7041 - val_acc: 0.4950\n",
      "Epoch 2/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.6724 - acc: 0.6110\n",
      "Epoch 00002: val_loss did not improve\n",
      "1800/1800 [==============================] - 21s 12ms/step - loss: 0.6720 - acc: 0.6122 - val_loss: 0.7329 - val_acc: 0.4600\n",
      "Epoch 3/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.5322 - acc: 0.7673\n",
      "Epoch 00003: val_loss did not improve\n",
      "1800/1800 [==============================] - 21s 12ms/step - loss: 0.5315 - acc: 0.7672 - val_loss: 1.0324 - val_acc: 0.4800\n",
      "Epoch 4/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.2295 - acc: 0.9241\n",
      "Epoch 00004: val_loss did not improve\n",
      "1800/1800 [==============================] - 21s 12ms/step - loss: 0.2313 - acc: 0.9233 - val_loss: 1.6214 - val_acc: 0.4850\n",
      "Epoch 5/10\n",
      "1792/1800 [============================>.] - ETA: 0s - loss: 0.1034 - acc: 0.9654\n",
      "Epoch 00005: val_loss did not improve\n",
      "1800/1800 [==============================] - 25s 14ms/step - loss: 0.1032 - acc: 0.9656 - val_loss: 2.1360 - val_acc: 0.4450\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f8bb7b7bef0>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 32\n",
    "epochs = 10\n",
    "model_from_scratch.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, validation_data=(X_valid, y_valid), callbacks=get_callbacks(weights_from_scratch))\n",
    "model_pt_frozen.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, validation_data=(X_valid, y_valid), callbacks=get_callbacks(weights_pt_frozen))\n",
    "model_pt_trainable.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, validation_data=(X_valid, y_valid), callbacks=get_callbacks(weights_pt_trainable)) \n",
    "history = model_no_embedding.fit(X_train_enc, y_train, batch_size=batch_size, epochs=epochs, validation_data=(X_valid_enc, y_valid), callbacks=get_callbacks(weights_no_embedding))\n",
    "model_bidirectional.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, validation_data=(X_valid, y_valid), callbacks=get_callbacks(weights_bidirectional)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmczfX3wPHXMcaWklC2rCnZEpNo\n+SaVytZClihpkUolbbSiTdu37askWULoR6FS2pRWDBXZ9xqUoaTFNpzfH+dOpjFj7jD3fu7MPc/H\n4z7c+/l87tzjMvfc93beoqo455xzB1Io6ACcc87FPk8WzjnncuTJwjnnXI48WTjnnMuRJwvnnHM5\n8mThnHMuR54snDtIIlJNRFRECodx7VUi8kU04nIuEjxZuLggImtFZJeIlM10/LvQB361YCLLXdJx\nLiieLFw8WQN0SX8gIvWB4sGF41z+4cnCxZMxwJUZHncHXst4gYiUEpHXRCRVRNaJyH0iUih0LkFE\nnhKRzSKyGmidxXNfFZGNIrJeRB4WkYRDCVhEiorIsyKyIXR7VkSKhs6VFZF3RGSriPwqIp9niPXu\nUAx/iMgyETnnUOJwzpOFiyffAEeIyImhD/FOwNhM17wAlAJqAGdhyaVH6Nx1QBvgZCAJ6JDpuaOB\nNOC40DUtgWsPMeZ7gaZAQ+AkoAlwX+jc7UAKUA44BrgHUBE5AegNnKKqhwPnA2sPMQ4X5zxZuHiT\n3ro4D1gKrE8/kSGB9FfVP1R1LfA0cEXoko7As6r6k6r+CjyW4bnHABcCfVT1L1XdBDwDdD7EeLsC\ng1R1k6qmAgMzxLMbqABUVdXdqvq5WrG3PUBRoI6IJKrqWlVddYhxuDjnycLFmzHA5cBVZOqCAsoC\nRYB1GY6tAyqF7lcEfsp0Ll1VIBHYGOoW2gq8DBx9iPFWzCKeiqH7TwIrgQ9EZLWI9ANQ1ZVAH2AA\nsElEJohIRZw7BJ4sXFxR1XXYQHcr4M1Mpzdj39arZjhWhX2tj43AsZnOpfsJ2AmUVdUjQ7cjVLXu\nIYa8IYt4NoT+Ln+o6u2qWgNoC/RNH5tQ1ddV9YzQcxV4/BDjcHHOk4WLR9cALVT1r4wHVXUP8Abw\niIgcLiJVgb7sG9d4A7hFRCqLSGmgX4bnbgQ+AJ4WkSNEpJCI1BSRs3IRV1ERKZbhVggYD9wnIuVC\n034fSI9HRNqIyHEiIsA2rPtpj4icICItQgPhO4DtoXPOHTRPFi7uqOoqVU3O5vTNwF/AauAL4HVg\nROjcK8AM4HtgPvu3TK7EurEWA78Bk7AxhXD9iX2wp99aAA8DycACYGHodR8OXV8L+Cj0vK+BF1X1\nU2y8YjDWUvoZ6wq7JxdxOLcf8c2PnHPO5cRbFs4553LkycI551yOPFk455zLUUSThYhcECo1sDJ9\nDnim81VEZKaIfCsiC0SkVYZz/UPPWyYi50cyTueccwcWsQHu0GrY5dhK2RRgLtBFVRdnuGYY8K2q\nviQidYDpqlotdH88VtqgIjbj4/jQ1MYslS1bVqtVqxaRv4tzzhVU8+bN26yq5XK6LpIlkZsAK1V1\nNYCITAAuwqYVplPgiND9UoQWG4Wum6CqO4E1IrIy9PO+zu7FqlWrRnJydrMhnXPOZUVE1uV8VWS7\noSrx79IIKewrm5BuANBNRFKA6dgc93Cfi4j0FJFkEUlOTU3Nq7idc85lEslkIVkcy9zn1QUYpaqV\nsfILY0KrVsN5Lqo6TFWTVDWpXLkcW1HOOecOUiS7oVL4dx2dyuzrZkp3DXABgKp+LSLFsGJu4TzX\nOedclEQyWcwFaolIdawQW2es2mdGPwLnAKNE5ESgGJAKTANeF5H/YgPctYA5uQ1g9+7dpKSksGPH\njoP/W7j9FCtWjMqVK5OYmBh0KM65KIlYslDVNBHpjdXSSQBGqOoiERkEJKvqNGzzlldE5Dasm+mq\nUD3+RSLyBjYYngbcdKCZUNlJSUnh8MMPp1q1alitNXeoVJUtW7aQkpJC9erVgw7HORclEd0gXlWn\nYwPXGY89kOH+YuD0bJ77CPDIobz+jh07PFHkMRGhTJky+IQC5+JLgV/B7Yki7/l76lz8KfDJwjnn\nCqodO2DcOBg2LPKv5ckigrZs2ULDhg1p2LAh5cuXp1KlSv883rVrV1g/o0ePHixbtizs1xw+fDh9\n+vQ52JCdc/nAihVw551QuTJ06wYjR0Kkd5uI6JhFvCtTpgzfffcdAAMGDKBkyZLccccd/7pGVVFV\nChXKOm+PHDky4nE652Lf7t0wbRoMHQoffQQJCXDxxdCrF7RoAZHuHfaWRQBWrlxJvXr16NWrF40a\nNWLjxo307NmTpKQk6taty6BBg/659owzzuC7774jLS2NI488kn79+nHSSSfRrFkzNm3aFPZrjh07\nlvr161OvXj3uucc2TUtLS+OKK6745/jzzz8PwDPPPEOdOnU46aST6NatW97+5Z1zufLjj3D//VCl\nCnToAMuWwUMP2fFJk+DccyGb75p5Kr5aFs2b73+sY0e48Ub4+29o1Wr/81ddZbfNm+1fKqNPPz3o\nUBYvXszIkSMZOnQoAIMHD+aoo44iLS2Ns88+mw4dOlCnTp1/Pef333/nrLPOYvDgwfTt25cRI0bQ\nr99+xXz3k5KSwn333UdycjKlSpXi3HPP5Z133qFcuXJs3ryZhQsXArB161YAnnjiCdatW0eRIkX+\nOeaci549e+D9960VMX26dTG1amWtiAsvtFZFtHnLIiA1a9bklFNO+efx+PHjadSoEY0aNWLJkiUs\nXrx4v+cUL16cCy+8EIDGjRuzdu3asF5r9uzZtGjRgrJly5KYmMjll1/OrFmzOO6441i2bBm33nor\nM2bMoFSpUgDUrVuXbt26MW7cOF9451wUbdwIjzwCNWpAmzYwdy707w9r1sA779ixIBIFxFvL4kAt\ngRIlDny+bNlDaklkdthhh/1zf8WKFTz33HPMmTOHI488km7dumW56rxIkSL/3E9ISCAtLS2s18qu\nDH2ZMmVYsGAB7733Hs8//zyTJ09m2LBhzJgxg88++4ypU6fy8MMP88MPP5AQ1P9Q5wq4vXvhk0/g\n5ZdhyhRIS7Oupaefhosuglj5vuYtixiwbds2Dj/8cI444gg2btzIjBkz8vTnN23alJkzZ7JlyxbS\n0tKYMGECZ511Fqmpqagql112GQMHDmT+/Pns2bOHlJQUWrRowZNPPklqaip///13nsbjnLOe7aee\nghNOgPPOg5kzoU8fWL4cPvzQer1jJVFAvLUsYlSjRo2oU6cO9erVo0aNGpx+epaL2sP26quvMmnS\npH8eJycnM2jQIJo3b46q0rZtW1q3bs38+fO55pprUFVEhMcff5y0tDQuv/xy/vjjD/bu3cvdd9/N\n4Ycffqh/ReccNvbw5Zc2FvF//we7dsEZZ8CAAdC+PRQrFnSE2YvYTnnRlpSUpJk3P1qyZAknnnhi\nQBEVbP7eOhe+33+HMWMsSSxaBEccAVdeCddfD/XqBRubiMxT1aScrvOWhXPORUhysiWI8eNtwmVS\nEgwfDp07Q4Zhy3zBk4VzzuWhv/6y5DB0KMybZ3NnLr/cWhFJOX5/j12eLJxzLg8sXGgzmsaMgW3b\nrHtpyBDo2hVCs9LzNU8Wzjl3kHbssFXUQ4fawHXRorbOt1cvaNYs8iU4osmThXPO5dLy5daKGDUK\nfv0VatWydRHdu0OZMkFHFxmeLJxzLgy7dsHUqdaK+OQTKFwYLrnEWhFnn12wWhFZ8UV5EdS8efP9\nFtg9++yz3HjjjQd8XsmSJQHYsGEDHTLXo8rwszNPFT7QcefcwVm7Fu691wr5dewIq1ZZSY6ffoI3\n3ohOxddY4Mkigrp06cKECRP+dWzChAl06dIlrOdXrFjxX4vrnHPRsWcPvP02tG5tdZoGD4ZTT7Wi\nfqtWwT33QPnyQUcZXZ4sIqhDhw6888477Ny5E4C1a9eyYcMGzjjjDP7880/OOeccGjVqRP369Zk6\ndep+z1+7di31Qit2tm/fTufOnWnQoAGdOnVi+/btYcexY8cOevToQf369Tn55JOZOXMmAIsWLaJJ\nkyY0bNiQBg0asGLFCv766y9at27NSSedRL169Zg4cWIevBPO5Q8bNlj57+rVoV07+PZbKw++dq11\nQQVV8TUWxM2YRZ8+ENqHKM80bAjPPpv9+TJlytCkSRPef/99LrroIiZMmECnTp0QEYoVK8Zbb73F\nEUccwebNm2natCnt2rXLdn/rl156iRIlSrBgwQIWLFhAo0aNwo5zyJAhACxcuJClS5fSsmVLli9f\nztChQ7n11lvp2rUru3btYs+ePUyfPp2KFSvy7rvvAlYW3bmCbO9e+PhjG4uYOtVaFS1bwnPPWZXX\nWKrPFCRvWURYxq6ojF1Qqso999xDgwYNOPfcc1m/fj2//PJLtj9n1qxZ/2xE1KBBAxo0aBB2DF98\n8QVXXHEFALVr16Zq1aosX76cZs2a8eijj/L444+zbt06ihcvTv369fnoo4+4++67+fzzz/8pW+5c\nQZOaCk8+Cccfb8lh1iy4/XbbsnTGDBu89kSxT9y0LA7UAoikiy++mL59+zJ//ny2b9/+T4tg3Lhx\npKamMm/ePBITE6lWrVqWZckzyq7VkZPs6n9dfvnlnHrqqbz77rucf/75DB8+nBYtWjBv3jymT59O\n//79admyJQ888MBBva5zsUYVvvjCWhGTJtkMp//8x7qeLr3U1km4rMVNsghKyZIlad68OVdfffW/\nBrZ///13jj76aBITE5k5cybr1q074M/5z3/+w7hx4zj77LP54YcfWLBgQdgxpD+3RYsWLF++nB9/\n/JETTjiB1atXU6NGDW655RZWr17NggULqF27NkcddRTdunWjZMmSjBo16mD/6s7FjK1b4bXXLEks\nWWIrqnv1shIcmTakdNnwZBEFXbp04dJLL/3XzKiuXbvStm1bkpKSaNiwIbVr1z7gz7jhhhvo0aMH\nDRo0oGHDhjRp0iTba1u3bv3PDnfNmjVjzJgx9OrVi/r161O4cGFGjRpF0aJFmThxImPHjiUxMZHy\n5cvzwAMPMHfuXO68804KFSpEYmIiL730Ut68Cc5FmartNDd0KEyYANu3Q5MmMGIEdOpkNZtc+LxE\nuTso/t66WPXnn/D665Ykvv3Wqrt27WqtiFzMC4kbXqLcORdXvv/eSnCMHQt//AENGsBLL1nF1yOO\nCDq6/M+ThXMu39q+3XacGzoUvv7adprr1MnGI049NT5WVkdLgU8W6VuGurxTULouXf61dKm1IkaP\nht9+s32sn3nGdp876qigoyuYCnSyKFasGFu2bKFMmTKeMPKIqrJlyxaKxfJmwa5A2rUL3nrLWhGf\nfmprIC691FoRZ53lrYhIK9DJonLlyqSkpJCamhp0KAVKsWLFqFy5ctBhuDixZg0MG2azmDZtgmrV\n4LHHoEcPOOaYoKOLHwU6WSQmJlK9evWgw3DO5VJaGrz7rrUiZsywVkO7dtaKOO88KOS1J6KuQCcL\n51z+ogrPP29lONavh0qV4MEH4ZprwBuzwfJk4ZyLCX/+aV1LkybZHhFDhliJ8ML+KRUT/J/BORe4\nVavg4oth8WJ46ino29cHrGONJwvnXKA++AA6d7bk8P77NibhYk9Eh4lE5AIRWSYiK0WkXxbnnxGR\n70K35SKyNcO5PRnOTYtknM656FO1VsSFF9p4xNy5nihiWcRaFiKSAAwBzgNSgLkiMk1VF6dfo6q3\nZbj+ZuDkDD9iu6o2jFR8zrng/P03XHstjB8Pl10GI0daDScXuyLZsmgCrFTV1aq6C5gAXHSA67sA\n4yMYj3MuBqxdC6efbpVgH3sMJk70RJEfRDJZVAJ+yvA4JXRsPyJSFagOfJLhcDERSRaRb0Tk4mye\n1zN0TbIvvHMu9n3yCSQl2UK7d9+Ffv18IDu/iGSyyOq/QHZFhToDk1R1T4ZjVUJlcy8HnhWRmvv9\nMNVhqpqkqknlypU79IidcxGhartVtmxpq67nzrWxCpd/RDJZpADHZnhcGdiQzbWdydQFpaobQn+u\nBj7l3+MZzrl8Yvt26N4dbrsN2raFb76BWrWCjsrlViSTxVyglohUF5EiWELYb1aTiJwAlAa+znCs\ntIgUDd0vC5wOLM78XOdcbPvpJzjzTBgzBgYNgsmT4fDDg47KHYyIzYZS1TQR6Q3MABKAEaq6SEQG\nAcmqmp44ugAT9N91r08EXhaRvVhCG5xxFpVzLvbNmgUdOsCOHTBtmrUqXP5VoLdVdc5Fnyq8+CL0\n6QM1asDUqZDDFvMuQOFuq+q1G51zeWbnTls/0bs3XHABzJnjiaKg8GThnMsT69fbJkQjRsD991uL\nolSpoKNyecVrQznnDtlXX0H79lY5dvJk28HOFSzesnDOHZJhw6B5cyhZ0qbFeqIomDxZOOcOyq5d\ntnPd9dfDOefY+ETdukFH5SLFk4VzLtd+/tk2KHr5ZSvZ8c47ULp00FG5SPIxC+dcrsyZA5dcAlu3\nWhHAjh2DjshFg7csnHNhGznSVmQXLQpff+2JIp54snDO5Wj3brj5Zrj6aksWc+dCgwZBR+WiyZOF\nc+6ANm2Cc8+F//0Pbr/dtj4tUyboqFy0+ZiFcy5b8+bZ+ERqKowdC127Bh2RC4q3LJxzWRo7Fs44\nwzYn+vJLTxTxzpOFc+5f0tKgb1+44gpo2hSSk6FRo6CjckHzbijn3D82b4ZOnWz701tugaeegsTE\noKNyscCThXMOgO+/h4svho0bbYrsVVcFHZGLJd4N5ZxjwgRo1symyM6a5YnC7c+ThXNxbM8euPtu\n6NIFGje28YkmTYKOysUi74ZyubZzp/VjF/KvGvnar79akvjgA7jhBnj2WShSJOioXKzyX3cXtp9/\nhltvtQ1tjjsOHnsMfvkl6KjcwVi4EE45BWbOtBLjL77oicIdmCcLl6PUVLjzTttPecgQqwdUrRrc\ncw9UrgyXXQYffwx79wYdqQvH5Mk2PrF9O3z2GVx3XdARufzAk4XL1m+/wX33WZJ4+mnbCW3pUnjt\nNZtauXSpTa/85BMrB3HCCfDkk5ZcXOzZu9f+PTt0gPr1bXyiWbOgo3L5hScLt59t22DQIKheHR55\nBFq1gkWLYMwY635Kd8IJlkTWr7fVvuXLw113WWujSxf71qoa3N/D7bN1K7RrZ/+e11wDn34KFSsG\nHZXLTzxZuH/89RcMHmxJ4sEHbavM776zPQtOPDH75xUrZqUgPv8cfvjBdk97/317fp068MwzNpjq\ngrFkCZx6KsyYYWMTr7xiJcadyw1PFo7t2+G//7Uk0b+/fbDMnQtTpsBJJ+XuZ9WtC889Z62NUaNs\n97S+fe1b7BVXwBdfeGsjmqZNs3/PrVutu/CGG6zWk3O55ckiju3caQPWNWta6ekGDeCrr2D6dEhK\nOrSfXaIEdO9uP+/7763rY9o02wuhfn144QX7AHORsXcvDBwIF11k3YXJyfbeO3ewPFnEod27rSvi\n+OOhd28bh/j0U/joo8gMeDZoYElpwwYYPtwSyS23WGujRw/45htvbeSlbdvg0kthwABL2LNmwbHH\nBh2Vy+88WcSRtDQYPRpq14aePW1AesYMG4g+66zIv/5hh1kLY84c2yfhiitg0iRLUA0bwksv2Qed\nO3jLl1ul2Hfese7AkSOhePGgo3IFgSeLOLB3L4wfD/XqWc2fUqXg7bftG33LlsH0YTdqBC+/bK2N\noUMhIQFuvNFaG9ddZ90mLnfefddKdaSmwocfWuvNxydcXvFkUYCpwptv2iD15ZdbiY7Jk+1bfZs2\nsfFBcvjhcP31FtPs2VYe+/XXbXVx48a2uvjPP4OOMrapwqOPQtu2tiYmORnOPjvoqFxB48miAFK1\nbojGjW0h3e7d1rL4/nvry46FJJGZiH0rfvVVa238738W9/XXW2vjhhtsGq/7tz//tBX0995ra1u+\n+AKqVg06KlcQebIoQFStKFyzZvYt8/ffbfrqDz9A5875p/BfqVJw002W3L76yhLcqFFw8sk2DXTE\nCFsTEu9WrbJ/67fesk2Kxo61yQPORUI++fhwOUkfpD7/fPtmPmyYlePo3h0K59PawiL2YThqlP2d\nnn0W/vjDBskrVYKbb7ZEGI8++MC66tavtwWQt98emy1GV3B4ssjnvv7a6jI1bw4rV1r3zYoVNkhc\nkLbDLF3aKt4uWmRTQdu0sYRYvz6cfrrVq9q+PegoI0/V6m9deKGVVUlOhvPOCzoqFw88WeRT8+ZZ\nzabTToMFC2wF9qpV1n1TkEs5iNjisrFj7Vv1U0/ZvtHdu1tr47bbrEVVEP39t01UuOsuG4v6+msb\n0HYuGjxZ5DMLFtg+yUlJNnvoscdg9Wr7kIy3+fRly1r3y9KlVsqiZUtb/HfiidYl9/rrtkq9IFi7\n1lpQEyfav/nEibZuxbloiWiyEJELRGSZiKwUkX5ZnH9GRL4L3ZaLyNYM57qLyIrQrXsk48wPliyx\naaUnnWQb1gwcCGvWQL9+ULJk0NEFS8Smik6YACkp8Pjj9mfXrtbauOMOW6yWX33yiX05WLPG1lL0\n6+fjEy4AqhqRG5AArAJqAEWA74E6B7j+ZmBE6P5RwOrQn6VD90sf6PUaN26sBdGKFarduqkWKqRa\nsqTqvfeq/vpr0FHFvj17VD/4QLV9e9XChVVBtUUL1YkTVXfuDDq68Ozdq/rMM6oJCap16qguXx50\nRK4gApI1jM/0SLYsmgArVXW1qu4CJgAXHeD6LsD40P3zgQ9V9VdV/Q34ELgggrHGnLVrbdZP7dq2\nkO722+2b5cMP22CvO7BChWzgd9Ik+PFH28dh1SprnR17rH07X7066Cizt327jcPcdptNg/7mG6hV\nK+ioXDyLZLKoBPyU4XFK6Nh+RKQqUB34JDfPFZGeIpIsIsmpBWR7tvXrrezF8cfbIO5NN9mH2hNP\nWB+9y70KFWwL2FWrrKJus2Y2o6hmTZtq/OabtgAwVvz4ow3ijxljm1BNnmwr3Z0LUiSTRVa9qtnV\nFu0MTFLVPbl5rqoOU9UkVU0qV67cQYYZG37+Gfr0sQ+w4cOtVbFqlRWDK18+6OgKhoQEm3I6ZYp9\nIA8cCIsX28yiqlVty9F164KNcdYsG59YvtxKut9/f/5ZTOkKtkj+N0wBMhZGrgxsyObazuzrgsrt\nc/O1zZttKmSNGrZGomtX+6B46SWbR+8io1IleOAB6+57+20rbPjoo7YBVOvW9kGdlha9eFTt3/+c\nc6ybcc4c635yLmaEM7BxMDegMDYwXZ19A9x1s7juBGAtIBmOHQWswQa3S4fuH3Wg18tvA9y//mqD\n1SVLqorYILYPYAZr3TrV++9XrVDBBsQrVVJ98EHVn36K7Otu36569dX2mm3aqG7dGtnXcy4jwhzg\njliysBhoBSzHZkXdGzo2CGiX4ZoBwOAsnns1sDJ065HTa+WXZPH776qDBqmWKmXv/mWXqS5aFHRU\nLqNdu1TffFP1/PMtkRcqpNquneq776qmpeXta6WkqJ56qv1fuP9+m8XlXDSFmyzErs3/kpKSNDmG\nN0H46y/rZnjiCfj1V9vucuDA3O9x7aJrzRrbVXDECPjlF6hSxUqpXHONDZwfii+/hA4drHLs6NFW\nMNG5aBOReaqa40bKPnQGtqfoaafZPMUJE+wTIo+S6Pbt8MwzNibRr59VTZ071wZZPVHEvurVbSzj\nxx/hjTds+ur999v02/btraDf3r25/7nDhtlCwpIlbVqsJwoX6zxZAOzZY1Nlhg61TQFq1IBjjrHp\nSGBfKbduPfDPyGTnTnjxRdvfum9f26Xuyy9t6mZSjjncxZoiRWzfiI8+skKNffvazKXzz7cEMniw\n/TfJya5d0KuX7dNxzjk2kF23buTjd+6QhdNXlR9ueTJmsWuX6rx5qi++qNqjhz1WVb31VutUrl1b\ntXt3Oz9vni2xzeJHvPKKapUq9pQzzlCdOfPQQ3OxZ8cO1fHjVZs3t3/rxETVjh1VP/44y/8aumGD\n6mmn2bX9+uX9+IdzBwMfs8hDc+ZYf8Ps2XZLTbUO6/XrrUjPuHHsSSjCuA1nM3BIGVavFpo0gYce\nslXEXsen4Fu61LqWRo+2MalataBnT9vzvGxZ+29z6aXWQB05Ejp2DDpi50y4YxZhJQsRqQmkqOpO\nEWkONABeU9Xc9c1EUNQGuFVtcn5KCpx5Jnv3wv9V6cuD63uyjNo0LLyQhxpNofX1lZGre0Q+HhdT\nduywEiMvv2xbnBYpYl1VM2bY9rA+VuViTV4PcE8G9ojIccCr2NqJ1w8hvvxLBKpXR884k7fegoYN\nofP6/1K4RlUmXfMe8zo/RZutY5HZ39j1e/dC06Zw9dX2CfLdd9Fd7eWiqlgx6NYNPv/cdvG7/nrb\nGvacc2yjIk8ULr8Kt2UxX1UbicidwA5VfUFEvlXVkyMfYnii1bJQtUHqBx6A+fOthtOAAdatkJCQ\n4cL0QfPffrNPj9mzYcsWO1eihE2R6tnTRsI3bbLl2t5f5ZyLsnBbFuHuzrxbRLoA3YH0IgQFaNPO\nnKnaTJj777fP/erVbW/orl2z2eM6PXOULm2bEKhaRcD0cY86dez87Nm2U0+FCjavNuOtRIlo/fWc\nc+6Awk0WPYBewCOqukZEqgNjIxdWbJk1y5LErFk2v37YMBu4zNUe1yJWJbBmTdsbM13NmvDCCzbZ\nfvZs69QG2zOzaVPru/j2W0sedetmar4451x05Ho2lIiUBo5V1QWRCengRKIb6ptvLEl89NG+MtfX\nXRfhPa63bLHZVy1a2Avde6+tCgPbRzMpyRLHwIHWQe6cc4cgr2dDfQq0w1oi3wGpwGeq2vcQ48wz\neZks5s2zMYnp06FcOVt5fcMNAe1xrQorV+7rvpo926bspqRYa+Wuu2zxYHrXVePGvs+qcy5seT1m\nUUpVt4nItcBIVX1QRGKqZZEXFi60JDFlig01PPYY9O4d8GeviE3ar1XLBsrBBs/TB8MLFYLvv7cd\nfNIfX3yx7ZgDVqeiUiXvvnLOHZJwk0VhEakAdATujWA8gVi61GY0vfGG7Ug2YIBtRFSqVNCRZSPj\nB//gwXbbvNm6r2bP3he4qm01LWR6AAAVR0lEQVTUsHMnnHIKNGlirY/TTrNyJs45F6Zwk8UgYAbw\nparOFZEawIrIhRU9K1bYuHHx4tC/v+11fdRRQUd1EMqWhVat7JZuzx7473/3dV89/bSt8bjzTit/\nu2OHlcJN777y2VfOuWx4uQ9srdyll9r4RIG2fbvNrCpXzrq1kpOtxQHWWqlf3xLHTTfZfedcgZen\nK7hFpLKIvCUim0TkFxGZLCIFZtPP66+Pg0QB1nw67TRLFGAzq375xfYV7d/f3oQJE2whIViNinPP\ntWlgU6faRuHOubgUbjfUSKy8x2Whx91Cx86LRFAuio4+Gtq0sRv8e3OGHTsscTz55L4SJVWq2BqQ\nihVtTMRXnTsXF8KtDVVOVUeqalroNgqIh+/i8adQIbuBbec3bx5s22ZV8Z5+2lably9v52+91cZI\nRo3a1xpxzhVI4bYsNotIN2B86HEXYEtkQnIxp3hxOP10u2VUoQJMmwY9ethy9pYt7X779sHE6ZyL\nmHBbFldj02Z/BjYCHbASIC6e9e9vW9DOnm2tjIUL4cMP7ZyqzUXeti3YGJ1zeeKgZ0OJSB9VfTaP\n4zloUdvPwmVPFf76y1Yxzp1r6zqKFoULLrCyvG3b2kIW51zMyOv9LLISM6U+XIwQ2bfcvXFj28jh\nhhtsim7XrjbbavbsYGN0zh2UQ0kWPg3GZa9QIWjWzPbt+PFHGyC/8cZ9u/88/riNbUycCH/+GWys\nzrkchTvAnZWCsZrPRV6hQvsPkItYy+PNN20AvVUra31ccklwcTrnsnXAloWI/CEi27K4/QFUjFKM\nriC66y6rnPvZZ7bl7BdfwOjR+85/+CH8/Xdw8Tnn/sXLfbjYsGePrdUoWxbWrYNq1axWVdu2Njh+\n4YUB1Yh3rmCLxgC3c3knIcESBdh+5J98AldeaX+2b2+D4x99FGyMzsUxTxYu9iQkwNlnw0svwYYN\nliS6dt03OP7aa7Y17ZQpVpLEORdxnixcbCtcGM45x0oDp1d73LLFihxeconVturWzYohOucixpOF\ny39uu80q4M6YYeMZ06fDI4/sOz93rm345JzLMz7A7fK/3bth40ariLttm7VAihe37WU7drQy60WK\nBB2lczHJB7hd/EhMtEQBliSmTLEuqilToHVr20J26tRgY3Qun/Nk4QqWxESbZjtypG3s9M470K4d\nnHCCnZ8xA665xv7cvTvYWJ3LRzxZuIKraFFrWYweDbVr27E1a+D//s+KG5YvD9ddBx988O9Nn5xz\n+/Fk4eJLr16waZN1S114oW0j27Pnvh3/li7dtyugc+4fEU0WInKBiCwTkZUi0i+bazqKyGIRWSQi\nr2c4vkdEvgvdpkUyThdnihWzrqmxYy1xvPuuJYu0NDjzTNvUqVcvWxDoicM5IILJQkQSgCHAhUAd\noIuI1Ml0TS2gP3C6qtYF+mQ4vV1VG4Zu7SIVp4tzxYtD3br7Hg8bZrOnxo619R2VKlnrw7k4F8mW\nRRNgpaquVtVdwATgokzXXAcMUdXfAFR1UwTjce7AChe2WVTjx1uLY9IkaN7cEgbA/PnQuzfMmmW1\nrJyLI5FMFpWAnzI8Tgkdy+h44HgR+VJEvhGRCzKcKyYiyaHjF0cwTuf2V6LEvv02zjzTji1cCCNG\nwFlnwbHHws03w+ef++C4iwuRTBZZbY6UeQVgYaAW0BzoAgwXkSND56qEFopcDjwrIjX3ewGRnqGE\nkpyampp3kTuXle7drcUxcSKcdhoMHw5t2sCuXXZ+/XpPHK7AimSySAGOzfC4MrAhi2umqupuVV0D\nLMOSB6q6IfTnauBT4OTML6Cqw1Q1SVWTyqXXDXIukkqWtFXhkyZBaiq8/74NmKvaWEeVKlaO5Kuv\nPHG4AiWSyWIuUEtEqotIEaAzkHlW0xTgbAARKYt1S60WkdIiUjTD8dOBxRGM1bncK1nSto4FSxb3\n3w9JSfDii7YrYNWqMGpUoCE6l1cilixUNQ3oDcwAlgBvqOoiERkkIumzm2YAW0RkMTATuFNVtwAn\nAski8n3o+GBV9WThYlehQvvKpqem2myqRo0soQBs3myVcQtILTYXf7yQoHPR8MgjcN99cOqp8PDD\nNi1XshrWcy66vJCgc7HkrrvglVdsM6fzzrPNnb74IuionAubJwvnoiExEa69FlasgOeft7IizzwT\ndFTOhc2ThXPRVLSorc9YvRqGDLFjS5dChw6waFGwsTl3AJ4snAtCiRJW9RYsSXzwAdSvb1vErlwZ\nbGzOZcGThXNBa9/eSqffdRe8+aaVU7/55qCjcu5fPFk4FwvKlIHBg6176qab9k25BdiyJbi4nAvx\nZOFcLClfHp57Dh57zB5//LGtCu/Xz5OGC5QnC+diWfXqVgn3iSegRg0YOBC2bQs6KheHPFk4F8tq\n1LDV4AsX2vqMAQOgaVNfCe6irnDQATjnwlC3rhUvnDcPUlL27ew3ejR07WrFDJ2LIG9ZOJefNG4M\nF4X2EHvvPVvod/zxtjp89+5gY3MFmicL5/KrNm3go4+gYkXo2RPq1IFx47w0uosITxbO5VciVpDw\n669h2jQ47DCbReXjGS4CPFk4l9+JQNu2tkf4jBmQkAC//24D4u+958nD5QlPFs4VFIUKQaXQNver\nV1vZkFatbA/xTz8NNDSX/3mycK4gOvlkWLYMXnrJSomcfbZt+/r330FH5vIpTxbOFVRFikCvXtbC\neOYZGwgvUcLObdwYbGwu3/Fk4VxBV7w49OkDr71mj9eutZXhnTpZeXTnwuDJwrl4U7o03HknTJ9u\ni/2uusrGOJw7AE8WzsWbUqXgoYcsQdx2G0ycCPXqwa+/Bh2Zi2GeLJyLV+XKwVNP2ZjGq6/CUUfZ\n8VdegU2bgo3NxRxPFs7Fu0qVoEsXu79mDdxwgxUwvPde+O23YGNzMcOThXNun+rVbZvXNm3g0Uft\n8cMPw19/BR2ZC5gnC+fcv51wAkyYAN9/D2edBU8/7UUKnScL51w2GjSAqVNtcd+RR1qBwtatbaHf\nrl1BR+eizJOFc+7Ajj7a/tyyxWpO3XijtT5GjbI9NVxc8GThnAtPuXLw+edWnLBMGejRw9ZprFoV\ndGQuCjxZOOfCJwIXXABz58Kbb1oLo0oVO7dunVe4LcA8WTjnck8ELrnE9tFITLQChU2b2u3DDz1p\nFECeLJxzh65IEZti+/PP0LIlNG8OX3wRdFQuD3mycM4dusKF4ZprYPlyeOEF+/PMM2HOnKAjc3nE\nk4VzLu8ULQq9e9ug9+jRcMopdnzsWPjhh2Bjc4fEk4VzLu+VKAFXXmljGzt3WpXbBg2ga1dYsSLo\n6NxB8GThnIusokWthMjdd8OUKXDiiXDttbB+fdCRuVzwZOGci7yjjoLHHrPuqd694fXXvUhhPuPJ\nwjkXPeXLw7PPWquiXj07dvXVcNddtkLcxayIJgsRuUBElonIShHpl801HUVksYgsEpHXMxzvLiIr\nQrfukYzTORdlpUvbn3v22O2pp6zC7YMPWkkRF3MilixEJAEYAlwI1AG6iEidTNfUAvoDp6tqXaBP\n6PhRwIPAqUAT4EERKR2pWJ1zAUlIsFlTP/xg6zMGDbKk8fnnQUfmMolky6IJsFJVV6vqLmACcFGm\na64DhqjqbwCqmr491/nAh6r6a+jch8AFEYzVORekOnVg0iSYN8/WZ9Ssacc//hhee83304gBkUwW\nlYCfMjxOCR3L6HjgeBH5UkS+EZELcvFcRKSniCSLSHJqamoehu6cC0SjRlYWvWJFezx6NHTvDhUq\n2AyqL7/0UiIBiWSykCyOZf5XLgzUApoDXYDhInJkmM9FVYepapKqJpUrV+4Qw3XOxZzRo+Gzz6B9\ne9uQ6YwzrCaVi7pIJosU4NgMjysDG7K4Zqqq7lbVNcAyLHmE81znXEEnAv/5D4wcaXWnRoywhX1g\nXVOXXAJvvAE7dgQbZxyIZLKYC9QSkeoiUgToDEzLdM0U4GwAESmLdUutBmYALUWkdGhgu2XomHMu\nXpUsaXtoXHaZPV6xApKToVMn67bq3dvGPLybKiIilixUNQ3ojX3ILwHeUNVFIjJIRNqFLpsBbBGR\nxcBM4E5V3aKqvwIPYQlnLjAodMw550zDhrB2LcyYAeefD8OHQ1KSrRYHTxp5TLSAvKFJSUmanJwc\ndBjOuaD89pvt4nf55fa4Z0/YtMkW/V14oe274fYjIvNUNSmn63wFt3OuYChdel+iAJtB9fXXcNFF\nULky3HEHLFkSXHz5nCcL51zBNHAgpKTYVNzTToPnnoNXXrFze/d6bapc8mThnCu4EhOhXTt46y3Y\nsMFqUAF8+qm1PLp0gQ8+sJIj7oA8WTjn4kO5clbIEODYY22RX/rgeLVqcN998OefgYYYyzxZOOfi\nT61a8L//WWtj4kSrgDtyJBQrZufnz4c//gg2xhjjycI5F7+KFYOOHW0W1YoVtpf4nj02KF6hAlx1\nla0gLyCzRg+FJwvnnAPbChagUCFrbXTpAm++Cc2bw3HH2f045snCOecyErHZU6+8Ahs3WtXbqlWh\neHE7v3q17fS3fXuwcUaZJwvnnMvOYYfBFVfAJ5/Ywj6wRNG1q3VT9eoFc+bERTeVJwvnnMuNe+6x\nfTbatrVWx6mnWpmRAj791pOFc87lRqFC0KIFjBlj3VTDhtmAeEKCnb/9dhvf2LUr2DjzmNeGcs65\nvLJlC9Svb0mkbFnrrurRA046KejIsuW1oZxzLtrKlIEff4R337VZVC++aNVx/+//go7skHmycM65\nvFS4MLRqZQli40Z4/nlo2dLOvfLKvnUd+WyMw5OFc85FSpkycPPNUKqUPf77b5tZ1aoVVKkC/fvD\nsmXBxhgmTxbOORctt95qJUYmT4ZGjeDJJ+G66/ad37kzuNhy4MnCOeeiqUgRuPRSePtt+OknGDLE\njm/eDMccs29dx969wcaZiScL55wLSoUKNnsKrFXRubMlkXPOgZo1YcAA2+0vBniycM65WFCpEgwd\naoPi48ZZPaqHHoK//rLzP/1kYx4B8WThnHOxpHhx2x72ww9tfKN6dTveu7ftx3HddfDVV1EvMeLJ\nwjnnYtUxx+y7f8cdNtbx+utw+ulw4onw6qtRC8WThXPO5QdnngmjRsHPP8OIEXD00dbyiJLCUXsl\n55xzh+7ww62ESI8eUZ0x5S0L55zLrwpF7yPck4VzzrkcebJwzjmXI08WzjnncuTJwjnnXI48WTjn\nnMuRJwvnnHM58mThnHMuR54snHPO5Ug0ysWoIkVEUoF1h/AjygKb8yicvORx5Y7HlTseV+4UxLiq\nqmq5nC4qMMniUIlIsqomBR1HZh5X7nhcueNx5U48x+XdUM4553LkycI551yOPFnsMyzoALLhceWO\nx5U7HlfuxG1cPmbhnHMuR96ycM45lyNPFs4553IUV8lCRC4QkWUislJE+mVxvqiITAydny0i1WIk\nrqtEJFVEvgvdro1SXCNEZJOI/JDNeRGR50NxLxCRRjESV3MR+T3D+/VAlOI6VkRmisgSEVkkIrdm\ncU3U37Mw44r6eyYixURkjoh8H4prYBbXRP13Msy4AvmdDL12goh8KyLvZHEucu+XqsbFDUgAVgE1\ngCLA90CdTNfcCAwN3e8MTIyRuK4C/hfAe/YfoBHwQzbnWwHvAQI0BWbHSFzNgXcCeL8qAI1C9w8H\nlmfxbxn19yzMuKL+noXeg5Kh+4nAbKBppmuC+J0MJ65AfidDr90XeD2rf69Ivl/x1LJoAqxU1dWq\nuguYAFyU6ZqLgNGh+5OAc0REYiCuQKjqLODXA1xyEfCamm+AI0WkQgzEFQhV3aiq80P3/wCWAJUy\nXRb19yzMuKIu9B78GXqYGLplnnET9d/JMOMKhIhUBloDw7O5JGLvVzwli0rATxkep7D/L8w/16hq\nGvA7UCYG4gJoH+q2mCQix0Y4pnCFG3sQmoW6Ed4TkbrRfvFQ8/9k7FtpRoG+ZweICwJ4z0JdKt8B\nm4APVTXb9yuKv5PhxAXB/E4+C9wF7M3mfMTer3hKFlll18zfFsK5Jq+F85pvA9VUtQHwEfu+OQQt\niPcrHPOxejcnAS8AU6L54iJSEpgM9FHVbZlPZ/GUqLxnOcQVyHumqntUtSFQGWgiIvUyXRLI+xVG\nXFH/nRSRNsAmVZ13oMuyOJYn71c8JYsUIGP2rwxsyO4aESkMlCLy3R05xqWqW1R1Z+jhK0DjCMcU\nrnDe06hT1W3p3QiqOh1IFJGy0XhtEUnEPpDHqeqbWVwSyHuWU1xBvmeh19wKfApckOlUEL+TOcYV\n0O/k6UA7EVmLdVe3EJGxma6J2PsVT8liLlBLRKqLSBFs8GdapmumAd1D9zsAn2hopCjIuDL1abfD\n+pxjwTTgytAMn6bA76q6MeigRKR8ej+tiDTB/p9vicLrCvAqsERV/5vNZVF/z8KJK4j3TETKiciR\nofvFgXOBpZkui/rvZDhxBfE7qar9VbWyqlbDPic+UdVumS6L2PtVOC9+SH6gqmki0huYgc1AGqGq\ni0RkEJCsqtOwX6gxIrISy8adYySuW0SkHZAWiuuqSMcFICLjsVkyZUUkBXgQG+xDVYcC07HZPSuB\nv4EeMRJXB+AGEUkDtgOdo5D0wb75XQEsDPV3A9wDVMkQWxDvWThxBfGeVQBGi0gClpzeUNV3gv6d\nDDOuQH4nsxKt98vLfTjnnMtRPHVDOeecO0ieLJxzzuXIk4VzzrkcebJwzjmXI08WzjnncuTJwrlc\nEJE9GSqNfidZVAk+hJ9dTbKppOtc0OJmnYVzeWR7qAyEc3HFWxbO5QERWSsij4f2QZgjIseFjlcV\nkY9DBec+FpEqoePHiMhbocJ934vIaaEflSAir4jto/BBaAWxc4HzZOFc7hTP1A3VKcO5baraBPgf\nVh2U0P3XQgXnxgHPh44/D3wWKtzXCFgUOl4LGKKqdYGtQPsI/32cC4uv4HYuF0TkT1UtmcXxtUAL\nVV0dKtr3s6qWEZHNQAVV3R06vlFVy4pIKlA5QzG69PLhH6pqrdDju4FEVX048n8z5w7MWxbO5R3N\n5n5212RlZ4b7e/BxRRcjPFk4l3c6Zfjz69D9r9hXzK0r8EXo/sfADfDPRjtHRCtI5w6Gf2txLneK\nZ6jcCvC+qqZPny0qIrOxL2FdQsduAUaIyJ1AKvuqzN4KDBORa7AWxA1A4OXdncuOj1k4lwdCYxZJ\nqro56FiciwTvhnLOOZcjb1k455zLkbcsnHPO5ciThXPOuRx5snDOOZcjTxbOOedy5MnCOedcjv4f\nYmRcxMnHzbgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8bb76edcf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd4VGX2wPHvIZTgCkpTaQoo6tKF\niCiKYGGxIQpSbICFn4VVVkVhdV3rKvYCNkRFRUBRlLUXQFcRNCigokjXUDRUQWqS8/vj3MAQEjIk\nmbmTyfk8zzzO3Ln3zpmRyZn7lvOKquKcc84VVbmwA3DOOVe6eSJxzjlXLJ5InHPOFYsnEuecc8Xi\nicQ551yxeCJxzjlXLJ5InCuAiDQQERWR8lHs209EPo9HXM4lGk8kLimIyBIR2SYiNfNsnxUkgwbh\nRLZLLH8RkY0i8m7YsThXkjyRuGSyGOiT+0BEmgOVwwtnNz2ArUBnEakdzxeO5qrKuaLyROKSyUvA\nxRGP+wIvRu4gIvuJyIsikikiS0XkFhEpFzyXIiIPiMgqEVkEnJHPsaNEZIWILBORu0QkZS/i6ws8\nBcwBLshz7voi8kYQ12oRGR7x3OUi8qOIbBCRuSLSOtiuInJYxH4viMhdwf2OIpIhIjeJyErgeRGp\nJiJvB6+xNrhfL+L46iLyvIgsD55/M9j+vYicFbFfheAzarUX790lMU8kLplMB6qKyF+DP/C9gJfz\n7PM4sB/QCDgRSzz9g+cuB84EjgLSsCuISKOBLOCwYJ/OwGXRBCYiBwMdgTHB7eKI51KAt4GlQAOg\nLjAueO484LZg/6pAV2B1NK8JHARUBw4BBmDf9+eDxwcDm4HhEfu/BOwDNAUOAB4Otr8IXBix3+nA\nClWdFWUcLtmpqt/8VupvwBLgFOAW4B6gC/ARUB5Q7A90Cta01CTiuP8Dpgb3JwNXRDzXOTi2PHBg\ncGzliOf7AFOC+/2Az/cQ3y3ArOB+HSAbOCp4fCyQCZTP57gPgGsLOKcCh0U8fgG4K7jfEdgGpO4h\nplbA2uB+bSAHqJbPfnWADUDV4PEE4Maw/5/7LXFu3m7qks1LwGdAQ/I0awE1gYrYL/9cS7ErALA/\nmL/meS7XIUAFYIWI5G4rl2f/PbkYGAmgqstF5FOsqetboD6wVFWz8jmuPrAwytfIK1NVt+Q+EJF9\nsKuMLkC1YHOV4IqoPrBGVdfmPUkQ7xdAdxGZCJwGXFvEmFwS8qYtl1RUdSnW6X468Eaep1cB27Gk\nkOtgYFlwfwX2BzXyuVy/YlckNVV1/+BWVVWbFhaTiBwHNAaGisjKoM/iGKBP0An+K3BwAR3ivwKH\nFnDqTVhTVK6D8jyft7T39cARwDGqWhXokBti8DrVRWT/Al5rNNa8dR7wpaouK2A/VwZ5InHJ6FLg\nJFX9M3KjqmYDrwJ3i0gVETkEuI6d/SivAteISD0RqQYMiTh2BfAh8KCIVBWRciJyqIicGEU8fbFm\ntiZYc1IroBmWBE4DvsKS2L3BEOFUEWkfHPsscIOItBFzWBA3wCzg/GCQQBesz2dPqmD9IutEpDrw\n7zzv7z3giaBTvoKIdIg49k2gNXYlkvdKz5Vxnkhc0lHVhaqaXsDTfwf+BBYBnwOvAM8Fz43E+iRm\nA9+w+xXNxVjT2FxgLdZXsMdhvCKSCvQEHlfVlRG3xVgzXN8gwZ2FdeL/AmRgAwVQ1deAu4M4N2B/\n0KsHp782OG4dNgrszT3FAjyCDYdehQ1MeD/P8xdhV2w/Ab8Dg3KfUNXNwOtYk2Hez8WVcaLqC1s5\n5wonIrcCh6vqhYXu7MoU72x3zhUqaAq7FLtqcW4X3rTlnNsjEbkc64x/T1U/Czsel3i8acs551yx\n+BWJc865YikTfSQ1a9bUBg0ahB2Gc86VKjNnzlylqrUK269MJJIGDRqQnl7QaFDnnHP5EZGlhe/l\nTVvOOeeKyROJc865YvFE4pxzrljKRB9JfrZv305GRgZbtmwpfGeXr9TUVOrVq0eFChXCDsU5F6Iy\nm0gyMjKoUqUKDRo0IKIsuIuSqrJ69WoyMjJo2LBh2OE450JUZpu2tmzZQo0aNTyJFJGIUKNGDb+i\nc86V3UQCeBIpJv/8nHNQxhOJc84lrW+/hZUr4/JSnkhCsnr1alq1akWrVq046KCDqFu37o7H27Zt\ni+oc/fv3Z968eXv92meccQYnnHDCXh/nnEtwWRGrNf/nPzBiRFxetsx2toetRo0azJo1C4DbbruN\nfffdlxtuuGGXfVQVVaVcufzz/fPPP7/Xr7t69Wq+++47UlNT+eWXXzj44IMLP8g5l9iWL4enn4Zn\nnoEpU+DII+HBB6Fq1bi8vF+RJJgFCxbQrFkzrrjiClq3bs2KFSsYMGAAaWlpNG3alDvuuGPHvscf\nfzyzZs0iKyuL/fffnyFDhtCyZUuOPfZYfv/993zPP2HCBLp160avXr0YP378ju0rV67k7LPPpkWL\nFrRs2ZIZM2YAlqxyt/Xv3z+2b945Fz1V+Owz6NULDjkE7rwT2rTZeVVy8MGw//5xCcWvSHJ17Lj7\ntp494aqrYNMmOP303Z/v189uq1ZBjx67Pjd1apFDmTt3Ls8//zxPPfUUAPfeey/Vq1cnKyuLTp06\n0aNHD5o0abLLMevXr+fEE0/k3nvv5brrruO5555jyJAhu5177Nix3HPPPey3335ceOGFDB48GICr\nr76aU089lYEDB5KVlcWmTZuYPXs2w4YNY9q0aVSvXp01a9YU+T0550qIKojA+vXQpQukpsKgQXDl\nldCoUSgheSJJQIceeihHH330jsdjx45l1KhRZGVlsXz5cubOnbtbIqlcuTKnnXYaAG3atOF///vf\nbuddtmwZv/zyC+3atUNEyM7O5qeffuLII49k6tSpjBs3DoDy5ctTtWpVJk+eTK9evahe3ZYIz/2v\ncy4E8+fDE0/A3LnwwQd2tfHBB3YVss8+oYbmiSTXnq4g9tlnz8/XrFmsK5C8/vKXv+y4P3/+fB59\n9FG++uor9t9/fy688MJ8525UrFhxx/2UlBSyIjvdAuPHj2f16tU7JhCuX7+ecePGcdtttwG7D+dV\nVR/i61yYcnLgvfdg+HB4/30oXx7OOw82b4bKlSFBBs14H0mC++OPP6hSpQpVq1ZlxYoVfPDBB0U+\n19ixY/n4449ZsmQJS5Ys4auvvmLs2LEAdOrUaUdTWnZ2Nn/88QennHIK48aN29Gk5U1bzsXZmDFw\n5pkwezbcfjv88gu88oolkQTiVyQJrnXr1jRp0oRmzZrRqFEj2rdvX6TzLFy4kJUrV5KWlrZjW+PG\njalUqRIzZ85k+PDhXH755Tz99NOUL1+ep59+mrZt23LjjTfSoUMHypcvT5s2bRg1alRJvTXnXF7f\nfmtDdtu2hQED4NxzoVIlOOccSOCadmVizfa0tDTNu7DVjz/+yF//+teQIkoe/jk6V0zbtsEbb1jz\n1RdfWFP60KFwyy1hR4aIzFTVtML28ysS55wLU+/eMHEiHHooPPSQjQStVi3sqPaK95E451y8qML/\n/gd9+sBvv9m2f/wD3n0Xfv7Z7peyJAJ+ReKcc7H355/WST58OMyZY0N358yBU09NmJFXxeGJxDnn\nYumPP6BhQ1izBlq2hGeftSuSkOd+lKSYNm2JSBcRmSciC0Rk92nWtk9PEZkrIj+IyCsR27NFZFZw\nmxSxvaGIzBCR+SIyXkQq5nde55wLRU6ONVXdeac9rloVhgyxJq1vv4VLL02qJAIxTCQikgKMAE4D\nmgB9RKRJnn0aA0OB9qraFBgU8fRmVW0V3LpGbB8GPKyqjYG1wKWxeg/OORe1tWuts/zww+GMM6yI\n4saN9tzgwXD88VbaJAnF8oqkLbBAVRep6jZgHHB2nn0uB0ao6loAVc2/0mBAbJr1ScCEYNNooFuJ\nRh0nHTt23G1y4SOPPMJVV121x+P23XdfAJYvX06PvPW9Is6dd7hzrszMTCpUqMDTTz9dhKidc/l6\n5x2oWxeuvx7q1IHx42HxYgi+r8kulomkLvBrxOOMYFukw4HDReQLEZkuIl0inksVkfRge26yqAGs\nU9Xc+h/5nRMAERkQHJ+emZlZ/HdTwvr06bOjtlWucePG0adPn6iOr1OnDhMmTCh8xzxee+012rVr\nt2NGu3OuCLZvt2QxebI9TkuDiy6CWbOsIm/Pngk9gbCkxTKR5HcNl3f2Y3mgMdAR6AM8KyK5dY8P\nDibCnA88IiKHRnlO26j6jKqmqWparVq1ihJ/TPXo0YO3336brVu3ArBkyRKWL1/O8ccfz8aNGzn5\n5JNp3bo1zZs356233trt+CVLltCsWTMANm/eTO/evWnRogW9evVi8+bNBb7u2LFjefDBB8nIyGDZ\nsmU7tr///vu0bt2ali1bcvLJJwOwceNG+vfvT/PmzWnRogWvv/56SX4EzpU+K1ZYqZJDDrH5HyNH\n2vYDD7SmrJYtw40vJLEctZUB1I94XA9Yns8+01V1O7BYROZhieVrVV0OoKqLRGQqcBTwOrC/iJQP\nrkryO+deGzTIfkiUpFat4JFHCn6+Ro0atG3blvfff5+zzz6bcePG0atXL0SE1NRUJk6cSNWqVVm1\nahXt2rWja9euBRZQfPLJJ9lnn32YM2cOc+bMoXXr1vnu9+uvv7Jy5Uratm1Lz549GT9+PNdddx2Z\nmZlcfvnlfPbZZzRs2HBHTa0777yT/fbbj++++w6AtWvXFu9Dca40GzoUHnjA1vs47TQbfdWlS+HH\nlQGxvCL5GmgcjLKqCPQGJuXZ502gE4CI1MSauhaJSDURqRSxvT0wV62eyxQgt3OgL7D7z/VSIrJ5\nK7JZS1X55z//SYsWLTjllFNYtmwZv+VOXsrHZ599xoUXXghAixYtaNGiRb77jRs3jp49ewLQu3fv\nHc1b06dPp0OHDjuqAueWi//444+5+uqrdxxfrRROlHKuyDZtglGjbA4IWCf63/9u5dzffdfWKCpg\n9dKyJmZXJKqaJSIDgQ+AFOA5Vf1BRO4A0lV1UvBcZxGZC2QDg1V1tYgcBzwtIjlYsrtXVecGp74J\nGCcidwHfAsWuIrinK4dY6tatG9dddx3ffPMNmzdv3nElMWbMGDIzM5k5cyYVKlSgQYMG+ZaOjxRN\nufexY8fy22+/MWbMGMA67OfPn19guXgvI+/KpIULbd2P556DdetsqG6fPuArhBYopulUVd9V1cNV\n9VBVvTvYdmuQRFBznao2UdXmqjou2D4teNwy+O+oiHMuUtW2qnqYqp6nqltj+R5iad9996Vjx45c\ncsklu3Syr1+/ngMOOIAKFSowZcoUli5dusfzdOjQYUdy+P7775kzZ85u+8ybN48///yTZcuW7Sgj\nP3ToUMaNG8exxx7Lp59+yuLFi4Gd5eI7d+7M8OHDd5zDm7ZcUvvzTxu227gxPPYYdO5sHee9e4cd\nWcLz67KQ9enTh9mzZ9M74h/rBRdcQHp6OmlpaYwZM4Yjjzxyj+e48sor2bhxIy1atOC+++6jbdu2\nu+0zduxYzjnnnF22de/enbFjx1KrVi2eeeYZzj33XFq2bEmvXr0AuOWWW1i7di3NmjWjZcuWTJky\npQTesXMJZO1a+PBDu/+Xv9jCUbfeCkuX2qisE05I2rkfJcnLyLti8c/RlUqzZ9u6H2PG2Ez0lSth\nv/3CjirhRFtG3q9InHNlx8yZ0KGDDat8+WU4/3z48ktPIsXkRRudc8ltxQpb47xRI5tpvny5DePt\n3x+CEYqueMp0IvFRScVTFppFXSmlCtOmWdn2CROgWzd47TU44ggbvuvf+xJVZpu2UlNTWb16tf8x\nLCJVZfXq1aSmpoYdinO7evVVaN3aiiS+957N/fjPf3Y+70mkxJXZK5J69eqRkZFBItbhKi1SU1Op\nV69e2GE4B4sWWdmSlBT47jvIzraSJRdcYKOxXEyV2VFbzrlSLifHhu4OH24zzSdOhLPPhm3brGCi\nX3kUW7SjtsrsFYlzrpTauhWefNKG7y5YYAUTb7kFjj7anq/oa93FmycS51xiUrUJg8uX223TJus0\nL1/eZp7Xrg133AHdu3vyCJknEudc/G3cCBkZO5PE8uWWNO65x56/+mormLg1ogJSrVqWSFJSID3d\nh+4mEE8kzrmSt2ABfP31roli+XJ4/32oXNnKkDz88K7HVKlia31UrAjt2lkneZ06u95yeRJJKJ5I\nnHOF27LFJvYdeKBVw501C8aO3T1RpKdb0cNJk2zZWbDEUbeuNUVt2GCP+/SxVQVzE0Tt2pZIcl10\nUTjv0xWJJxLnyrJt26zO1PLllijatIGDD7aEcPPNOxNEUBGayZOhUyf4+WdbfyE3ETRrZtVyc+cV\nXXCBLf5Upw5Urbr7CKqjj97ZOe5KPU8kziWj3EKEea8Y/vY3aN/e5lqcfDLknUc1ejRcfLEt2LRu\nHRx2mNWmyk0YRxxh+3XvDuedV/AQ2wMPtJsrEzyROFcaZWXZHIq8iaJbN7jkEru6yDtZtFw561to\n397+yJ977q79D7VrW+IAmxk+Y0bBr5+SErv35kodTyTOJQJVaz7aunVnp/J999m6GLlJYsUKm3D3\n+ON2JdC1q83gBhvRVKeODZEFSxRPPrlrojjgABs6C3b/qafi/z5dUvJE4lwYVOHZZ60padkySxTb\nttk64O+8Y/sMH26r9uUmgr/+FXIXLUtJgenTLSEcdNDu8yjKl4crrojve3JlVkwTiYh0AR7F1mx/\nVlXvzWefnsBtgAKzVfV8EWkFPAlUxdZyv1tVxwf7vwCcCKwPTtFPVWfF8n04FxOTJlmiOP74ncmi\nSZOdzy9caKU+CpJWaOUK5+IiZolERFKAEcCpQAbwtYhMUtW5Efs0BoYC7VV1rYgcEDy1CbhYVeeL\nSB1gpoh8oKrrgucHq+qEWMXuXEyowksv2fKtDRvCK6/Y+hgFdVjvKYk4l0BiWUa+LbBAVRep6jZg\nHHB2nn0uB0ao6loAVf09+O/Pqjo/uL8c+B2oFcNYnYut336Dc86Bvn2tjwNs3oQXFnRJIJaJpC7w\na8TjjGBbpMOBw0XkCxGZHjSF7UJE2gIVgYURm+8WkTki8rCIVMrvxUVkgIiki0i6l4p3oXrtNWja\n1GZ1P/gg3H9/2BE5V6JimUjy+6mVt2Z9eaAx0BHoAzwrIvvvOIFIbeAloL+q5gSbhwJHAkcD1YGb\n8ntxVX1GVdNUNa1WLb+YcSEZNQp69rRlXr/9Fq67zofOuqQTy872DKB+xON6wPJ89pmuqtuBxSIy\nD0ssX4tIVeAd4BZVnZ57gKquCO5uFZHngRti9QacK7ING6zpqmdPuz9w4M6ht84lmVhekXwNNBaR\nhiJSEegNTMqzz5tAJwARqYk1dS0K9p8IvKiqr0UeEFylILbYejfg+xi+B+f2zvr1NiGwXTurT1Wl\nCgwa5EnEJbWYJRJVzQIGAh8APwKvquoPInKHiHQNdvsAWC0ic4Ep2Gis1UBPoAPQT0RmBbdWwTFj\nROQ74DugJnBXrN6Dc3vl44+heXObG9Ktm80kd64M8KV2nSuuzZth8GBbse+IIyyRHHNM2FE5V2zR\nLrXrP5mcK67y5eGrr+Af/7AOdU8irozxhlvnimLLFlvN79prrRDi//4HlfIdie5c0vMrEuf21tdf\nW3XcO+6At96ybZ5EXBnmicS5aG3bZkvEHnss/PGHTTDs3z/sqJwLnScS56I1ZAjceaet/vf997ZI\nlHPO+0ic26OsLJsbUqOGjcw68URbE8Q5t4NfkThXkHnzrMR79+62dG3t2p5EnMuHJxLn8srJgUcf\nhVat4OefbYEor9LrXIG8acu5SCtWQJ8+8OmncMYZMHKkXYk45wrkicS5SPvuC+vWwXPPQb9+fiXi\nXBS8acu5Zcvgqqt2Fln85hsb1utJxLmoeCJxZVfu0rfNmll9rG++se1ebNG5veLfGFc2/fYbnHsu\nXHyxrV44ezYcd1zYUTlXKnkicWVTv37w3nu27O2nn8Jhh4UdkXOllne2u7JjzRrr96hWDR55xCYb\nNm0adlTOlXp+ReLKhnfesb6Qa6+1x0cc4UnEuRLiicQltz/+gEsvhTPPhJo1bc0Q51yJ8kTiktfM\nmbb07QsvwNChVv79qKPCjsq5pBPTRCIiXURknogsEJEhBezTU0TmisgPIvJKxPa+IjI/uPWN2N5G\nRL4LzvmYiA/2dwWoXRvq1IEvvoD//MfXDHEuRmKWSEQkBRgBnAY0AfqISJM8+zQGhgLtVbUpMCjY\nXh34N3AM0Bb4t4hUCw57EhgANA5uXWL1HlwpNG0aXHKJ1cuqU8cet2sXdlTOJbVYXpG0BRao6iJV\n3QaMA/KWTr0cGKGqawFU9fdg+9+Aj1R1TfDcR0AXEakNVFXVL1VVgReBbjF8D6602LIFbrzRqvVO\nmWKz1cFnpzsXB7FMJHWBXyMeZwTbIh0OHC4iX4jIdBHpUsixdYP7ezonACIyQETSRSQ9MzOzGG/D\nJbyZM6FNG5sTcvnlMGcO1K8fdlTOlRmxnEeS309Bzef1GwMdgXrA/0Sk2R6OjeactlH1GeAZgLS0\ntHz3cUkgO9uq9f75p00w7OItnc7FWywTSQYQ+bOwHrA8n32mq+p2YLGIzMMSSwaWXCKPnRpsr1fI\nOV1ZMHcuNGoEqakwYYJdgVSrVvhxzrkSF8umra+BxiLSUEQqAr2BSXn2eRPoBCAiNbGmrkXAB0Bn\nEakWdLJ3Bj5Q1RXABhFpF4zWuhh4K4bvwSWa7GwYNsyG8d5zj21r0cKTiHMhitkViapmichALCmk\nAM+p6g8icgeQrqqT2Jkw5gLZwGBVXQ0gIndiyQjgDlVdE9y/EngBqAy8F9xcWfDzz9C3L0yfbsvf\nDhwYdkTOOUBs8FNyS0tL0/T09LDDcMXx6qtWaDE1FUaMgN69fUSWczEmIjNVNa2w/Xxmuysdmja1\njvTvv7fOdU8iziUMTyQuManaeulXXGGPmzaFN96wSYbOuYTiicQlnmXL4PTTYcAAmD8fNm8OOyLn\n3B4UmkhEZGBEeRLnYkcVXn7Zyr1/+ik8/jh89BFUrhx2ZM65PYjmiuQg4GsReTUowuiN0y421qyB\na66BJk1s6duBA339dOdKgUK/pap6CzZJcBTQD5gvIv8RkUNjHJsrKyZPtiKLNWrA55/DZ59B48Zh\nR+Wci1JUP/eCAokrg1sWUA2YICL3xTA2l+zWroULL4STT4YxY2xbkyaQkhJuXM65vVLohEQRuQbo\nC6wCnsUmDW4XkXLAfODG2IboktK778Jll0FmJtx+u80Lcc6VStHMbK8JnKuqSyM3qmqOiJwZm7Bc\nUrv1VrjzThvS+/bb0Lp12BE554ohmkTyLpBbngQRqQI0UdUZqvpjzCJzyUfVJhKedBJs3Qp33OGr\nFjqXBKLpI3kS2Bjx+M9gm3PR2bTJRmP985/2uGNHK7zoScS5pBBNIhGNKMilqjnEtvy8SybTpkGr\nVjYnZOtWuypxziWVaBLJIhG5RkQqBLdrsVLvzhVsyxa46SY44QTYts2G+D70kNfIci4JRZNIrgCO\nA5ZhC0sdAwyIZVAuCSxeDI8+CpdeCt99B506hR2Rcy5GCm2iUtXfsUWpnNuz7dth0iRbK+Svf7X1\nQw4+OOyonHMxFs08klTgUqApkJq7XVUviWFcrrTJyoJzzoF33oEZM6BtW08izpUR0TRtvYTV2/ob\n8Cm2TvqGWAblShlVuPpqSyIjRlgScc6VGdEkksNU9V/An6o6GjgDaB7NyYMij/NEZIGIDMnn+X4i\nkikis4LbZcH2ThHbZonIFhHpFjz3gogsjniuVfRv18XEsGHwzDMwZAhcdVXY0Tjn4iyaYbzbg/+u\nE5FmWL2tBoUdJCIpwAjgVKyT/msRmaSqc/PsOl5Vd1l8W1WnAK2C81QHFgAfRuwyWFUnRBG7i7X5\n8+Hmm23VwrvvDjsa51wIorkieSZYj+QWYBIwFxgWxXFtgQWqukhVtwHjgLOLEGMP4D1V3VSEY12s\nNW5sa4Y8/7yXfHcxtXkzpKeHHYXLzx6/+UFhxj9Uda2qfqaqjVT1AFV9Oopz1wV+jXicEWzLq7uI\nzBGRCSJSP5/newNj82y7OzjmYRHJd3q0iAwQkXQRSc/MzIwiXLdXfvoJPgwuEk86yWepu5jKybHB\ngEcfDRO8LSLh7DGRBLPYB+5pnz3Ib+ZZ3mnN/wUaqGoL4GNg9C4nEKmN9cd8ELF5KHAkcDRQHbgp\nvxdX1WdUNU1V02rVqlW0d+Dy99tvthTuJZf4MrguLm67Dd57Dw46yKYmLVwYdkQuUjRtER+JyA0i\nUl9EqufeojguA4i8wqgHLI/cQVVXq+rW4OFIoE2ec/QEJqrq9ohjVqjZCjyPNaG5ePnzTzjrLEsm\nEyf6Mrgu5iZNsmLR/fvDl19aC2rPnlZxxyWGaBLJJcDVwGfAzOAWTUvl10BjEWkoIhWxJqpJkTsE\nVxy5ugJ5qwn3IU+zVu4xwZK/3YDvo4jFlYTsbDj/fJg5E8aOtXYG52Jo3jy46CJIS4MnnoAGDeCF\nF+Cbb2Dw4LCjc7mimdnesCgnVtUsERmINUulAM+p6g8icgeQrqqTgGtEpCu26uIabClfAESkAXZF\n82meU48RkVpY09ksrISLi4fx4+3n4fDh0LVr2NG4JLdhA5x7LlSsCK+/DqnBdOizz4Z//AMefhhO\nPNH6Tly4RAupxioiF+e3XVVfjElEMZCWlqbpPtyj+FTh44/h1FPDjsQlOVVrvnrjDRsUeNJJuz6/\nbZvVA/3pJ/j2W2jUKJw4k52IzFTVtML2i6Zp6+iI2wnAbVgzlCsr3nnH6maJeBJxcXHffTY6a9iw\n3ZMI2FXK+PHeX5IoCk0kqvr3iNvlwFFAxdiH5hLCtGnQowfccEPYkbgy4qOPbA20nj3h+usL3i+3\nv2TmTO8vCVtRZpBtAhqXdCAuAc2fb30h9erBc8+FHY0rA5Ysgd69oUkTGDWq8OVrcvtLHn/c+lFc\nOKKp/vtfds7/KAc0AV6NZVAuAWRm2lwRsAH8NWuGG49Leps3W+d6draNLN933+iOu/de+OILm9Z0\n1FHeXxKGaGptPRBxPwtYqqr+sNumAAAgAElEQVQZMYrHJYrbb4dff7WVDQ87LKpDJk60i5jBg30h\nRLd3VOGKK6zj/O23o/4nB1h/ybhx0Lq1NYd98YUXWoi3aJq2fgFmqOqnqvoFsDoYmuuS2f33wyef\nwHHHRbX7o4/ar8mbbrIc5NzeeOIJePFFm8F+xhl7f3zDhlbuzftLwhFNInkNyIl4nB1sc8lo5EhY\nv95mrLdvX+juqtYxOmiQJZK+fS2RjBwZh1hdUvj8c/v3c+aZ8K9/Ff083brZeR5/3IYNu/iJpmmr\nfFC9FwBV3RbMVHfJZvhw+PvfYdUqGDq00N2zsqw5YtQoGDDAflXm5Fj1lCuugNq17Y+DcwVZvhzO\nO8+uKF56qfgFpIcN29lf0qqV95fESzT/2zKD2ecAiMjZwKrYheRCMWkSXHutjdK68cZCd9+82f4A\njBplvyKfegpSUqBCBXjtNev07NnTVt11Lj/bttm/oQ0b7Api//2Lf87c+SUi0KuXzy+JG1Xd4w04\nFJiO9ZX8AkzDVk0s9NhEubVp00bdHnz1lWrlyqpHH626cWOhu69bp9qhgyqoPvZY/vusXKnaqJFq\nzZqqP/9cwvG6pHDllfZvaPz4kj/3xIl27muuKflzlyVYOavC80Q0O9n52BeoEu3+iXTzRLIHOTmq\nbdqoNmxof/0LsXy5asuWqhUqqL7yyp73/flnSySNGkV1aleGPPec/fUZPDh2rzFokL3G66/H7jWS\nXbSJJJpaW/8B7lPVdcHjasD1qnpLjC6SSpzX2ipERoaVhz/iiD3utnAhdO5sfSBvvGH3CzNjBnTq\nZBPMpk6Nfm6AS17p6XD88XZ7/30oH01PbRFs22av8fPPVi3Y+0v2XknW2jotN4kAqOpa4PTiBOcS\nwNatNrwlO9tmrheSRL791kYCr19vU0uiSSIAxxwDr75qx593HmzfXvgxLnllZtrovgMPtLkfsUoi\n4P0l8RRNIkmJXM5WRCoDPt2nNMvJgX794JprbOxlIaZOtXLdlSrZ7m33cimxM8+0zvj337fRXYVc\nBLsklZVl5U9+/92uaONRLCF3fkl6elRjSFwRRZNIXgY+EZFLReRS4CPyLInrSpmbb7afg/fcYxli\nD954A/72N6hf3+o3Hnlk0V7y8svh3/+2Inu33lq0c7jS7Z//tKvZp56CNnnXQo2hbt1sQOJjj/n8\nkpiJpiMF6IKVSnkQuAUYEc1xiXLzzvYITz9tPZADBlhHeyG7liuneuyxqqtXF/+lc3JUL73UXv7J\nJ4t/Pld6jB9v/9+vuiqc19+61QYl7ref6qJF4cRQGhFlZ3u0039WYrPbuwMns/uSuK40WLXK6nKf\ndhqMGFFgQSxVuPtu+L//s6uRjz6C6tWL//Ii9mv09NPh6qvhrbeKf06X+L7/3iYIHnecrWoYhtz+\nErD+km3b9ry/2zsFJhIROVxEbhWRH4HhwK/YioqdVHV43CJ0JadmTZgyxb5RBfRy5uRYM8Att8CF\nF9of+7/8peRCKF/eOt/btLH28i+/LLlzu8Szbh2ccw5UqWITVSuGWBMjt7/k66+9v6Sk7emK5Cfs\n6uMsVT1eVR/H6mxFTUS6iMg8EVkgIkPyeb6fiGSKyKzgdlnEc9kR2ydFbG8oIjNEZL6IjPdyLVH4\n5Rd4+WW7n5Zm3+p8bNtmyePxx+G662D0aJupXtL+8her8Fq3rnXEz5tX8q/hwpeTAxddZGuMvPYa\n1KkTdkSW1K691oqMTpwYdjRJpKA2L+AcYDx2JTISSyqLo2kvC45PARYCjbAVFWcDTfLs0w8YXsDx\nGwvY/irQO7j/FHBlYbGU6T6StWtVmza1xuHffy9wtw0bVDt3tnbsYcMK7T4pEfPnq9aqpdqggeqK\nFbF/PRdft91m/56GDw87kl15f0n0KG4fiapOVNVewJHAVOAfwIEi8qSIRDOLoC2wQFUXqRV9HAec\nHW2Cy4+ICHASMCHYNBroVpxzJrVt26B7d/vJ//rrUKtWvrutWgUnnwwff2y1s268MT7riRx2mC0H\n//vv1m+yYUPsX9PFx9tvW0n4vn3hqqvCjmZX3l9S8qJZs/1PVR2jqmcC9YBZwG7NVPmoi13N5MoI\ntuXVXUTmiMgEEakfsT1VRNJFZLqI5CaLGsA6Vc0q5JyIyIDg+PTMzMwowk0yqjbmdvJkePZZyxT5\n+OUXm/07Z45d6l9ySXzDPPpoa/aYM8eWhvcvdek3f741kbZuDU8+mZiLnHl/SQmL5rKlKDfgPODZ\niMcXAY/n2acGUCm4fwUwOeK5OsF/GwFLsOKRtbCrnNx96gPfFRZLmWzamjLF2hVuu63AXX74QbVu\nXbvE//TT+IWWn1GjLNyLLopPs5qLjQ0brCW1Rg3VJUvCjqZw115r/+7eeCPsSBITJTz8tygygj/0\nueoByyN3UNXVqppbuGAk0CbiueXBfxdhTWtHYeXr9xeR3CFHu53TBTp2hM8+K3D235df2pVIdrbt\n1qFDfMPL65JL4I47bE2Km28ONxZXNKpw6aXw44823/WQQ8KOqHD33WfjT/r3h8WLw46m9IplIvka\naByMsqoI9AYmRe4gIrUjHnYlmJ8iItVyy7KISE2gPTA3yJBTgB7BMX0Bn40Q6ZNPbAo6wAkn5Nuu\n8N571tJVo4bt2qJFnGMswC23WAmVe+6xaS6udHnwQRvafc89cMopYUcTHe8vKSHRXLYU9YYVd/wZ\nG711c7DtDqBrcP8e4AdsRNcU4Mhg+3HAd8H274BLI87ZCPgKWIAt+VupsDjKTNPWnDmqVauqtm1b\nYPvQSy+pli+vetRRiVnafft21bPOUhXx5obS5OOPrQpCjx6ls2ny9detiWvQoLAjSSyUVBn5ZFAm\nysgvWwbt2tng/RkzrKJvHg8/bPNDOnWCN9+EqlVDiDMKmzbBSSfB7Nk2kiyKpeNdiJYutQmmBx4I\n06cXOE0p4eXW45o40epzuZItI+8S3YYNcMYZVuP93Xd3SyKqMGSIJZHu3W2XRE0iAPvsA//9rxWK\nPOssa3N3iWnzZvs3tX27/QEurUkEvL+kODyRJIPhw62g0WuvQcuWuzyVlQWXXQbDhlntrPHjITU1\npDj3Qq1aVna+QgXo0gWW+5CKhKNqc0RmzrRBEocfHnZExVOpkn0/VL2/ZG95IkkGN94I//ufVViM\nkPtr8bnnbPDWk09CSkpIMRZBo0Z29bR6tU1Y/OOPsCNykZ56aueyAF27hh1NyWjUyL4vX38NN90U\ndjSlhyeS0mzUKFsmNyUFjj12l6fWrbO88t//Wu2s229PzIlhhWnTBiZMgB9+sJX1/FdiYpg2zfoU\nTj/d1plJJueea2u+PfKI9SW6wnkiKa3GjLE2q/vv3+2pFStsvarp02HsWBg4MIT4SlCXLjY5/5NP\nbL5JTk7YEZVtK1ZYFYKDD7ZaoOWS8K+I95fsnST8J1AGTJ1q/8JPPNH+xUeYP9/WfVi40OpY9eoV\nToglrW9fWyNlzBgYOjTsaMqubdvgvPNsXMfEiVCtWtgRxYb3l+wdTySlzY8/Wi3sww6zb3KlSjue\n+uYbm62+caMtO3LqqSHGGQNDh8KVV1rufOyxsKMpm66/Hr74wlpVmzcPO5rY8v6S6HkiKW1uusmS\nx7vv7vJzcPJkq4qSmgqff27FEJONiPX3nH02DBpkBY1d/Lz4og0QvO46W5SsLDj3XPj7362/xFf0\nLJhPSCxt1q+3DvamTXdsev11OP98aNwYPvjAFoxKZps2WQmOb76xZYBPOCHsiJLfN9/YxNBjj4UP\nPyxwgc2ktHWrvfeFC+Hbb6FBg7Ajih+fkJhMsrOtPWfTJthvv12SyNNPW5t1WpoVX0z2JAI7Jyw2\naGDDTufODTui5LZqlf0yr1Vrj6s0J61KlayGWE6O95cUxBNJolO1sYg33WSrBUVsvvNOuOIKG4L5\n0UdQvXqIccZZjRo2YTE11UZ1LVsWdkTJKTsb+vSBlSv3uDZa0svtL/nqK6sS4XbliSTRPfggPPEE\nDB4MPXsC9svo73+3iWAXX2x97vvsE3KcIWjQwLqK1q6F006zVj9Xsm6+2eqdPfFEcva77Y3u3e17\n9/DD3l+ym2gqO5b2W6mt/vvqq1aStGdP1exsVbX1pnv1ss3XX79jc5n24YdW0bhTJ9UtW8KOJnm8\n9pr9O/u//ws7ksSxZYtqmzaq+++vunhx2NHEHgmwsJUrji1bbHhM+/YwejSUK8eGDXDmmdZOfd99\n8MADyTkZbG+deqo1O0yZAv36+YTFkjB3rn2W7drBo4+GHU3i8P6S/PmfoUSVmmpjet96C1JTycy0\nxagmT7Y/moMHhx1gYrnoIrj3XluZz9fgLp71662M+r77WnmaiKlKDu8vyY8nkkSTmWk/AVVtPG+N\nGixdahMNv/vO+kP69w87yMR0441w9dXWrfTII2FHUzrl5Fi/2+LFVky6LIwCLArvL9mVJ5JEsnmz\njWcdMgQWLACsWOFxx8Fvv9nIrLPOCjnGBCZiOficc6xV8NVXw46o9Ln7bpg0CR56yOfnFOb++62o\naL9+sGRJ2NGEyxNJosjOhgsusNUNX3kFGjdm2jT7Mqtalfjjjw87yMSXkmL1uI47zpq7Pv007IhK\nj3fftUq+F15Y+gt9xkNuPa6cHJvpX5b7S2KaSESki4jME5EFIrJba6KI9BORTBGZFdwuC7a3EpEv\nReQHEZkjIr0ijnlBRBZHHNMqlu8hbm64wdqtHnoIzjmHd96x2ds1a1rJ7mSva1SSKle2X9WNGlk5\nle+/DzuixLdggf2OadnSJrmWxiUHwnDooVZ3bMaMMl5MNJqhXUW5ASnAQqARUBGYDTTJs08/YHg+\nxx4ONA7u1wFWAPsHj18AeuxNLAk//PeHH2z86jXXqKrq6NGqKSk2zPC330KOrRRbskS1dm3VunVV\nf/kl7GgS18aNqs2bq1avrrpoUdjRlE4DB9pQ6TffDDuSkkUCDP9tCyxQ1UWqug0YB5wdzYGq+rOq\nzg/uLwd+B5J3Tm2TJrZ4yEMP8eCDVjK9Y0cbznrAAWEHV3odcgi8956trHjaabbYl9uVqi1r8/33\ntnZNw4ZhR1Q6PfAAtG5ddvtLYplI6gK/RjzOCLbl1T1ovpogIvXzPikibbErmoURm+8OjnlYRPId\nnCgiA0QkXUTSMzMzi/E2YmjGDGvOArR1G24cmsINN1jtrHfegSpVQo4vCbRsaR/xzz/bkNYtW8KO\nKLE8/LANmb77bujcOexoSq/I+SVlsr8kmsuWotyA84BnIx5fBDyeZ58aQKXg/hXA5DzP1wbmAe3y\nbBOgEjAauLWwWBKyaWvBAtVatVQPO0y3b9yi/frZpfGVV6pmZYUdXPIZM8Y+3/PO82oAuSZPtibU\nc85RzckJO5rkkFsN4Lrrwo6kZJAATVsZQOQVRj1geeQOqrpaVbcGD0cCbXKfE5GqwDvALao6PeKY\nFcF73Ao8jzWhlS6rV1ulxexsNk94h3P7VOKFF+C222DECBt55ErW+edbNYDXXrOhwZr8qyfs0a+/\n2szsxo3hhRe8c72k9OhhI94eeqiMzS+JJtsU5QaUBxYBDdnZ2d40zz61I+6fA0wP7lcEPgEG5XPe\n2sF/BXgEuLewWBLqimTzZtXjj1etVEnXvDtNjz9eVUR1xIiwA0t+OTk2ngFUH3gg7GjCs3mz6tFH\nq1apovrTT2FHk3y2bFFt3drqcS1ZEnY0xUOUVyQxSyQWA6cDP2P9GzcH2+4Augb37wF+CJLMFODI\nYPuFwHZgVsStVfDcZOA74HvgZWDfwuJIqETy4ouqoMueeFObN1etUEF1/Piwgyo7srJUe/Swf/lj\nx4YdTfzl5Kheeqm9/4kTw44meS1YoFq1quoxx1ih1dIqIRJJotwSKpGo6rzXv9MGDVT33Vf1o4/C\njqbs2bxZ9YQTLIl/8knY0cTX00/bt/7mm8OOJPnl9pdcf33YkRRdtInEZ7bHy0svwZw5zJwJx1/R\njI0bbXjvKaeEHVjZk5pq7deNG1s5lTlzwo4oPqZPt/b7Ll3g9tvDjib59eixs/bbpElhRxNbnkji\n4Z13oF8/Prl2Eh072iJUX3xhy+O6cFSrZissVqlic0x++SXsiGJr5UorNFi/vpWQ8QEd8RE5v2Tp\n0rCjiR1PJLE2cyb06sVrh9zA6dNupkEDK3ly+OFhB+bq17cJixs32q/0NWvCjig2tm+3xTXXroU3\n3ihbSzKHLTXV5pdkZyf3+iWeSGJp6VI480yerDSIXkvu5eijhc8+gzp1wg7M5WreHN58ExYutLpc\nyThh8YYbrOjns8/aBE0XX4ceap/9jBnwz3+GHU1seCKJIb3nXm5few1XrbmLM84QPvzQmlRcYunU\nCV58ET7/3CrfZmeHHVHJeflleOwxGDTI5tK4cJx3XnL3l3giiZHsbBjIcG7bOpS+fa1JYZ99wo7K\nFaRXL/uSv/46/OMfyTFhcdYsGDAATjzRJmO6cCVzf4knkpKmytY77+P87lt44ukUBg+G55+HChXC\nDswV5rrrLIk8/rh96UuzNWtsRFr16rZmhv/7C18y95d4IilhG4bczRm3tubVt1K5/377JejlJ0qP\nBx6wjukbb7TRTaVRdjb06QPLl9sV1oEHhh2Ry5Ws/SWeSEpQ5qOv0Om+LkyVTrzwvHLDDWFH5PZW\nuXLWX9KxI/TvDx9/HHZEe+9f/4IPP4Thw+GYY8KOxuUV2V/y3/+GHU3JEE2GxuBCpKWlaXp6ekxf\nY8mLn9G570FklDuYV18vz5ndysf09VxsrVtnyxwvXQqffQatSsk6nG+8YfNFLr8cnnkm7GhcQbZs\ngfbtYfFi68s6+OCwI8qfiMxU1UJnvPkVSQn4bnYOx136VzJTDuKj97I9iSSB/fe3OSb77WcTFkvD\nYkU//miLorVta/08LnGlplrfVVaW9Zds3x52RMXjiaSYPv8cOnQsh9Sszv8+2Ez7zn8JOyRXQurV\ns9nvW7bYhMXVq8OOqGB//GGd6/vsY/0ilfJd7s0lksMOs/6S6dNLf3+JJ5Ji+O/4TZzaaTsH1FK+\n+DKFZid7r2ayadrU6nItXgxdu8LmzWFHtLucHLsSWbDARgXVqxd2RC5aPXvCVVfZII/S3F/iiaSI\nXhiVxTl9KtEsazafP/YNDRqEHZGLlQ4dbGLfl1/apL5Em7B47702O/+BB2zOiCtdHnwQjjrKfgyU\n1ppvnkiK4P77lP6XlaeTTmbyiB+p1aVN4Qe5Uu2882x98zffhGuuSZwJi++/D7fcYgnu2mvDjsYV\nRe78ktLcX+KJZC+owuDBcONNQk/G8/bQaVS56qKww3Jxcu21VrfqiSdg2LCwo4FFiyyBNG8OI0f6\nfKXSrLT3l/jwoiht325DKkePhqsrPMOjPb8g5e4Xwg7LxdmwYbBsGQwdasU3L744nDg2bYJzz7Uf\nNxMnevmdZNCzJ3z66c4myjPPDDui6MX0ikREuojIPBFZICJD8nm+n4hkisis4HZZxHN9RWR+cOsb\nsb2NiHwXnPMxkdj/Dsv90o4ebQsCPT7rBFKe85+AZVG5clby5qST4NJLbeJfvKnaj5o5c2DsWGjU\nKP4xuNgotf0l0SyjWJQbkIKt1d4IqIity94kzz79gOH5HFsdWBT8t1pwv1rw3FfAsYAA7wGnFRZL\ncZbaXbNGtX17VZEcfbL31CKfxyWXdetUW7Sw5ZJnzozvaz/yiC3hetdd8X1dFx/z56tWqaJ67LGq\n27aFGwsJsNRuW2CBqi5S1W3AOODsKI/9G/CRqq5R1bXAR0AXEakNVFXVL4M3+SLQLRbBgzVhdOgA\nX3+tvFrjKq6Y2jt5Vz9ye2W//WzCYvXqcPrpNjw4Hj79FK6/3tZOGTo0Pq/p4iu3v+TLL+Hmm8OO\nJjqxTCR1gV8jHmcE2/LqLiJzRGSCiNQv5Ni6wf3CzllsqjaCYskS5b1DrqTH5pfg7bd9eTm3Q506\nNmpq2zabsLhqVWxfLyPD2tEPO8zqgZXzoTJJq2dPuPJKuP9++7OT6GL5TzG/DoS8gyb/CzRQ1RbA\nx8DoQo6N5px2ApEBIpIuIumZmZlRhhx5PDw1PIuprQZx0sKRNj6vjQ/zdbv6619toaKlS+Gss6w/\nLRa2boUePez8EydC1aqxeR2XOB56yGq8lYb+klgmkgygfsTjesDyyB1UdbWqbg0ejgTaFHJsRnC/\nwHNGnPsZVU1T1bRatWoV6Q00WzWVNl88DiNGWPuFc/k4/ngrOT9jhpVvz8oq+de45ho7/+jRlrxc\n8sudX7J9O/TundjzS2KZSL4GGotIQxGpCPQGdllkMujzyNUV+DG4/wHQWUSqiUg1oDPwgaquADaI\nSLtgtNbFwFsxewennAKzZ8MVV8TsJVxy6N7dlrSdNAkGDizZCYvPPmuVfIcOtdGDruxo3NjmCCV6\nf0nM5pGoapaIDMSSQgrwnKr+ICJ3YCMBJgHXiEhXIAtYg43iQlXXiMidWDICuENVc3u5rwReACpj\no7bei9V7AGy2l3NRGDjQ+jGGDYP69Uvmiz9jhq1d0bkz3Hln8c/nSp9evWyQxf332+CfRJxf4uuR\nOFeCcgsovvyyzTfp16/o5/rtN+uWq1AB0tOhRo0SC9OVMlu2wLHHWl/JrFn2QyUefD0S50JQrhyM\nGmWtopddZkOEi2L7dvslunq1LVblSaRsS/T+Ek8kzpWwihVtTZDmza3YY1Euhm+6yZozRo60mc7O\n5faXTJtmhToTiScS52KgalV4912oWRPOOAMWLoz+2FdesUrD11wDF14Yuxhd6dOrl80vue8+eOed\nsKPZyROJczFSu7ZNWMzKsgmL0UxnmjPHmsROOMGK9zmXV+78kosvhl9/LXz/ePBE4lwMHXmkDQnO\nyLDRNn/+WfC+a9bYcrnVqll7eIUK8YvTlR65/SXbtiVOf4knEudirH17q9Kbnm5f/PwmLGZnwwUX\n2C/MCRPgoIPiH6crPRKtv8QTiXNx0K0bDB9udZOuumr3CYu33WbNYI8/bsM8nStM7942VzoR+ks8\nkTgXJ1deaavfjRy56+TCN9+Eu+6CSy6BAQPCi8+VPg8/DC1bht9f4iskOhdHd91l/SX//jfUrWt1\nui6+GNLSrKSbr5Xm9kZqKrz2GrRubVcoU6eG07fmVyTOxZGI1c7629/g//7PSp9UqmTzTlJTw47O\nlUaJ0F/iicS5OKtQwX5Ftmxpi6e9+iocfHDYUbnSLLK/5N134//6XmvLuZD88YfVTmrWLOxIXDLY\nsgXatbOm02+/LZl6XF5ry7kEV7WqJxFXcnL7S7Zujf/8Ek8kzjmXJCL7S/71r/i9ricS55xLIr17\n20COYcPi11/iicQ555JMvOeXeCJxzrkkU7myjQY8+uj4zE3yCYnOOZeEDj+86Aur7a2YXpGISBcR\nmSciC0RkyB726yEiKiJpweMLRGRWxC1HRFoFz00Nzpn73AGxfA/OOef2LGZXJCKSAowATgUygK9F\nZJKqzs2zXxXgGmBG7jZVHQOMCZ5vDrylqrMiDrtAVX1iiHPOJYBYXpG0BRao6iJV3QaMA87OZ787\ngfuALQWcpw8wNjYhOuecK65YJpK6QOR4gYxg2w4ichRQX1Xf3sN5erF7Ink+aNb6l0j+XUkiMkBE\n0kUkPTOapemcc84VSSwTSX5/4HfUYxGRcsDDwPUFnkDkGGCTqn4fsfkCVW0OnBDcLsrvWFV9RlXT\nVDWtVq1aRYnfOedcFGKZSDKAyGov9YDlEY+rAM2AqSKyBGgHTMrtcA/0Js/ViKouC/67AXgFa0Jz\nzjkXklgmkq+BxiLSUEQqYklhUu6TqrpeVWuqagNVbQBMB7rmdqIHVyznYX0rBNvKi0jN4H4F4Ewg\n8mrFOedcnMVs1JaqZonIQOADIAV4TlV/EJE7gHRVnbTnM9AByFDVRRHbKgEfBEkkBfgYGBmD8J1z\nzkWpTJSRF5FMYGkRD68JrCrBcEqKx7V3PK6943HtnWSN6xBVLbSTuUwkkuIQkfRo6vHHm8e1dzyu\nveNx7Z2yHpfX2nLOOVcsnkicc84ViyeSwj0TdgAF8Lj2jse1dzyuvVOm4/I+Euecc8XiVyTOOeeK\nxROJc865YvFEEihs7RQRqSQi44PnZ4hIgwSJq5+IZEasz3JZHGJ6TkR+F5F8qwqIeSyIeY6ItI51\nTFHG1VFE1kd8VrfGKa76IjJFRH4UkR9E5Np89on7ZxZlXHH/zEQkVUS+EpHZQVy357NP3L+PUcYV\n9+9jxGuniMi3IrJbEdyYf16qWuZv2Cz5hUAjoCIwG2iSZ5+rgKeC+72B8QkSVz9geJw/rw5Aa+D7\nAp4/HXgPK9zZDpiRIHF1BN4O4d9XbaB1cL8K8HM+/x/j/plFGVfcP7PgM9g3uF8BW6uoXZ59wvg+\nRhNX3L+PEa99HVZ/cLf/X7H+vPyKxESzdsrZwOjg/gTg5IJK2Mc5rrhT1c+ANXvY5WzgRTXTgf1F\npHYCxBUKVV2hqt8E9zcAP5JnSQVC+MyijCvugs9gY/CwQnDLOyoo7t/HKOMKhYjUA84Ani1gl5h+\nXp5ITKFrp0Tuo6pZwHqgRgLEBdA9aA6ZICL183k+3qKNOwzHBk0T74lI03i/eNCkcBQRK4IGQv3M\n9hAXhPCZBc00s4DfgY9UtcDPK47fx2jignC+j48ANwI5BTwf08/LE4nZ49ope7FPSYvmNf8LNFDV\nFlgRy9G7HxJ3YXxW0fgGqx3UEngceDOeLy4i+wKvA4NU9Y+8T+dzSFw+s0LiCuUzU9VsVW2FLT/R\nVkSa5dkllM8rirji/n0UkTOB31V15p52y2dbiX1enkhMYWun7LKPiJQH9iP2zSiFxqWqq1V1a/Bw\nJNAmxjFFI5rPM+5U9Y/cpglVfReoIMGyBLEmVrH6dWCMqr6Rzy6hfGaFxRXmZxa85jpgKtAlz1Nh\nfB8LjSuk72N7oKvYusXWR0wAAALASURBVE7jgJNE5OU8+8T08/JEYva4dkpgEtA3uN8DmKxBz1WY\nceVpR++KtXOHbRJwcTASqR2wXlVXhB2UiByU2y4sIm2xf/+r4/C6AowCflTVhwrYLe6fWTRxhfGZ\niUgtEdk/uF8ZOAX4Kc9ucf8+RhNXGN9HVR2qqvXU1nXqjX0WF+bZLaafV8zWIylNNLq1U0YBL4nI\nAiyT906QuK4Rka5AVhBXv1jHJSJjsdE8NUUkA/g31vGIqj4FvIuNQloAbAL6xzqmKOPqAVwpIlnA\nZqB3HH4MgP1ivAj4LmhfB/gncHBEbGF8ZtHEFcZnVhsYLSIpWOJ6VVXfDvv7GGVccf8+FiSen5eX\nSHHOOVcs3rTlnHOuWDyROOecKxZPJM4554rFE4lzzrli8UTinHOuWDyROFcCRCQ7ouLrLMmnUnMx\nzt1ACqho7Fwi8HkkzpWMzUHpDOfKHL8icS6GRGSJiAwL1rH4SkQOC7YfIiKfBMX9PhGRg4PtB4rI\nxKBI4mwROS44VYqIjBRbB+PDYGa1cwnBE4lzJaNynqatXhHP/aGqbYHhWJVWgvsvBsX9xgCPBdsf\nAz4NiiS2Bn4ItjcGRqhqU2Ad0D3G78e5qPnMdudKgIhsVNV989m+BDhJVRcFBRJXqmoNEVkF1FbV\n7cH2FapaU0QygXoRhf9yS7x/pKqNg8c3ARVU9a7YvzPnCudXJM7FnhZwv6B98rM14n423r/pEogn\nEudir1fEf78M7k9jZ+G8C4DPg/ufAFfCjkWUqsYrSOeKyn/VOFcyKkdU0AV4X1VzhwBXEpEZ2A+3\nPsG2a4DnRGQwkMnOar/XAs+IyKXYlceVQOgl+J3bE+8jcS6Ggj6SNFVdFXYszsWKN20555wrFr8i\ncc45Vyx+ReKcc65YPJE455wrFk8kzjnnisUTiXPOuWLxROKcc65Y/h8+S78Ll5hxjQAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8bb770ceb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_history(history):\n",
    "    plt.plot(history.history['loss'],'r--')\n",
    "    plt.plot(history.history['val_loss'],'b-')\n",
    "    plt.title('Model Loss')\n",
    "    plt.legend(['Train Loss', 'Valid Loss'])\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.ylabel('Loss')\n",
    "    plt.show();\n",
    "    plt.plot(history.history['acc'],'r--')\n",
    "    plt.plot(history.history['val_acc'],'b-')\n",
    "    plt.title('Model Accuracy')\n",
    "    plt.legend(['Train Acc', 'Valid Acc'])\n",
    "    plt.ylabel('Accuracy')\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.show()\n",
    "\n",
    "# requires history=model.fit, fit_generator...\n",
    "plot_history(history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_from_scratch.load_weights(weights_from_scratch)\n",
    "model_pt_frozen.load_weights(weights_pt_frozen)\n",
    "model_pt_trainable.load_weights(weights_pt_trainable)\n",
    "model_no_embedding.load_weights(weights_no_embedding)\n",
    "model_bidirectional.load_weights(weights_bidirectional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'X_test' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-35-63528c60fae8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mX_data\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_test\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mX_data_enc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX_test_enc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_test\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"--- FROM SCRATCH EMBEDDINGS ---\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0macc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel_from_scratch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_data\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_data\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'loss / accuracy = {:.4f} / {:.4f}'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0macc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'X_test' is not defined"
     ]
    }
   ],
   "source": [
    "X_data, y_data = X_test, y_test\n",
    "X_data_enc, y_data = X_test_enc, y_test\n",
    "print(\"--- FROM SCRATCH EMBEDDINGS ---\")\n",
    "loss, acc = model_from_scratch.evaluate(X_data, y_data, batch_size)\n",
    "print('loss / accuracy = {:.4f} / {:.4f}'.format(loss, acc))\n",
    "print(\"--- ARABIC EMBEDDINGS - FROZEN ---\")\n",
    "loss, acc = model_pt_frozen.evaluate(X_data, y_data, batch_size)\n",
    "print('loss / accuracy = {:.4f} / {:.4f}'.format(loss, acc))\n",
    "print(\"--- ARABIC EMBEDDINGS - TRAINABLE ---\")\n",
    "loss, acc = model_pt_trainable.evaluate(X_data, y_data, batch_size)\n",
    "print('loss / accuracy = {:.4f} / {:.4f}'.format(loss, acc))\n",
    "print(\"--- NO EMBEDDING ---\")\n",
    "loss, acc = model_no_embedding.evaluate(X_data_enc, y_data, batch_size)\n",
    "print('loss / accuracy = {:.4f} / {:.4f}'.format(loss, acc))\n",
    "print(\"--- BIDIRECTIONAL - ARABIC EMBEDDINGS - TRAINABLE ---\")\n",
    "loss, acc = model_bidirectional.evaluate(X_data, y_data, batch_size)\n",
    "print('loss / accuracy = {:.4f} / {:.4f}'.format(loss, acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(525, 85)\n",
      "(525, 2)\n"
     ]
    }
   ],
   "source": [
    "X_erik_train, y_erik_train, _ = get_data(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Erik/train\")\n",
    "X_erik_valid, y_erik_valid, _ = get_data(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Erik/valid\")\n",
    "X_erik_test, y_erik_test, _ = get_data(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Erik/test\")\n",
    "X_erik = np.concatenate((X_erik_train,X_erik_valid,X_erik_test), axis=0)\n",
    "y_erik = np.concatenate((y_erik_train,y_erik_valid,y_erik_test), axis=0)\n",
    "print(X_erik.shape)\n",
    "print(y_erik.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(525, 85, 300)\n",
      "(525, 2)\n"
     ]
    }
   ],
   "source": [
    "X_erik_train_enc, y_erik_train, _ = get_data_encoded(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Erik/train\")\n",
    "X_erik_valid_enc, y_erik_valid, _ = get_data_encoded(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Erik/valid\")\n",
    "X_erik_test_enc, y_erik_test, _ = get_data_encoded(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Erik/test\")\n",
    "X_erik_enc = np.concatenate((X_erik_train_enc,X_erik_valid_enc,X_erik_test_enc), axis=0)\n",
    "y_erik = np.concatenate((y_erik_train,y_erik_valid,y_erik_test), axis=0)\n",
    "print(X_erik_enc.shape)\n",
    "print(y_erik.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2000, 85)\n",
      "(2000, 2)\n"
     ]
    }
   ],
   "source": [
    "X_twitter_train, y_twitter_train, _ = get_data(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Twitter/Combined/train\")\n",
    "X_twitter_valid, y_twitter_valid, _ = get_data(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Twitter/Combined/valid\")\n",
    "X_twitter = np.concatenate((X_twitter_train,X_twitter_valid), axis=0)\n",
    "y_twitter = np.concatenate((y_twitter_train,y_twitter_valid), axis=0)\n",
    "print(X_twitter.shape)\n",
    "print(y_twitter.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2000, 85, 300)\n",
      "(2000, 2)\n"
     ]
    }
   ],
   "source": [
    "X_twitter_train_enc, y_twitter_train, _ = get_data_encoded(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Twitter/Combined/train\")\n",
    "X_twitter_valid_enc, y_twitter_valid, _ = get_data_encoded(\"/home/zmccoy/sandbox/sentiment-analysis-arabic/Twitter/Combined/valid\")\n",
    "X_twitter_enc = np.concatenate((X_twitter_train_enc,X_twitter_valid_enc), axis=0)\n",
    "y_twitter = np.concatenate((y_twitter_train,y_twitter_valid), axis=0)\n",
    "print(X_twitter_enc.shape)\n",
    "print(y_twitter.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_data, y_data = X_erik, y_erik\n",
    "X_data_enc, y_data = X_erik_enc, y_erik\n",
    "#X_data, y_data = X_twitter, y_twitter\n",
    "#X_data_enc, y_data = X_twitter_enc, y_twitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- FROM SCRATCH EMBEDDINGS ---\n",
      "525/525 [==============================] - 2s 4ms/step\n",
      "loss / accuracy = 0.7023 / 0.1981\n",
      "--- ARABIC EMBEDDINGS - FROZEN ---\n",
      "525/525 [==============================] - 2s 3ms/step\n",
      "loss / accuracy = 0.7333 / 0.1105\n",
      "--- ARABIC EMBEDDINGS - TRAINABLE ---\n",
      "525/525 [==============================] - 2s 4ms/step\n",
      "loss / accuracy = 0.7142 / 0.2686\n",
      "--- NO EMBEDDING ---\n",
      "525/525 [==============================] - 2s 3ms/step\n",
      "loss / accuracy = 0.6926 / 0.6152\n",
      "--- BIDIRECTIONAL - ARABIC EMBEDDINGS - TRAINABLE ---\n",
      "525/525 [==============================] - 1s 2ms/step\n",
      "loss / accuracy = 0.6336 / 0.8648\n"
     ]
    }
   ],
   "source": [
    "print(\"--- FROM SCRATCH EMBEDDINGS ---\")\n",
    "loss, acc = model_from_scratch.evaluate(X_data, y_data, batch_size)\n",
    "print('loss / accuracy = {:.4f} / {:.4f}'.format(loss, acc))\n",
    "print(\"--- ARABIC EMBEDDINGS - FROZEN ---\")\n",
    "loss, acc = model_pt_frozen.evaluate(X_data, y_data, batch_size)\n",
    "print('loss / accuracy = {:.4f} / {:.4f}'.format(loss, acc))\n",
    "print(\"--- ARABIC EMBEDDINGS - TRAINABLE ---\")\n",
    "loss, acc = model_pt_trainable.evaluate(X_data, y_data, batch_size)\n",
    "print('loss / accuracy = {:.4f} / {:.4f}'.format(loss, acc))\n",
    "print(\"--- NO EMBEDDING ---\")\n",
    "loss, acc = model_no_embedding.evaluate(X_data_enc, y_data, batch_size)\n",
    "print('loss / accuracy = {:.4f} / {:.4f}'.format(loss, acc))\n",
    "print(\"--- BIDIRECTIONAL - ARABIC EMBEDDINGS - TRAINABLE ---\")\n",
    "loss, acc = model_bidirectional.evaluate(X_data, y_data, batch_size)\n",
    "print('loss / accuracy = {:.4f} / {:.4f}'.format(loss, acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.43361151  0.56638849]]\n",
      "[0 1]\n",
      "[[ 0.43797344  0.56202662]]\n",
      "[0 1]\n",
      "[[ 0.4871538   0.51284623]]\n",
      "[0 1]\n",
      "[[ 0.49115032  0.50884968]]\n",
      "[0 1]\n",
      "[[ 0.50159067  0.49840936]]\n",
      "[0 1]\n",
      "[[ 0.48351571  0.51648432]]\n",
      "[0 1]\n",
      "[[ 0.49443078  0.50556916]]\n",
      "[0 1]\n",
      "[[ 0.45593712  0.54406291]]\n",
      "[0 1]\n",
      "[[ 0.45716313  0.54283684]]\n",
      "[0 1]\n",
      "[[ 0.44520226  0.55479771]]\n",
      "[0 1]\n",
      "[[ 0.44149619  0.55850375]]\n",
      "[0 1]\n",
      "[[ 0.46386287  0.53613716]]\n",
      "[0 1]\n",
      "[[ 0.4507618  0.5492382]]\n",
      "[0 1]\n",
      "[[ 0.49024183  0.50975811]]\n",
      "[0 1]\n",
      "[[ 0.46664053  0.53335947]]\n",
      "[0 1]\n",
      "[[ 0.48411196  0.51588804]]\n",
      "[0 1]\n",
      "[[ 0.51821077  0.48178923]]\n",
      "[1 0]\n",
      "[[ 0.43972757  0.56027246]]\n",
      "[0 1]\n",
      "[[ 0.47426388  0.52573609]]\n",
      "[0 1]\n",
      "[[ 0.41714111  0.58285892]]\n",
      "[0 1]\n",
      "[[ 0.42593279  0.57406718]]\n",
      "[0 1]\n",
      "[[ 0.43666127  0.56333876]]\n",
      "[1 0]\n",
      "[[ 0.43590912  0.56409079]]\n",
      "[0 1]\n",
      "[[ 0.43561319  0.56438679]]\n",
      "[0 1]\n",
      "[[ 0.47973943  0.52026051]]\n",
      "[0 1]\n",
      "[[ 0.47588331  0.52411669]]\n",
      "[0 1]\n",
      "[[ 0.42623574  0.57376426]]\n",
      "[0 1]\n",
      "[[ 0.46010175  0.53989828]]\n",
      "[0 1]\n",
      "[[ 0.4254297  0.5745703]]\n",
      "[0 1]\n",
      "[[ 0.50159067  0.49840936]]\n",
      "[0 1]\n",
      "[[ 0.47870195  0.52129805]]\n",
      "[0 1]\n",
      "[[ 0.46729136  0.53270864]]\n",
      "[0 1]\n",
      "[[ 0.40626103  0.59373891]]\n",
      "[0 1]\n",
      "[[ 0.45471418  0.54528588]]\n",
      "[0 1]\n",
      "[[ 0.49009073  0.50990933]]\n",
      "[0 1]\n",
      "[[ 0.50159067  0.49840936]]\n",
      "[1 0]\n",
      "[[ 0.43476626  0.56523377]]\n",
      "[0 1]\n",
      "[[ 0.50827146  0.49172848]]\n",
      "[0 1]\n",
      "[[ 0.45688567  0.54311436]]\n",
      "[0 1]\n",
      "[[ 0.45479596  0.54520398]]\n",
      "[0 1]\n",
      "[[ 0.42565572  0.57434428]]\n",
      "[0 1]\n",
      "[[ 0.46804997  0.53195006]]\n",
      "[1 0]\n",
      "[[ 0.41170704  0.58829296]]\n",
      "[0 1]\n",
      "[[ 0.45844585  0.54155415]]\n",
      "[0 1]\n",
      "[[ 0.41201028  0.58798969]]\n",
      "[0 1]\n",
      "[[ 0.4289676  0.5710324]]\n",
      "[0 1]\n",
      "[[ 0.43569982  0.56430018]]\n",
      "[0 1]\n",
      "[[ 0.43929607  0.56070399]]\n",
      "[0 1]\n",
      "[[ 0.46289632  0.53710371]]\n",
      "[0 1]\n",
      "[[ 0.48837706  0.51162297]]\n",
      "[0 1]\n",
      "[[ 0.48038602  0.51961398]]\n",
      "[0 1]\n",
      "[[ 0.44383281  0.55616713]]\n",
      "[0 1]\n",
      "[[ 0.4277935   0.57220644]]\n",
      "[0 1]\n",
      "[[ 0.42805383  0.5719462 ]]\n",
      "[0 1]\n",
      "[[ 0.45687497  0.54312497]]\n",
      "[0 1]\n",
      "[[ 0.49380204  0.50619793]]\n",
      "[0 1]\n",
      "[[ 0.47618902  0.52381092]]\n",
      "[0 1]\n",
      "[[ 0.44099632  0.55900365]]\n",
      "[0 1]\n",
      "[[ 0.43166459  0.56833541]]\n",
      "[0 1]\n",
      "[[ 0.48198736  0.51801258]]\n",
      "[0 1]\n",
      "[[ 0.46729136  0.53270864]]\n",
      "[0 1]\n",
      "[[ 0.44240358  0.55759645]]\n",
      "[0 1]\n",
      "[[ 0.48411196  0.51588804]]\n",
      "[0 1]\n",
      "[[ 0.44099632  0.55900365]]\n",
      "[0 1]\n",
      "[[ 0.43569982  0.56430018]]\n",
      "[0 1]\n",
      "[[ 0.46858433  0.5314157 ]]\n",
      "[0 1]\n",
      "[[ 0.49958682  0.50041318]]\n",
      "[0 1]\n",
      "[[ 0.48144183  0.5185582 ]]\n",
      "[0 1]\n",
      "[[ 0.43604404  0.56395596]]\n",
      "[1 0]\n",
      "[[ 0.44703251  0.55296743]]\n",
      "[0 1]\n",
      "[[ 0.42345694  0.57654303]]\n",
      "[0 1]\n",
      "[[ 0.44361818  0.55638182]]\n",
      "[0 1]\n",
      "[[ 0.48522645  0.51477349]]\n",
      "[0 1]\n",
      "[[ 0.44519788  0.55480212]]\n",
      "[1 0]\n",
      "[[ 0.48962149  0.51037848]]\n",
      "[0 1]\n",
      "[[ 0.44966295  0.55033702]]\n",
      "[0 1]\n",
      "[[ 0.456065    0.54393506]]\n",
      "[0 1]\n",
      "[[ 0.4484804   0.55151963]]\n",
      "[0 1]\n",
      "[[ 0.41062292  0.58937711]]\n",
      "[0 1]\n",
      "[[ 0.46443361  0.53556645]]\n",
      "[1 0]\n",
      "[[ 0.45287526  0.54712468]]\n",
      "[1 0]\n",
      "[[ 0.44769126  0.55230868]]\n",
      "[0 1]\n",
      "[[ 0.46892673  0.53107327]]\n",
      "[0 1]\n",
      "[[ 0.46842983  0.53157014]]\n",
      "[0 1]\n",
      "[[ 0.50178379  0.49821615]]\n",
      "[0 1]\n",
      "[[ 0.49982864  0.50017136]]\n",
      "[0 1]\n",
      "[[ 0.44273445  0.55726558]]\n",
      "[0 1]\n",
      "[[ 0.41614276  0.5838573 ]]\n",
      "[0 1]\n",
      "[[ 0.48736233  0.51263767]]\n",
      "[0 1]\n",
      "[[ 0.43686631  0.56313372]]\n",
      "[1 0]\n",
      "[[ 0.47704172  0.52295828]]\n",
      "[0 1]\n",
      "[[ 0.46808073  0.5319193 ]]\n",
      "[0 1]\n",
      "[[ 0.4463124   0.55368763]]\n",
      "[0 1]\n",
      "[[ 0.45406279  0.54593718]]\n",
      "[0 1]\n",
      "[[ 0.4449676   0.55503237]]\n",
      "[0 1]\n",
      "[[ 0.48652434  0.5134756 ]]\n",
      "[0 1]\n",
      "[[ 0.46976495  0.53023511]]\n",
      "[0 1]\n",
      "[[ 0.44921529  0.55078477]]\n",
      "[0 1]\n",
      "[[ 0.44552839  0.55447155]]\n",
      "[0 1]\n",
      "[[ 0.4065432  0.5934568]]\n",
      "[0 1]\n",
      "[[ 0.4349533   0.56504667]]\n",
      "[0 1]\n",
      "[[ 0.48107412  0.51892585]]\n",
      "[1 0]\n",
      "[[ 0.43432617  0.56567383]]\n",
      "[1 0]\n",
      "[[ 0.47943375  0.52056628]]\n",
      "[0 1]\n",
      "[[ 0.4681384   0.53186166]]\n",
      "[0 1]\n",
      "[[ 0.46096474  0.53903526]]\n",
      "[1 0]\n",
      "[[ 0.45355791  0.54644209]]\n",
      "[0 1]\n",
      "[[ 0.47882715  0.52117282]]\n",
      "[0 1]\n",
      "[[ 0.4210535   0.57894659]]\n",
      "[0 1]\n",
      "[[ 0.4402045  0.5597955]]\n",
      "[0 1]\n",
      "[[ 0.42395091  0.57604909]]\n",
      "[0 1]\n",
      "[[ 0.48921779  0.5107823 ]]\n",
      "[0 1]\n",
      "[[ 0.47587219  0.52412772]]\n",
      "[0 1]\n",
      "[[ 0.45069894  0.54930109]]\n",
      "[0 1]\n",
      "[[ 0.43889824  0.56110179]]\n",
      "[0 1]\n",
      "[[ 0.46124861  0.53875136]]\n",
      "[0 1]\n",
      "[[ 0.46582514  0.53417492]]\n",
      "[0 1]\n",
      "[[ 0.4351317   0.56486833]]\n",
      "[0 1]\n",
      "[[ 0.47643104  0.52356893]]\n",
      "[0 1]\n",
      "[[ 0.43158114  0.56841892]]\n",
      "[0 1]\n",
      "[[ 0.51009476  0.48990524]]\n",
      "[0 1]\n",
      "[[ 0.42422143  0.5757786 ]]\n",
      "[0 1]\n",
      "[[ 0.47943375  0.52056628]]\n",
      "[0 1]\n",
      "[[ 0.49011591  0.50988412]]\n",
      "[0 1]\n",
      "[[ 0.46576256  0.53423744]]\n",
      "[0 1]\n",
      "[[ 0.41997021  0.58002979]]\n",
      "[0 1]\n",
      "[[ 0.42519635  0.57480359]]\n",
      "[1 0]\n",
      "[[ 0.45699164  0.54300833]]\n",
      "[0 1]\n",
      "[[ 0.47643682  0.52356315]]\n",
      "[0 1]\n",
      "[[ 0.43596953  0.56403047]]\n",
      "[0 1]\n",
      "[[ 0.47418252  0.52581751]]\n",
      "[0 1]\n",
      "[[ 0.46653906  0.53346097]]\n",
      "[0 1]\n",
      "[[ 0.44478101  0.55521899]]\n",
      "[0 1]\n",
      "[[ 0.43041405  0.56958598]]\n",
      "[0 1]\n",
      "[[ 0.42613664  0.57386333]]\n",
      "[0 1]\n",
      "[[ 0.40347156  0.59652841]]\n",
      "[0 1]\n",
      "[[ 0.44799063  0.55200934]]\n",
      "[0 1]\n",
      "[[ 0.50540066  0.49459934]]\n",
      "[0 1]\n",
      "[[ 0.44043255  0.55956745]]\n",
      "[0 1]\n",
      "[[ 0.46601784  0.53398216]]\n",
      "[0 1]\n",
      "[[ 0.44462824  0.55537182]]\n",
      "[0 1]\n",
      "[[ 0.44799161  0.55200839]]\n",
      "[0 1]\n",
      "[[ 0.41594216  0.58405787]]\n",
      "[0 1]\n",
      "[[ 0.43034244  0.56965756]]\n",
      "[0 1]\n",
      "[[ 0.44157556  0.55842441]]\n",
      "[0 1]\n",
      "[[ 0.49942562  0.50057435]]\n",
      "[0 1]\n",
      "[[ 0.43166986  0.56833017]]\n",
      "[0 1]\n",
      "[[ 0.46618563  0.53381437]]\n",
      "[1 0]\n",
      "[[ 0.46154377  0.5384562 ]]\n",
      "[1 0]\n",
      "[[ 0.44762099  0.55237901]]\n",
      "[0 1]\n",
      "[[ 0.40501213  0.59498781]]\n",
      "[0 1]\n",
      "[[ 0.46711642  0.53288358]]\n",
      "[1 0]\n",
      "[[ 0.43561894  0.564381  ]]\n",
      "[0 1]\n",
      "[[ 0.46105793  0.53894204]]\n",
      "[0 1]\n",
      "[[ 0.43896109  0.56103891]]\n",
      "[0 1]\n",
      "[[ 0.43537587  0.56462413]]\n",
      "[0 1]\n",
      "[[ 0.4759607   0.52403927]]\n",
      "[0 1]\n",
      "[[ 0.4553026   0.54469746]]\n",
      "[1 0]\n",
      "[[ 0.41714111  0.58285892]]\n",
      "[0 1]\n",
      "[[ 0.44066313  0.5593369 ]]\n",
      "[0 1]\n",
      "[[ 0.49696636  0.50303364]]\n",
      "[0 1]\n",
      "[[ 0.42695856  0.57304144]]\n",
      "[0 1]\n",
      "[[ 0.45350614  0.54649389]]\n",
      "[0 1]\n",
      "[[ 0.44634798  0.55365205]]\n",
      "[0 1]\n",
      "[[ 0.47886738  0.52113265]]\n",
      "[1 0]\n",
      "[[ 0.43646225  0.56353772]]\n",
      "[0 1]\n",
      "[[ 0.47943375  0.52056628]]\n",
      "[0 1]\n",
      "[[ 0.47573704  0.52426296]]\n",
      "[0 1]\n",
      "[[ 0.49696636  0.50303364]]\n",
      "[0 1]\n",
      "[[ 0.45721361  0.54278642]]\n",
      "[0 1]\n",
      "[[ 0.44998667  0.5500133 ]]\n",
      "[0 1]\n",
      "[[ 0.48317906  0.51682097]]\n",
      "[0 1]\n",
      "[[ 0.47664785  0.52335209]]\n",
      "[0 1]\n",
      "[[ 0.41852501  0.58147508]]\n",
      "[0 1]\n",
      "[[ 0.46029475  0.53970528]]\n",
      "[0 1]\n",
      "[[ 0.47498146  0.52501851]]\n",
      "[0 1]\n",
      "[[ 0.48565847  0.51434153]]\n",
      "[0 1]\n",
      "[[ 0.4625707   0.53742927]]\n",
      "[1 0]\n",
      "[[ 0.43146861  0.56853139]]\n",
      "[1 0]\n",
      "[[ 0.47806242  0.52193761]]\n",
      "[0 1]\n",
      "[[ 0.43590912  0.56409079]]\n",
      "[0 1]\n",
      "[[ 0.44457754  0.55542243]]\n",
      "[0 1]\n",
      "[[ 0.46211922  0.53788078]]\n",
      "[0 1]\n",
      "[[ 0.4652158  0.5347842]]\n",
      "[1 0]\n",
      "[[ 0.49038857  0.50961137]]\n",
      "[0 1]\n",
      "[[ 0.47346094  0.52653909]]\n",
      "[0 1]\n",
      "[[ 0.49958682  0.50041318]]\n",
      "[0 1]\n",
      "[[ 0.48813424  0.51186579]]\n",
      "[0 1]\n",
      "[[ 0.44812837  0.55187166]]\n",
      "[0 1]\n",
      "[[ 0.4676879  0.5323121]]\n",
      "[0 1]\n",
      "[[ 0.51288563  0.48711437]]\n",
      "[0 1]\n",
      "[[ 0.41873583  0.58126414]]\n",
      "[0 1]\n",
      "[[ 0.48068467  0.5193153 ]]\n",
      "[0 1]\n",
      "[[ 0.40909579  0.59090424]]\n",
      "[0 1]\n",
      "[[ 0.4458648   0.55413526]]\n",
      "[0 1]\n",
      "[[ 0.46711642  0.53288358]]\n",
      "[0 1]\n",
      "[[ 0.48829207  0.5117079 ]]\n",
      "[0 1]\n",
      "[[ 0.49182135  0.50817859]]\n",
      "[0 1]\n",
      "[[ 0.40897518  0.59102482]]\n",
      "[0 1]\n",
      "[[ 0.43975922  0.56024075]]\n",
      "[1 0]\n",
      "[[ 0.47265539  0.52734464]]\n",
      "[0 1]\n",
      "[[ 0.48355865  0.51644135]]\n",
      "[0 1]\n",
      "[[ 0.4161019   0.58389807]]\n",
      "[0 1]\n",
      "[[ 0.49162716  0.5083729 ]]\n",
      "[0 1]\n",
      "[[ 0.47263542  0.52736467]]\n",
      "[0 1]\n",
      "[[ 0.49443078  0.50556916]]\n",
      "[0 1]\n",
      "[[ 0.48276493  0.51723504]]\n",
      "[0 1]\n",
      "[[ 0.4591803  0.5408197]]\n",
      "[1 0]\n",
      "[[ 0.4519721   0.54802793]]\n",
      "[0 1]\n",
      "[[ 0.44169879  0.55830115]]\n",
      "[0 1]\n",
      "[[ 0.42601064  0.57398933]]\n",
      "[0 1]\n",
      "[[ 0.44422656  0.5557735 ]]\n",
      "[0 1]\n",
      "[[ 0.4920615  0.5079385]]\n",
      "[0 1]\n",
      "[[ 0.42573473  0.57426524]]\n",
      "[0 1]\n",
      "[[ 0.46759588  0.53240412]]\n",
      "[0 1]\n",
      "[[ 0.451974    0.54802603]]\n",
      "[0 1]\n",
      "[[ 0.51171339  0.48828661]]\n",
      "[0 1]\n",
      "[[ 0.42903119  0.57096881]]\n",
      "[0 1]\n",
      "[[ 0.48718613  0.51281393]]\n",
      "[0 1]\n",
      "[[ 0.47246793  0.52753204]]\n",
      "[0 1]\n",
      "[[ 0.485021  0.514979]]\n",
      "[0 1]\n",
      "[[ 0.45836982  0.54163021]]\n",
      "[0 1]\n",
      "[[ 0.44801274  0.55198717]]\n",
      "[0 1]\n",
      "[[ 0.48071426  0.5192858 ]]\n",
      "[0 1]\n",
      "[[ 0.44038269  0.55961728]]\n",
      "[1 0]\n",
      "[[ 0.46729136  0.53270864]]\n",
      "[0 1]\n",
      "[[ 0.45114204  0.54885799]]\n",
      "[1 0]\n",
      "[[ 0.47335497  0.526645  ]]\n",
      "[0 1]\n",
      "[[ 0.4614931   0.53850693]]\n",
      "[0 1]\n",
      "[[ 0.42422143  0.5757786 ]]\n",
      "[0 1]\n",
      "[[ 0.47180033  0.52819973]]\n",
      "[0 1]\n",
      "[[ 0.48787627  0.5121237 ]]\n",
      "[0 1]\n",
      "[[ 0.47797748  0.52202255]]\n",
      "[0 1]\n",
      "[[ 0.48763692  0.51236308]]\n",
      "[0 1]\n",
      "[[ 0.42468622  0.57531375]]\n",
      "[0 1]\n",
      "[[ 0.49290577  0.50709426]]\n",
      "[0 1]\n",
      "[[ 0.42116553  0.57883447]]\n",
      "[0 1]\n",
      "[[ 0.46044576  0.53955424]]\n",
      "[0 1]\n",
      "[[ 0.44416979  0.55583018]]\n",
      "[0 1]\n",
      "[[ 0.44180903  0.55819088]]\n",
      "[0 1]\n",
      "[[ 0.46546301  0.53453702]]\n",
      "[0 1]\n",
      "[[ 0.48944274  0.51055717]]\n",
      "[1 0]\n",
      "[[ 0.48010823  0.51989174]]\n",
      "[0 1]\n",
      "[[ 0.45695633  0.54304373]]\n",
      "[0 1]\n",
      "[[ 0.43935934  0.56064069]]\n",
      "[0 1]\n",
      "[[ 0.4278726  0.5721274]]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1]\n",
      "[[ 0.44072333  0.5592767 ]]\n",
      "[0 1]\n",
      "[[ 0.4882955  0.5117045]]\n",
      "[0 1]\n",
      "[[ 0.45699164  0.54300833]]\n",
      "[0 1]\n",
      "[[ 0.48953336  0.51046664]]\n",
      "[0 1]\n",
      "[[ 0.48571095  0.51428908]]\n",
      "[0 1]\n",
      "[[ 0.44676372  0.55323631]]\n",
      "[0 1]\n",
      "[[ 0.46289632  0.53710371]]\n",
      "[0 1]\n",
      "[[ 0.4418968  0.5581032]]\n",
      "[1 0]\n",
      "[[ 0.47830251  0.52169746]]\n",
      "[0 1]\n",
      "[[ 0.41850203  0.58149797]]\n",
      "[0 1]\n",
      "[[ 0.49415597  0.50584406]]\n",
      "[0 1]\n",
      "[[ 0.49355856  0.50644147]]\n",
      "[0 1]\n",
      "[[ 0.42519635  0.57480359]]\n",
      "[0 1]\n",
      "[[ 0.4284437   0.57155633]]\n",
      "[0 1]\n",
      "[[ 0.46717709  0.53282297]]\n",
      "[0 1]\n",
      "[[ 0.44764605  0.55235398]]\n",
      "[0 1]\n",
      "[[ 0.44676372  0.55323631]]\n",
      "[0 1]\n",
      "[[ 0.43975922  0.56024075]]\n",
      "[1 0]\n",
      "[[ 0.49456522  0.50543475]]\n",
      "[1 0]\n",
      "[[ 0.4634729   0.53652704]]\n",
      "[0 1]\n",
      "[[ 0.45656061  0.54343939]]\n",
      "[0 1]\n",
      "[[ 0.47240064  0.52759939]]\n",
      "[1 0]\n",
      "[[ 0.47590959  0.52409041]]\n",
      "[0 1]\n",
      "[[ 0.4458648   0.55413526]]\n",
      "[0 1]\n",
      "[[ 0.47170439  0.52829564]]\n",
      "[0 1]\n",
      "[[ 0.50159067  0.49840936]]\n",
      "[0 1]\n",
      "[[ 0.47223482  0.52776527]]\n",
      "[1 0]\n",
      "[[ 0.49223778  0.50776219]]\n",
      "[0 1]\n",
      "[[ 0.45999831  0.54000169]]\n",
      "[0 1]\n",
      "[[ 0.43026564  0.56973433]]\n",
      "[0 1]\n",
      "[[ 0.46118307  0.53881687]]\n",
      "[0 1]\n",
      "[[ 0.42949817  0.5705018 ]]\n",
      "[0 1]\n",
      "[[ 0.45952389  0.54047608]]\n",
      "[0 1]\n",
      "[[ 0.47889113  0.52110893]]\n",
      "[0 1]\n",
      "[[ 0.43972757  0.56027246]]\n",
      "[0 1]\n",
      "[[ 0.4528003   0.54719967]]\n",
      "[0 1]\n",
      "[[ 0.47502267  0.52497727]]\n",
      "[0 1]\n",
      "[[ 0.42431444  0.57568556]]\n",
      "[0 1]\n",
      "[[ 0.46047413  0.53952587]]\n",
      "[0 1]\n",
      "[[ 0.45594165  0.54405832]]\n",
      "[0 1]\n",
      "[[ 0.4632104   0.53678966]]\n",
      "[0 1]\n",
      "[[ 0.48089662  0.51910341]]\n",
      "[0 1]\n",
      "[[ 0.45119831  0.54880166]]\n",
      "[0 1]\n",
      "[[ 0.4544414   0.54555863]]\n",
      "[0 1]\n",
      "[[ 0.44614068  0.55385929]]\n",
      "[0 1]\n",
      "[[ 0.48068467  0.5193153 ]]\n",
      "[0 1]\n",
      "[[ 0.46787545  0.53212458]]\n",
      "[0 1]\n",
      "[[ 0.49095374  0.50904626]]\n",
      "[0 1]\n",
      "[[ 0.44852448  0.55147552]]\n",
      "[0 1]\n",
      "[[ 0.43842328  0.56157666]]\n",
      "[1 0]\n",
      "[[ 0.49615216  0.50384784]]\n",
      "[0 1]\n",
      "[[ 0.45114204  0.54885799]]\n",
      "[0 1]\n",
      "[[ 0.47109109  0.52890885]]\n",
      "[0 1]\n",
      "[[ 0.42250594  0.57749403]]\n",
      "[0 1]\n",
      "[[ 0.49118504  0.50881493]]\n",
      "[0 1]\n",
      "[[ 0.50279546  0.4972046 ]]\n",
      "[0 1]\n",
      "[[ 0.4765783  0.5234217]]\n",
      "[0 1]\n",
      "[[ 0.47738793  0.52261209]]\n",
      "[1 0]\n",
      "[[ 0.4357464   0.56425357]]\n",
      "[0 1]\n",
      "[[ 0.45353356  0.54646647]]\n",
      "[0 1]\n",
      "[[ 0.47943375  0.52056628]]\n",
      "[0 1]\n",
      "[[ 0.4664112   0.53358877]]\n",
      "[0 1]\n",
      "[[ 0.4544414   0.54555863]]\n",
      "[0 1]\n",
      "[[ 0.49443078  0.50556916]]\n",
      "[0 1]\n",
      "[[ 0.4575524   0.54244757]]\n",
      "[0 1]\n",
      "[[ 0.49603578  0.50396425]]\n",
      "[0 1]\n",
      "[[ 0.47418252  0.52581751]]\n",
      "[0 1]\n",
      "[[ 0.48047385  0.51952624]]\n",
      "[0 1]\n",
      "[[ 0.44150305  0.55849689]]\n",
      "[0 1]\n",
      "[[ 0.48039839  0.51960158]]\n",
      "[0 1]\n",
      "[[ 0.42468622  0.57531375]]\n",
      "[1 0]\n",
      "[[ 0.48067722  0.51932275]]\n",
      "[0 1]\n",
      "[[ 0.46808073  0.5319193 ]]\n",
      "[0 1]\n",
      "[[ 0.41560432  0.58439565]]\n",
      "[0 1]\n",
      "[[ 0.47964746  0.5203526 ]]\n",
      "[0 1]\n",
      "[[ 0.44776943  0.55223054]]\n",
      "[0 1]\n",
      "[[ 0.474208    0.52579194]]\n",
      "[0 1]\n",
      "[[ 0.48320872  0.51679128]]\n",
      "[0 1]\n",
      "[[ 0.48447394  0.51552606]]\n",
      "[0 1]\n",
      "[[ 0.45925733  0.54074264]]\n",
      "[0 1]\n",
      "[[ 0.48006809  0.51993197]]\n",
      "[0 1]\n",
      "[[ 0.50508624  0.49491376]]\n",
      "[0 1]\n",
      "[[ 0.45585704  0.54414296]]\n",
      "[0 1]\n",
      "[[ 0.45344239  0.54655755]]\n",
      "[1 0]\n",
      "[[ 0.4706988   0.52930123]]\n",
      "[0 1]\n",
      "[[ 0.48223546  0.51776451]]\n",
      "[0 1]\n",
      "[[ 0.4317759   0.56822413]]\n",
      "[0 1]\n",
      "[[ 0.45844585  0.54155415]]\n",
      "[0 1]\n",
      "[[ 0.44042841  0.55957156]]\n",
      "[1 0]\n",
      "[[ 0.41715872  0.58284128]]\n",
      "[0 1]\n",
      "[[ 0.42953795  0.57046199]]\n",
      "[0 1]\n",
      "[[ 0.46337318  0.53662682]]\n",
      "[0 1]\n",
      "[[ 0.45585704  0.54414296]]\n",
      "[0 1]\n",
      "[[ 0.44646549  0.55353451]]\n",
      "[0 1]\n",
      "[[ 0.48260713  0.51739287]]\n",
      "[0 1]\n",
      "[[ 0.50940979  0.49059024]]\n",
      "[0 1]\n",
      "[[ 0.44634357  0.5536564 ]]\n",
      "[1 0]\n",
      "[[ 0.47553322  0.52446687]]\n",
      "[0 1]\n",
      "[[ 0.45309338  0.54690665]]\n",
      "[0 1]\n",
      "[[ 0.43410259  0.56589741]]\n",
      "[0 1]\n",
      "[[ 0.49044454  0.5095554 ]]\n",
      "[0 1]\n",
      "[[ 0.4916999   0.50830013]]\n",
      "[0 1]\n",
      "[[ 0.45528474  0.54471523]]\n",
      "[0 1]\n",
      "[[ 0.4284437   0.57155633]]\n",
      "[0 1]\n",
      "[[ 0.46050695  0.53949308]]\n",
      "[0 1]\n",
      "[[ 0.45738697  0.54261303]]\n",
      "[0 1]\n",
      "[[ 0.43356106  0.56643891]]\n",
      "[0 1]\n",
      "[[ 0.47103357  0.52896643]]\n",
      "[0 1]\n",
      "[[ 0.43361151  0.56638849]]\n",
      "[0 1]\n",
      "[[ 0.4634729   0.53652704]]\n",
      "[0 1]\n",
      "[[ 0.48063138  0.51936865]]\n",
      "[0 1]\n",
      "[[ 0.45019928  0.54980069]]\n",
      "[0 1]\n",
      "[[ 0.47734156  0.52265841]]\n",
      "[0 1]\n",
      "[[ 0.47983038  0.52016962]]\n",
      "[0 1]\n",
      "[[ 0.4706988   0.52930123]]\n",
      "[0 1]\n",
      "[[ 0.47882715  0.52117282]]\n",
      "[0 1]\n",
      "[[ 0.46105793  0.53894204]]\n",
      "[0 1]\n",
      "[[ 0.476567    0.52343297]]\n",
      "[0 1]\n",
      "[[ 0.4553026   0.54469746]]\n",
      "[0 1]\n",
      "[[ 0.4658066   0.53419334]]\n",
      "[0 1]\n",
      "[[ 0.45346668  0.54653329]]\n",
      "[0 1]\n",
      "[[ 0.45562413  0.5443759 ]]\n",
      "[0 1]\n",
      "[[ 0.48276493  0.51723504]]\n",
      "[0 1]\n",
      "[[ 0.4871538   0.51284623]]\n",
      "[0 1]\n",
      "[[ 0.46575871  0.53424126]]\n",
      "[0 1]\n",
      "[[ 0.48903877  0.51096123]]\n",
      "[0 1]\n",
      "[[ 0.44676372  0.55323631]]\n",
      "[0 1]\n",
      "[[ 0.47768015  0.52231991]]\n",
      "[0 1]\n",
      "[[ 0.47969601  0.52030402]]\n",
      "[0 1]\n",
      "[[ 0.46346417  0.53653586]]\n",
      "[0 1]\n",
      "[[ 0.4455938   0.55440617]]\n",
      "[0 1]\n",
      "[[ 0.46574363  0.5342564 ]]\n",
      "[0 1]\n",
      "[[ 0.49264356  0.50735646]]\n",
      "[0 1]\n",
      "[[ 0.47878534  0.52121466]]\n",
      "[0 1]\n",
      "[[ 0.47845617  0.52154374]]\n",
      "[1 0]\n",
      "[[ 0.43117157  0.56882846]]\n",
      "[0 1]\n",
      "[[ 0.40626103  0.59373891]]\n",
      "[0 1]\n",
      "[[ 0.45563313  0.54436684]]\n",
      "[0 1]\n",
      "[[ 0.43830699  0.56169301]]\n",
      "[0 1]\n",
      "[[ 0.42565572  0.57434428]]\n",
      "[0 1]\n",
      "[[ 0.49603578  0.50396425]]\n",
      "[1 0]\n",
      "[[ 0.49172956  0.50827044]]\n",
      "[0 1]\n",
      "[[ 0.43327886  0.56672114]]\n",
      "[0 1]\n",
      "[[ 0.43456328  0.56543672]]\n",
      "[0 1]\n",
      "[[ 0.45190012  0.54809994]]\n",
      "[0 1]\n",
      "[[ 0.47590959  0.52409041]]\n",
      "[0 1]\n",
      "[[ 0.4760066   0.52399337]]\n",
      "[0 1]\n",
      "[[ 0.44831458  0.55168539]]\n",
      "[0 1]\n",
      "[[ 0.49336067  0.5066393 ]]\n",
      "[0 1]\n",
      "[[ 0.47573704  0.52426296]]\n",
      "[0 1]\n",
      "[[ 0.48829207  0.5117079 ]]\n",
      "[0 1]\n",
      "[[ 0.44006562  0.55993438]]\n",
      "[0 1]\n",
      "[[ 0.44449884  0.5555011 ]]\n",
      "[0 1]\n",
      "[[ 0.49022117  0.50977886]]\n",
      "[1 0]\n",
      "[[ 0.50428933  0.49571064]]\n",
      "[0 1]\n",
      "[[ 0.47923145  0.52076852]]\n",
      "[0 1]\n",
      "[[ 0.46545833  0.53454167]]\n",
      "[0 1]\n",
      "[[ 0.4349533   0.56504667]]\n",
      "[1 0]\n",
      "[[ 0.48983297  0.510167  ]]\n",
      "[0 1]\n",
      "[[ 0.48423746  0.51576251]]\n",
      "[1 0]\n",
      "[[ 0.48082602  0.51917398]]\n",
      "[1 0]\n",
      "[[ 0.45670697  0.54329306]]\n",
      "[0 1]\n",
      "[[ 0.44789895  0.55210096]]\n",
      "[0 1]\n",
      "[[ 0.4717195  0.5282805]]\n",
      "[0 1]\n",
      "[[ 0.49317649  0.50682348]]\n",
      "[0 1]\n",
      "[[ 0.44043255  0.55956745]]\n",
      "[0 1]\n",
      "[[ 0.45438904  0.54561096]]\n",
      "[0 1]\n",
      "[[ 0.47895947  0.5210405 ]]\n",
      "[1 0]\n",
      "[[ 0.4876292   0.51237077]]\n",
      "[0 1]\n",
      "[[ 0.47703311  0.52296692]]\n",
      "[0 1]\n",
      "[[ 0.4625124   0.53748763]]\n",
      "[1 0]\n",
      "[[ 0.47916761  0.52083242]]\n",
      "[0 1]\n",
      "[[ 0.45706591  0.54293412]]\n",
      "[1 0]\n",
      "[[ 0.46618563  0.53381437]]\n",
      "[0 1]\n",
      "[[ 0.43298072  0.56701928]]\n",
      "[0 1]\n",
      "[[ 0.43012872  0.56987119]]\n",
      "[0 1]\n",
      "[[ 0.5022046   0.49779537]]\n",
      "[1 0]\n",
      "[[ 0.42218897  0.577811  ]]\n",
      "[0 1]\n",
      "[[ 0.4976885   0.50231141]]\n",
      "[0 1]\n",
      "[[ 0.44839358  0.55160642]]\n",
      "[0 1]\n",
      "[[ 0.40660352  0.59339643]]\n",
      "[0 1]\n",
      "[[ 0.43560505  0.56439495]]\n",
      "[0 1]\n",
      "[[ 0.45240572  0.54759431]]\n",
      "[0 1]\n",
      "[[ 0.43690765  0.56309229]]\n",
      "[0 1]\n",
      "[[ 0.44443652  0.55556345]]\n",
      "[0 1]\n",
      "[[ 0.45215777  0.5478422 ]]\n",
      "[0 1]\n",
      "[[ 0.48463774  0.51536232]]\n",
      "[1 0]\n",
      "[[ 0.40785173  0.59214824]]\n",
      "[0 1]\n",
      "[[ 0.41979137  0.5802086 ]]\n",
      "[0 1]\n",
      "[[ 0.4576045   0.54239547]]\n",
      "[0 1]\n",
      "[[ 0.48367691  0.51632315]]\n",
      "[0 1]\n",
      "[[ 0.43738329  0.56261671]]\n",
      "[0 1]\n",
      "[[ 0.49393269  0.50606722]]\n",
      "[0 1]\n",
      "[[ 0.46221897  0.53778106]]\n",
      "[0 1]\n",
      "[[ 0.47879311  0.52120697]]\n",
      "[0 1]\n",
      "[[ 0.45237577  0.54762423]]\n",
      "[1 0]\n",
      "[[ 0.46208897  0.537911  ]]\n",
      "[0 1]\n",
      "[[ 0.45443374  0.54556632]]\n",
      "[0 1]\n",
      "[[ 0.43352503  0.56647497]]\n",
      "[1 0]\n",
      "[[ 0.44766071  0.55233932]]\n",
      "[0 1]\n",
      "[[ 0.4583151   0.54168499]]\n",
      "[0 1]\n",
      "[[ 0.42753395  0.57246602]]\n",
      "[0 1]\n",
      "[[ 0.48048517  0.51951474]]\n",
      "[0 1]\n",
      "[[ 0.42988431  0.57011575]]\n",
      "[0 1]\n",
      "[[ 0.45809054  0.5419094 ]]\n",
      "[0 1]\n",
      "[[ 0.46352127  0.5364787 ]]\n",
      "[0 1]\n",
      "[[ 0.43063328  0.56936675]]\n",
      "[1 0]\n",
      "[[ 0.4181155   0.58188444]]\n",
      "[0 1]\n",
      "[[ 0.47650403  0.52349597]]\n",
      "[0 1]\n",
      "[[ 0.45704436  0.54295564]]\n",
      "[0 1]\n",
      "[[ 0.43908525  0.56091481]]\n",
      "[1 0]\n",
      "[[ 0.48800316  0.51199681]]\n",
      "[0 1]\n",
      "[[ 0.47949278  0.52050722]]\n",
      "[0 1]\n",
      "[[ 0.43957883  0.56042117]]\n",
      "[0 1]\n",
      "[[ 0.43067262  0.56932735]]\n",
      "[0 1]\n",
      "[[ 0.47971416  0.5202859 ]]\n",
      "[0 1]\n",
      "[[ 0.48602509  0.51397491]]\n",
      "[0 1]\n",
      "[[ 0.44002876  0.55997127]]\n",
      "[0 1]\n",
      "[[ 0.49106675  0.50893325]]\n",
      "[0 1]\n",
      "[[ 0.50190216  0.49809787]]\n",
      "[0 1]\n",
      "[[ 0.47208112  0.52791882]]\n",
      "[0 1]\n",
      "[[ 0.48987615  0.51012385]]\n",
      "[0 1]\n",
      "[[ 0.47883725  0.52116269]]\n",
      "[0 1]\n",
      "[[ 0.47887233  0.52112764]]\n",
      "[0 1]\n",
      "[[ 0.49893323  0.50106674]]\n",
      "[0 1]\n",
      "[[ 0.41958714  0.58041286]]\n",
      "[0 1]\n",
      "[[ 0.48861679  0.51138318]]\n",
      "[0 1]\n",
      "[[ 0.42022088  0.57977915]]\n",
      "[0 1]\n",
      "[[ 0.45105758  0.54894239]]\n",
      "[1 0]\n",
      "[[ 0.47110903  0.52889097]]\n",
      "[0 1]\n",
      "[[ 0.48217317  0.5178268 ]]\n",
      "[0 1]\n",
      "[[ 0.48546985  0.51453012]]\n",
      "[0 1]\n",
      "[[ 0.44139668  0.55860329]]\n",
      "[0 1]\n",
      "[[ 0.45719749  0.54280251]]\n",
      "[0 1]\n",
      "[[ 0.42564243  0.57435757]]\n",
      "[0 1]\n",
      "[[ 0.4403224  0.5596776]]\n",
      "[0 1]\n",
      "[[ 0.41870371  0.58129632]]\n",
      "[0 1]\n",
      "[[ 0.44445041  0.55554962]]\n",
      "[0 1]\n",
      "[[ 0.4235982   0.57640189]]\n",
      "[0 1]\n",
      "[[ 0.42205888  0.57794118]]\n",
      "[0 1]\n",
      "[[ 0.43991387  0.56008613]]\n",
      "[0 1]\n",
      "[[ 0.42535886  0.57464117]]\n",
      "[0 1]\n",
      "[[ 0.44351217  0.55648786]]\n",
      "[0 1]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.45898774  0.54101223]]\n",
      "[0 1]\n",
      "[[ 0.43498111  0.56501895]]\n",
      "[0 1]\n",
      "[[ 0.46783352  0.53216654]]\n",
      "[0 1]\n",
      "[[ 0.46935496  0.53064501]]\n",
      "[0 1]\n",
      "[[ 0.47076625  0.52923381]]\n",
      "[0 1]\n",
      "[[ 0.43756971  0.56243032]]\n",
      "[0 1]\n",
      "[[ 0.45619521  0.54380476]]\n",
      "[0 1]\n",
      "[[ 0.49004981  0.50995016]]\n",
      "[0 1]\n",
      "[[ 0.47301233  0.52698767]]\n",
      "[1 0]\n",
      "[[ 0.4550077   0.54499227]]\n",
      "[0 1]\n",
      "[[ 0.47195199  0.52804804]]\n",
      "[0 1]\n",
      "[[ 0.4187642   0.58123571]]\n",
      "[0 1]\n",
      "[[ 0.41130325  0.58869672]]\n",
      "[0 1]\n",
      "[[ 0.50030977  0.49969023]]\n",
      "[0 1]\n",
      "[[ 0.4167954   0.58320463]]\n",
      "[0 1]\n",
      "[[ 0.48965871  0.51034135]]\n",
      "[0 1]\n",
      "[[ 0.43381453  0.56618553]]\n",
      "[1 0]\n",
      "[[ 0.45770463  0.54229534]]\n",
      "[0 1]\n",
      "[[ 0.47058684  0.5294131 ]]\n",
      "[0 1]\n",
      "[[ 0.43881345  0.56118649]]\n",
      "[0 1]\n",
      "[[ 0.44644329  0.55355674]]\n",
      "[0 1]\n",
      "[[ 0.43626603  0.56373399]]\n",
      "[0 1]\n",
      "[[ 0.45604759  0.54395241]]\n",
      "[0 1]\n",
      "[[ 0.46246618  0.53753376]]\n",
      "[1 0]\n",
      "[[ 0.43489584  0.56510413]]\n",
      "[0 1]\n",
      "[[ 0.46011424  0.53988576]]\n",
      "[0 1]\n",
      "[[ 0.46550131  0.53449869]]\n",
      "[0 1]\n",
      "[[ 0.43043163  0.56956834]]\n",
      "[0 1]\n",
      "[[ 0.45426863  0.54573137]]\n",
      "[1 0]\n",
      "[[ 0.47644377  0.52355617]]\n",
      "[1 0]\n",
      "[[ 0.47247091  0.52752912]]\n",
      "[0 1]\n",
      "[[ 0.45742032  0.54257971]]\n",
      "[0 1]\n",
      "[[ 0.44664049  0.55335957]]\n",
      "[0 1]\n",
      "[[ 0.49166834  0.50833166]]\n",
      "[0 1]\n",
      "[[ 0.51774234  0.48225772]]\n",
      "[0 1]\n",
      "[[ 0.43431547  0.56568456]]\n",
      "[0 1]\n"
     ]
    }
   ],
   "source": [
    "def get_predictions(model, X_data, y_data, display=False):\n",
    "    preds = np.zeros((0,2))\n",
    "    labels = np.zeros((0,2))\n",
    "    for ii in range(len(X_data)):\n",
    "        prediction = model.predict(np.expand_dims(X_data[ii], axis=0))\n",
    "        label = y_data[ii]\n",
    "        preds = np.vstack((preds,prediction))\n",
    "        labels = np.vstack((labels,label))\n",
    "        if display is True:\n",
    "            print(prediction)\n",
    "            print(label)\n",
    "    return preds, labels\n",
    "\n",
    "preds, labels = get_predictions(model_bidirectional, X_data, y_data, display=True)\n",
    "#preds, labels = get_predictions(model_no_embedding, X_data_enc, y_data, display=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import itertools\n",
    "import math\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "def print_statistics(cm):\n",
    "    tn, fp, fn, tp = cm.ravel()\n",
    "    # TP\n",
    "    print(\"TP: \" + str(tp))\n",
    "    # TN\n",
    "    print(\"TN: \" + str(tn))\n",
    "    # FP\n",
    "    print(\"FP: \" + str(fp))\n",
    "    # FN\n",
    "    print(\"FN: \" + str(fn))\n",
    "    # TPR\n",
    "    recall = tp/(tp+fn)\n",
    "    print(\"TPR/recall: \" + str(recall))\n",
    "    # TNR\n",
    "    specificity = tn/(tn+fp)\n",
    "    print(\"TNR/specificity: \" + str(specificity))\n",
    "    # PPV\n",
    "    precision = tp/(tp+fp)\n",
    "    print(\"PPV/precision: \" + str(precision))\n",
    "    # NPV\n",
    "    npv = tn/(tn+fn)\n",
    "    print(\"NPV/negative predictive value: \" + str(npv))\n",
    "    # FNR\n",
    "    miss_rate = 1-recall\n",
    "    print(\"FNR/false negative rate: \" + str(miss_rate))\n",
    "    # FPR\n",
    "    fall_out = 1-specificity\n",
    "    print(\"FPR/false positive rate: \" + str(fall_out))\n",
    "    # FDR\n",
    "    fdr = 1-precision\n",
    "    print(\"FDR/false discovery rate: \" + str(fdr))\n",
    "    # FOR\n",
    "    fomr = 1-npv\n",
    "    print(\"FOR/false ommission rate: \" + str(fomr))\n",
    "    # F1\n",
    "    f1 = 2*((precision*recall)/(precision+recall))\n",
    "    print(\"F1 score: \" + str(f1))\n",
    "    # accuracy\n",
    "    acc = (tp+tn)/(tp+tn+fp+fn)\n",
    "    print(\"Accuracy: \" + str(acc))\n",
    "    # Matthews correlation coefficient (MCC)\n",
    "    mcc = (tp*tn-fp*fn)/math.sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn))\n",
    "    print(\"MCC/Matthews correlation coefficient: \" + str(mcc))\n",
    "    # Informedness or Bookmaker Informedness (BM)\n",
    "    bm = recall+specificity-1\n",
    "    print(\"BM/Bookmaker Informedness: \" + str(bm))\n",
    "    # Markedness (MK)\n",
    "    mk = precision+npv-1\n",
    "    print(\"MK/Markedness: \" + str(mk))\n",
    "    \n",
    "    return fall_out, recall\n",
    "\n",
    "# credit: https://github.com/scikit-learn/scikit-learn/blob/master/examples/model_selection/plot_confusion_matrix.py\n",
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    #plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "    s = [['TN','FP'], ['FN', 'TP']]\n",
    "    #s = [['TP','FN'], ['FP', 'TN']]\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i in range(2):\n",
    "        for j in range(2):\n",
    "            plt.text(j,i, (str(s[i][j])+\" = \"+str(format(cm[i][j],fmt))),horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TP: 50\n",
      "TN: 54\n",
      "FP: 4\n",
      "FN: 417\n",
      "TPR/recall: 0.107066381156\n",
      "TNR/specificity: 0.931034482759\n",
      "PPV/precision: 0.925925925926\n",
      "NPV/negative predictive value: 0.114649681529\n",
      "FNR/false negative rate: 0.892933618844\n",
      "FPR/false positive rate: 0.0689655172414\n",
      "FDR/false discovery rate: 0.0740740740741\n",
      "FOR/false ommission rate: 0.885350318471\n",
      "F1 score: 0.191938579655\n",
      "Accuracy: 0.198095238095\n",
      "MCC/Matthews correlation coefficient: 0.0393187702999\n",
      "BM/Bookmaker Informedness: 0.0381008639149\n",
      "MK/Markedness: 0.0405756074546\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAASAAAAEmCAYAAADC2k2dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd8leX9xvHPxd6ggCIOsFr3QBHB\nVa0THHUj7oF71llr3da6R7VaVytat9YtCo4fDhQHuK0LBcXNcCCojO/vj+cJHkISEpKTm+Rc79cr\nL3KedX/PSXJxP/ezFBGYmaXQJHUBZla6HEBmlowDyMyScQCZWTIOIDNLxgFkZsk4gMwsGQdQkUga\nJ2m6pKkFX90l9ZQUBdPGSTq5gvX3k/SmpGmSvpT0T0mdCuafmW/n6HLr/TGffmYldbWQdImkCXn7\nH0u6rNwye0h6JZ//haRHJW1Y0O6MfN63kp6XtF4F7SwrabakqwumLVPu8whJPxa83ihfbl1JQ/Pt\nT5b0kqT983mbSJpQQXsjJB1YyXveT9Kscm1PldS94Gf1i6Qu5dZ7La+xZ/56SL7cVEk/SBotaeMa\ntjM9X7fssztUUpOCbZS18UP+9Zak8yR1LNfOcwWvx0n6SlLbgmkHShpR8FqSjpT0RsHv1AhJgwqW\nWVXScElT8vpGS9q6os+0rjiAimu7iGhX8PV5wbxOEdEO2AU4TdIWZTMkHQ9cAJwIdAT6AT2AxyW1\nKNjG+8C+5drcJ59emT8D6wDrAu2B3wOvFrR9HHA58DdgcWAZ4Gpg+4Jt3JnX3gX4P+DuCtrZB5gC\nDJLUEiAiPin8PPLl1iyY9mweZk8BTwPLA52Bw4ABVbyn6nih3M+i/M/jY2D3gs9hdaB1Bdu5MK+9\nI/BP4F5JTWvQznYR0Z7s53k+8CfgXxW00R7oCuxP9vMfWRgwFWgGHFPF/CuAPwLHk32mSwKnAv0L\nlnkIeJzs574YcDTwfRXbrDUHUGIR8QrwNtALQFIH4CzgqIh4LCJmRMQ4YCDZL+1eBau/DLSRtGq+\n7qpkfzQvV9FkH+C+iPg8MuMi4uZ8/Y7A2cAREXFvRPyYt/9QRJxYQe0zgVuBJSV1LTd7H7Jf8BnA\ndjX4SC4CboqICyJiYl7j6IgYWINtLIj/kNVcZl/g5soWjojZwG3AomR/sDUSEd9FxIPAbsC+klar\nYJmfIuJl4A9kobF/FZu8CDihsJdcRtIKwOHAoIh4PCKmR8SsiHguIvbLl+kCLAtcHxG/5F8jI+K5\n8turSw6gxCT1A1YDPswnrQ+0Au4tXC4ipgKPAlswt8I/nCr/aHKjgOMkHS5pdUkqmLde3vZ91ay9\nRd72JLLeTtn0jYClgDuAu5j7D7uq7bXJa7inOsvXsVFAB0kr5z2a3YBbKls4X2Yfsp7TVwvaaES8\nBEwANqpimR/IeiaVLgO8AowATqhg3qbAp/l/dpWZRPY7eIukHSTVOFQXhAOouO7P96W/lXR/uXkT\nJU0HXiDbxSmb3wWYmPcuyvsin1/oFmB3Sc2BQVTxR5M7j2z3bk+yX9rPJJXtxnWuou1CAyV9C0wH\nDgJ2KbfOvsCjETGFrJcwQNJi89kmwCJkv5NfzGe57gWf67d5LRvOZ51+5dYZW8EyZWG+BfAu8FkF\ny5yQt/cj2a7qaRExq4btlPc5WU+qtsucDhxVQW+0C/Bl4QRlY4DfSvpJUo/ILgr9PTAOuAT4QtIz\nkn5bjfoXmAOouHaIiE751w7l5nUB2pH9j7UJ0DyfPhHoIqlZBdtbIp8/R0R8QvY/19+ADyLi06oK\nyrveV0XEBkAn4Fzg35JWJvtfsLK2C90VEZ3Idj3eAnqXzZDUGtiVbNeMiHgB+ATYYz7bhKwXNTt/\nn1X5vOBz7ZTXMr9dhVHl1lmugmX+k9e5H5X3JC/O22tNNpZ2kaTC8anqtFPeksDk2i4TEW8BDwPl\nD2pMotxnGhFLkf0OtgSUT5sQEUfmNfcgC9n59ahrxQGUUB4GlwA/ke2jQ9Yj+hnYqXDZfAByAPBk\nBZu6mWxwsUa/LPlYwFVkf/ir5G3/BJQPy8rWnwgcApwpqewXfEegA3B1fqTlS7I/nvnuhkXEtLyG\nnWvyPupKRIwn26XamnK7wBUsG/kf/EhgmwVtU1Ifss+n0gCV1A7YHHi2Gps8g6xXumTBtKeApSSt\nU9268v/IriIbHigaB9DC4XzgJEmtIuI7skHoKyX1l9Q8Pwx8N9lYwX8qWP9OYEuy8ZYqKTtMv4mk\n1pKa5btf7YFX87ZPB67KxwHa5O0PkHRhRduLiHeBYcBJ+aR9gX8Dq5MNrPcCNgB65UeW5uckYD9J\nJ0rqnNe8pqQ7qrFuXRgMbBoRP85vQUkrke36vV3TRiR1kLQt2TjZLRHxZgXLtJTUm2z3fApw4/y2\nGxEfkv0+HF0w7T3gWuAOSVvkP/umZOONZW0tIuksSctLapIPSh9ANjZWNA6ghcMjZL9gBwFExIXA\nKcDFZIdBXwQ+BTaLiJ/Lr5z3ZJ6IiOnVaGs62T7+l2S7c0cAO0fER/m2LgWOIzuC9U3e7pH8OkZV\nkYuAgyX1ADYDLo+ILwu+RgOPMe8pA/OIiOfJBk03BT6SNBm4DhhajfdWlfU07/k5fSpof+x8BmtP\nytf9ERhOFgrX1qCdhyT9QPa5/gW4lHmPbp2ULzOZrFc7Gli/OqGYOxsof8j+CLJD8Zfm250AnEM2\n2P4J8AvQE3iC7HfuLbKe+H7VbHOBKHxDMjNLxD0gM0vGAWRmyTiAzCwZB5CZJTO/E84MWLRzl1hq\n6R6py7BymjfV/BeyJMaMGT0xIsqfkT0PB1A1LLV0Dx58YmTqMqycbp1apS7BKtG6ucZXZznvgplZ\nMg4gM0vGAWRmyTiAzCwZB5CZJeMAMrNkHEBmlowDyMyScQCZWTIOIDNLxgFkZsk4gMwsGQeQmSXj\nADKzZBxAZpaMA8jMknEAmVkyDiAzS8YBZGbJOIDMLBkHkJkl4wAys2QcQGaWjAPIzJJxAJlZMg4g\nM0vGAWRmyTiAzCwZB5CZJeMAMrNkHEBmlowDyMyScQCZWTIOIDNLxgFkZsk4gMwsGQeQmSXjADKz\nZBxAZpaMA8jMknEAmVkyDiAzS8YBtJCbMnkSW2/Sl6036UufVXrSb/XfzHm9bNfW/PX0P81Z9rqr\nLuPyC/9a6zYHbb8lm/ZbY047E7/5eq75Qx+8l2W7tuaN10bXuq3GpG3LpvTt3WvO1/hx43jm6REs\n3rkj/dZZi16rr8y555xVZ+1ddunFtG4uJk6cWGfbrG/NUhdgVVtk0c4MHfEiAJdf+FfatG3LwUcc\nC8CKS3Vi2CMPcPgxJ7Jo5y512u7l19zIGr16zzN96tQfGHL91fTq3adO22sMWrduzYujX5tr2vjx\n49hgw42494GH+fHHH+m7Ti8GbL0ta/ee97OtiU8//ZSnnnicpZdZplbbSc09oAasWdNm7L73YP51\nzZX11ual553FIUcdR8uWreqtzcaibdu2rLV2bz76aGytt3XSCcdy7nkXIqkOKkvHPaAGbu/BhzBg\n4z4cctRxlS7zwnNPc86pJ80zvXWb1vx36IgK1znp6ENo0qQp/bfbgaOOOxlJvP3Ga3zx2QQ223Jr\nrr/q8rp6C43G9OnT6du7FwA9ll2Wu+65b675kyZN4qUXR/HnU06ba/oPP/zA5ptsVOE2h/znNlZe\nZZW5pj380IN0774ka6y5Zh1Wn0bRAkhSAJdGxPH56xOAdhFxZh23c0pE/K3g9fMRsX5dtrEwa9++\nAzsN3JMh119Nq1YV90rW23DjObtx1XH5NTfSbYklmTr1Bw7bf3fuves2dtx1d8457SQuvvL6uiq9\n0aloFwxg5HPP0m+dtWjSpAknnHgyq6y66lzz27dvX+F6FZk2bRoXnHcuDz86vE5qTq2YPaCfgZ0k\nnRcRxRwlOwWYE0ClFD5lDjjkSLbdbD123X2fCufXtAfUbYklAWjXrj3b77Qbr495mS0GbMv7777D\noB22BOCbr7/ioL124fpb7qlwrMh+VTYGVJma9IA+GjuW8eM+Zt3eWe/nswkTWG/dtXn2+Zfo1q1b\n3RZeD4oZQDOB64Bjgb8UzpDUFbgGKBtB+2NEjMyn3wZ0Bl4G+gO9I2KipPuBpYFWwN8j4jpJ5wOt\nJb0GvB0Re0qaGhHtJN0J3BQRQ/M2hwAPAfcD5wObAC2BqyLi2qJ9CvWg0yKLss32O3PXrUPYdY99\n55lfkx7QzJkz+f67b1m0cxdmzJjBk8OHsuHGm9KhQ0fGvDdhznKDtt+SU846z+FTB2rSA1pt9dX5\n5PNfj0quuHxPRo56hS5d6vYgRH0p9iD0VcCekjqWm/534LKI6APsDNyQTz8DeCoi1gbu49eAAjgg\nInoD6wBHS+ocEScD0yOiV0TsWa6NO4DdACS1ADYDhgKDge/ytvsAB0lato7ebzIHHnYMkydPqvV2\nfvn5Z/Yd+Af6b9yHbX7fl25LdGfQ3gfUQYVm81JEFGfDv/ZEzgZmANPJx4AkfQ18XrB4V2Al4Flg\nx4j4ON/GZGCFvAd0JrBjvnxPYKuIGFXWTgXttgI+AJYn60kNzHtI9wBrANPyVToCh0TEXDvVkg4G\nDgbovtTSvUe++n4dfTJWV7p18pG4hVXr5hodEevMb7n6OAp2OTAGuLFgWhNgvYiYXrigKjmmKGkT\nYPN8nWmSRpDtilUqIn7Kl9uKrCd0e9nmgKMiYth81r+ObBeSNXr1Lk5Km5W4op8HFBGTgbvIdn3K\nDAeOLHshqVf+7XPAwHzalsAi+fSOwJQ8fFYC+hVsa4ak5pU0fwewP7ARUBY4w4DDytaRtIKktgv4\n9sysFurrRMRLgMJRsqOBdSS9Iekd4NB8+lnAlpLGAAOAL4AfgMeAZpLeAM4BRhVs6zrgDUm3VtDu\ncOB3wBMR8Us+7QbgHWCMpLeAa/H5UGZJFG0MaEFIagnMioiZktYD/hkRvea3XrGt0at3PPjEyNRl\nWDkeA1p4LUxjQDWxDHCXpCbAL8BBiesxsyJaqAIoIj4A1kpdh5nVD1+MambJOIDMLBkHkJkl4wAy\ns2QcQGaWjAPIzJJxAJlZMg4gM0vGAWRmyTiAzCwZB5CZJeMAMrNkHEBmlowDyMyScQCZWTIOIDNL\nxgFkZsk4gMwsGQeQmSXjADKzZCq9Kb2kDlWtGBHf1305ZlZKqnoqxttAkD3KuEzZ6yB7hI6Z2QKr\nNIAiYun6LMTMSk+1xoAkDZJ0Sv79UpJ6F7csMysF8w0gSf8Afg/snU+aBlxTzKLMrDRU58mo60fE\n2pJeBYiIyZJaFLkuMysB1dkFm5E/qz0AJHUGZhe1KjMrCdUJoKuA/wJdJZ0FPAdcUNSqzKwkzHcX\nLCJuljQa2DyftGtEvFXcssysFFRnDAigKTCDbDfMZ0+bWZ2ozlGwvwC3A92BpYDbJP252IWZWeNX\nnR7QXkDviJgGIOlcYDRwXjELM7PGrzq7U+OZO6iaAR8VpxwzKyVVXYx6GdmYzzTgbUnD8tdbkh0J\nMzOrlap2wcqOdL0NPFIwfVTxyjGzUlLVxaj/qs9CzKz0zHcQWtJywLnAKkCrsukRsUIR6zKzElCd\nQeghwI1k9wEaANwF3FHEmsysRFQngNpExDCAiBgbEaeSXR1vZlYr1TkP6GdJAsZKOhT4DFisuGWZ\nWSmoTgAdC7QDjiYbC+oIHFDMosysNFTnYtQX829/4NebkpmZ1VpVJyLeR34PoIpExE5FqcjMSkZV\nPaB/1FsVC7m33v+Ulbc4IXUZVs57T1ycugSrpapORHyyPgsxs9Lje/uYWTIOIDNLptoBJKllMQsx\ns9JTnTsirivpTeCD/PWakq4semVm1uhVpwd0BbAtMAkgIl7Hl2KYWR2oTgA1iYjx5abNKkYxZlZa\nqnMpxqeS1gVCUlPgKOD94pZlZqWgOj2gw4DjgGWAr4B++TQzs1qpzrVgXwOD6qEWMysx1bkj4vVU\ncE1YRBxclIrMrGRUZwzoiYLvWwE7Ap8WpxwzKyXV2QW7s/C1pP8AjxetIjMrGQtyKcayQI+6LsTM\nSk91xoCm8OsYUBNgMnByMYsys9JQZQDl94Jek+w+0ACzI6LSm5SZmdVElbtgedjcFxGz8i+Hj5nV\nmeqMAb0kae2iV2JmJaeqe0I3i4iZwIbAQZLGAj+SPaAwIsKhZGa1UtUY0EvA2sAO9VSLmZWYqgJI\nkD0NtZ5qMbMSU1UAdZV0XGUzI+LSItRjZiWkqgBqSvZEVNVTLWZWYqoKoC8i4ux6q8TMSk5Vh+Hd\n8zGzoqoqgDartyrMrCRVGkARMbk+CzGz0uMHE5pZMg4gM0vGAWRmyTiAzCwZB5CZJeMAMrNkHEBm\nlowDyMyScQCZWTIOIDNLxgFkZsk4gMwsGQeQmSXjADKzZBxAZpaMA8jMknEAmVkyVd2U3hKa+soV\nvPXh53NeDzz2Onp078zwG45h52OuYegzbwHw378fyuU3P8mzoz+odZvt27bitXtP5cGnXufYC+4G\n4MwjtmPPbdelU4c2dN3g+DnLXnj8TvyuzwoAtGnVgq6LtmOJ351U6xoakimTJ7HHTlsD8M3XX9Gk\nSRM6d+kKwDtvvcEqq63BzJkzWX6FFbn0HzfQuk2bBW7rheee4aC9d2XpHj0B6L/N9hxz4ikAjHhy\nOGedcgKzZs9i0F77cfgxJ9bujdUjB9BCavrPM+g36Py5pvXo3pkJX07hT4O3mhNAdemMw7fh2dEf\nzjVt6DNvcs2dT/PmA2fMNf2kS+6d8/1hgzZmzRWXqvN6FnaLLNqZR0e8CMBlF/yVNm3bcsiRxwKw\nco8uc+Ydfch+3DLkeg46/Jhatden3wbcePu9c02bNWsWp/3pj9x6zyN0674kf9hiQzbvvy0rrLhy\nrdqqL94Fa2DeeP8zvpv6E5v2XalOt7vWykuzWOcOPPHC/+aa/tKb4/hy4vdVrjuwf2/uemx0ndbT\nmKzbbwPGffxRUbb92piX6bnscizTc1latGjBdjvuyuOPPlyUtorBPaCFVOuWzRl1x8kAjP9sErsd\nf/2ceRfc8BhnHLEtT734bqXrH7vPZuy2dZ95po8c8yHHX3jPXNMkcf5xOzH41JvYZN0Va1TnMkss\nQo/unRnx8ns1Wq9UzJw5kxFPDmfjTbeYZ94Rg/fio7Hz7jofeNjR7LzbnvNMH/PKi/TfeF0W67YE\np551HiustApffvE5S3T/tfe5RPcleXX0S3X7Joqo3gNI0izgzbzt/wH7RsS0Gm7jBuDSiHhH0ikR\n8beCec9HxPp1WnQCFe2ClRn56lgANlhruUrXv+zmJ7ns5ier1dYhAzdi2HNvM+Grb2tc565b9eb+\nJ19j9uyo8bqN2U/TpzNgk74A9Om3Prvttd88y1z1r1uqvb3V1uzF86++R9t27Xjq8cc4aO+BPP3y\nWxDzfu5Sw3mkX4oe0PSI6AUg6VbgUKBGz5mPiAMLXp4C/K1gXoMPn+q44IZhnDR4K2bOml3h/Jr0\ngPqusSwbrLUcBw/ciLatW9KieVOmTv+Z0654cL517LJVb449/64FexONWKvWreeMAVWmJj2g9u07\nzPl+0y36c9pJxzB50kS6dV+SLz6fMGfeF59/xuLdutey+vqTehfsWWANAEnHAQfk02+IiMsltQXu\nApYie1b9ORFxp6QRwAnALkBrSa8Bb0fEnpKmRkQ7SXcCN0XE0Hz7Q4CHgPuB84FNgJbAVRFxbb28\n2zr05Kh3OePwbVmia8cK59ekB7T/X26a8/1e2/Wl9yrLVCt8fttjMRbp0IZRr39cvaJtLjXpAX39\n1Zd0XWxxJPHamJeZPXs2iyzamQ4dO/HxRx/yyfhxdFuiOw/ddzdXXDukeEXXsWQBJKkZMAB4TFJv\nYH+gL9kjoV+U9DTwG+DziNgmX2euv7aIOFnSkWU9qnLuAHYDhkpqQfak18OAwcB3EdFHUktgpKTh\nETHXX5Gkg4GDAWjerq7edp264F/DuOfyQ4raxrnHbM9uA9ahTavmfPjYOdx43wuce+1QAAb2X4e7\nh3nwuT4Mfeg+brnxepo1a0arVq248vqbkUSzZs04+/zL2GfX7Zg1exYD99iXFVZaJXW51aaoYB+y\nqA3+OgYEWQ/oeLJg6BwRp+fLnAN8AzwGDCPrBT0cEc/m80cAJ0TEK2U9noLtl/WAWgEfAMsD/YGB\neQ/pHrJeV9m4U0fgkIgYXlnNTdosFi1XHFg3H4DVmfeeuDh1CVaJHl1aj46Idea3XNIxoDKqZNQs\nIt7Pe0dbA+flPZWzq9NIRPyUB9VWZD2h28uaA46KiGEL+gbMrG4sLOcBPQPsIKlNPu6zI/CspO7A\ntIi4BbgYWLuCdWdIal7Jdu8g27XbiKwnRf7vYWXrSFohb9PM6lnqQWgAImJMPkhcdgLDDRHxqqSt\ngIskzQZmkO2qlXcd8IakMRFR/uSJ4cDNwIMR8UvZtoGewJi85/UNsEOdviEzq5Z6HwNqiDwGtHDy\nGNDCq7pjQAvLLpiZlSAHkJkl4wAys2QcQGaWjAPIzJJxAJlZMg4gM0vGAWRmyTiAzCwZB5CZJeMA\nMrNkHEBmlowDyMyScQCZWTIOIDNLxgFkZsk4gMwsGQeQmSXjADKzZBxAZpaMA8jMknEAmVkyDiAz\nS8YBZGbJOIDMLBkHkJkl4wAys2QcQGaWjAPIzJJxAJlZMg4gM0vGAWRmyTiAzCwZB5CZJeMAMrNk\nHEBmlowDyMyScQCZWTIOIDNLxgFkZsk4gMwsGQeQmSXjADKzZBxAZpaMA8jMklFEpK5hoSfpG2B8\n6jrqSBdgYuoirEKN6WfTIyK6zm8hB1CJkfRKRKyTug6bVyn+bLwLZmbJOIDMLBkHUOm5LnUBVqmS\n+9l4DMjMknEPyMyScQCZWTIOIDNLxgFkZsk4gMwaAElKXUMxNEtdgC2cJCkiQtISZEdLP09dU6kq\n+1nk328OdABeBL6MiFlJi6sl94CsQnn47ADcDvxT0gWSlkpdVykqCJ9jgLOAvsBTwLop66oLDiCr\nkKTVgeOAbYGXgN8D3yUtqoRJWgHYOCI2AMYBn5D1gsrmN8hdNAeQVWYW8DCwK7ANMCgifpC0atqy\nSo+kzsDnwBuShgA7AAMiYrakfSV1jAZ6RrEDyOYiaRVJuwC/ABsBhwP7RMRHkgYA10vqlrTIEiKp\nH/BnYCbQDVgeGBwRMyXtBRwPtE9YYq34Ugybi6SDgP0iYgNJfyQbZ3gKmAb8BfhTRDycssbGKt+N\nUkTMLpi2LPAkcCDZbteFwBSgKbAWsGdEvJWg3DrhACpxBUe7mpYdUZF0G/BCRFwp6UCgB7Ao8EBE\nDC88KmN1p9zRrs7AzxExVdLOwO8j4khJvyXrCS0OvBwRDfpGeT4MX6LyQc01I+JuSesAG0saGxH3\nA/8GtgSIiBvy5ZtHxIx8msOnDuU9n9WB04BdJfUGTgbGSfo3MArYXtJvI+ID4IN01dYtjwGVribA\n15LaA58CLYEjJP0DmAFsLWnvguVnJqixJETmDeBISZsAr5GF0dfAfWRjccsBl0hqkazQInAAlaiI\neBcYSRY+O0TE34A/kI0t9AM6AftKapcv715PEUhqXfByIrA/8BbwcURcBBxDtvv7M7Ay0Kbeiywi\njwGVEEltgC0i4gFJfcmOdAl4DDg3Iv4uqQnZGMOuwIcR8Ui6ihs3Sa3IjmINJTu6tXpEnJ7vdq0H\n9IqInyU1A9oCnSPio3QV1z0HUInJzyNZB/gJOCgiXpW0NvAEcGpEXF1ueQ84F4GkLhExUdJGwNPA\nh2QB9HM+/0ayo1z9IuKnhKUWlXfBSkTBmbLnkXXpZ0bEqwARMQbYHPh7frr/HA6fuqXM0sBf893b\nd4AHgCXI/mMAICL2B94GnklSaD1xD6gEFBxqbwK0AxYhO9I1IyL6Fyz3W6BnRDyeqNSSIakDsBrQ\nNiIel7QpcD+wR0Q8LKlfRIyStFhEfJ222uJxD6iRKwifLYFTyS6pGB8RmwEtJD0kqa+kp4FJ+R9D\ng7yuaGFX+LlGxPfAmsDpkvpHxFPAXsDdki4B/i1pqcYcPuDzgBq9PHz6A5cARwK3S1oTOC0iNpV0\nO9kV1pdExOSyddJV3DiVO8lwD+C7iPinpBnAifn8ByVtAWxMdmRyQsqa64N3wRqxfJerPXAT2Xkl\niwMXAZ8B3wJHRcQUSZ0i4lsPOBefpCPILqsYmJ9UWBZIBwBX5CFUMj8H94AaoYJf4FYR8Z2kwWQD\nz2eTDXS2Ab4AJkg6KyK+Bfd8iinf/Voe2Ifs7gJfStoRWBq4BWgODJb0ZET8mK7S+uUAamQKxnz6\nAldL2i8i3pS0GNl5P4uQhdEw4L8RMT1lvY1ZYU8m//eDfKztDuBdsp/Dd8CiEXGmpAdKKXzAAdTo\n5OGzBbAT2VnNwyRtlYfQS8CtQE/gyIh4OWGpjVq5MZ/1yYL/NeBOsmu5noqIsZIOBdbIVyu5G755\nDKiRyW/fMBTYPz+MezqwH1m3fyzZLtjMiHgpXZWNV/nxG0knAIOAb4BJwHPArfnN3QYDh5Hd/qTB\n3lKjNtwDanwmAa+Q3baTiDg7P79nGLBBRDyfsLZS0IzsYl7yG7dtBWwUEdPz22psBKwq6RuyM533\nL9XwAZ8H1OCVnVsiqaOyW3N+T3bka6eCxYYAE4AHyi4utbqX7/reLOnk/NSHSWQnfv4OICL+S3bX\nge0jYixwfES8mazghYB7QA1cPuazHdkN5KdIGkV2C8/blT3FYhrZjeUHA0eRXdQ4NVW9jVUeOGcD\n/wEWA3Ynu3PhbcC6kqbku72jgRWU3QDu52QFLyTcA2qACs+oVXbP4FOAvcmeXnFQRPwPGEjW62lH\ndt7J4sD6wOx5Nmi1ImlRsnG3cyLiSuA6oBXQmexOAwIuk3Qd2Y3GbooG/jyvuuJB6AZGUleypyLc\nnt+u83dk9+5pSdYL2iMiPpbUMyLG5eusD9xMdnZtyY43FJOkbcju17xeRHwv6Vbg6Yi4TtIiwLJk\nRx9HRwO/jWpd8i5Yw7MB2YPpWua31mhKdoX7JLJHtXybj0Ucmh/inQSMBzbzL37xRMQjkmYDoyUN\nA1qTnWBIREwh2x0bk7DEhZKCgeMUAAAES0lEQVR7QA1EPmYwS1JTsh7QJsA7+fVE55ANOu9Cdk7J\n6cBJvplY/VP26OThQLeI+FpSq8Z8P5/acgA1AJJWJBvHGQ48k98lbwAwgCyErpF0Jtk9ZToB/46I\nYaV0TdHCJP/ZXEz2JItGfTV7bTmAGgBJGwP/R3YG7V3Ab8guKt0CaEH21Mwh+REx/4+7EJC0PXAG\n2Ymf4f8IKuYAaiAkbUj2qOS+wM5kp/bvSHaka3ngTLKbjBEFD7azdCS1iwif8lAFD0I3EBHxnKTd\ngXuA9fNT+R8me57UwWRPUXDwLEQcPvPnHlADI2lr4EqgT9kNxAqugPeYjzUo7gE1MBExND/c+66k\nFSNiSrlbPpg1GO4BNVD5iW8/RsSI1LWYLSgHUAPn3S5ryBxAZpaML0Y1s2QcQGaWjAPIzJJxAFm1\nSZol6TVJb0m6W1KbWmxrk/xESiT9QdLJVSzbSdLhC9DGmfk9mas1vdwyQyTtUoO2ekryrU5qyAFk\nNTE9InpFxGpkj/g5tHCmMjX+nYqIByPi/CoW6QTUOIBs4ecAsgX1LLB8/j///yRdTXa/m6UlbSnp\nBUlj8p5SO8huWyrpXUnPUXDPakn7SfpH/v3iku6T9Hr+tT5wPrBc3vu6KF/uREkvS3pD0lkF2/qL\npPckPQGsOL83IemgfDuvS/pvuV7d5pKelfS+pG3z5ZtKuqig7UNq+0GWMgeQ1ZikZmS3Aim7ofqK\nwM0RsRbwI3AqsHlErE32hI7jJLUCrge2I3syRLdKNn8F2Z0E1wTWBt4mu43p2Lz3daKkLYHfAusC\nvYDekn4nqTfZI3DWIgu4PtV4O/dGRJ+8vf+R3Tu7TE+y57RvA1yTv4fBZM9175Nv/yBlj0KyBeBL\nMawmWkt6Lf/+WeBfQHdgfESMyqf3A1YBRua3rm4BvACsRHbBbNnz0G8hu4i2vE3JHl9Mft/k7/Jb\nmhbaMv96NX/djiyQ2gP3RcS0vI0Hq/GeVpP0V7LdvHZkjy8qc1d+ge8Hkj7K38OWwBoF40Md87bf\nr0ZbVo4DyGpiekT0KpyQh0zh44QFPB4Ru5dbrhdQV2e9CjgvIq4t18YfF6CNIWT3yn5d0n5kd5os\nU35bkbd9VEQUBhWSetawXcO7YFb3RgEbSFoeQFIbSSuQPQt9WUnL5cvtXsn6T5I9LbRsvKUD8ANZ\n76bMMOCAgrGlJSUtBjwD7CiptaT2ZLt789Me+EJSc2DPcvN2ldQkr/k3wHt524flyyNpBUltq9GO\nVcA9IKtTEfFN3pO4XVLLfPKpEfG+pIOBRyRNJHtE8WoVbOIY4Dpljy2eBRwWES9IGpkf5n40Hwda\nGXgh74FNBfaKiDGS7iR7Bvt4st3E+TkNeDFf/k3mDrr3gKfJHml0aET8JOkGsrGhMcoa/4bsHt22\nAHwtmJkl410wM0vGAWRmyTiAzCwZB5CZJeMAMrNkHEBmlowDyMyS+X8d1na8+VnHDAAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8bb7570eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds, labels = get_predictions(model_from_scratch, X_data, y_data)\n",
    "\n",
    "y_trues = [np.argmax(ii) for ii in labels]\n",
    "y_preds = [np.argmax(ii) for ii in preds]\n",
    "\n",
    "cm = confusion_matrix(y_trues, y_preds)\n",
    "fpr, tpr = print_statistics(cm)\n",
    "\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm, classes=['Negative','Positive'], title='FROM SCRATCH EMBEDDINGS')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TP: 1\n",
      "TN: 57\n",
      "FP: 1\n",
      "FN: 466\n",
      "TPR/recall: 0.00214132762313\n",
      "TNR/specificity: 0.98275862069\n",
      "PPV/precision: 0.5\n",
      "NPV/negative predictive value: 0.108986615679\n",
      "FNR/false negative rate: 0.997858672377\n",
      "FPR/false positive rate: 0.0172413793103\n",
      "FDR/false discovery rate: 0.5\n",
      "FOR/false ommission rate: 0.891013384321\n",
      "F1 score: 0.00426439232409\n",
      "Accuracy: 0.110476190476\n",
      "MCC/Matthews correlation coefficient: -0.0768395881928\n",
      "BM/Bookmaker Informedness: -0.0151000516872\n",
      "MK/Markedness: -0.391013384321\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAASAAAAEmCAYAAADC2k2dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XeYVdW9xvHvS5GqIijR2GOLJYoO\nCKhEE3tHjaixS6yxxBL1WmKNscRuElsUjRV7iTfYrjWgBlTsBZXELkUEQQL4u3+sPXg4TmVmWAzn\n/TzPPMzZ9bfPDO+svfY+aysiMDPLoU3uAsyscjmAzCwbB5CZZeMAMrNsHEBmlo0DyMyycQCZWTYO\noGYk6QlJEyV1KJs+RNJ/JU2RNEHSI5J+XMP6m0gKSceXTV+hmD6l+PpM0p8ltS9Z5gNJm5W8XkrS\nXyV9ImmypDclnSGpSw37Ld9+9dduJfWHpB3K1rukmL5f8Xo/SbNK1n9P0qGN3M9/i3onS3pV0h8k\nLVqyjfJ9vC/pekmr1rCfdmX1r1+yzMqS5rgJTtLmkv6v2Pd4SS9JOkFSx2J+N0nXSfq0WOZtSSeU\nv58NVfzMppW9Fz+s4X36QNKJNay/n6RXJE0tavqLpG4l88vf5ymSZkh6r5b3cnYNJfV9Vvo7I+lX\nkp6Y22Mu5wBqJpJWAAYAAexQwyLnR0RXYGngI+CvNSyzLzCh+Lcm3Ypt/AToD/y6llq6A8OBTkD/\niFgY2BzoBqxUx2F0i4iuJV+3l8x7u7Su4j/3rsCYsm0Mr14f+AVwvqR1G7Gf84t6lwD2B/oBz5YF\n5/Bi+4sCmwHTgJGS1qrj2CYAZ9c2U9KuwJ3ALcDyEdED2A1YBli2WOxioCuwerHvHWo4/sbavuy9\n+LhkXreS9/FUSZuX1HsscB7w26KWfsDywCOSFgIo225XYNUa3ofh5cuV1dAOOKqJx1grB1Dz2QcY\nAQyh9gAhIqYBQ4FepdMldSb9ov0aWEVS7zq28TnwCLBGLYscA0wG9oqID4p1/hMRR0XE6AYeT7kH\ngA0lLVa83goYDXxaR52jgDdI/2EbJSK+iYgXSP/Je5DCqHyZWRExJiIOA54ETq9jkzcAa0vauHyG\nJAEXAWdGxDURMaHY/lsRcUREvFMs2ge4JSImRsS3EfFmRNzZ2GNrrIj4F/Aaxe+MpEWAM4AjIuIf\nETGj+DkPIoXQXjUcYzvS790DEXFdI3Z/AXBcacuqOTmAms8+wM3F15aSflDTQsVf8j2Ad8tm7QJM\nAe4AhhXbq1HRRN6SFHg12Qy4OyK+bcwB1OMb4H5g9+L1PsCNda0gqQ/pr+6/5nanETGZFLYD6ln0\n7nqWmQqcA/y+hnmrkVo6d9WzjxHA7yXtL2mVepZtNpL6AWvx3e/MBkBH0jHPFhFTgP8ltXbLnQ90\nAQ5v5O7/BTwBHNfI9RrEAdQMJG1E+sszNCJGkprlvyxb7DhJX5JaJhsBe5fN3xe4PSJmkU4D9lBJ\nH09hXLGNj4CvSacMNekBfDIXhzJO0pclX+UtlxuBfYo+mY2Be2vYRr9i3SnA88DfgHfKlqlvP+U+\nBro3wzJXActJ2rps+uLFv7Nbc5JuK2qbKqn6Z3UE6Q/M4cDrkt6tYVuNdW/J+1D+fo6TNI10Ov1n\nvnu/FwfGRcTMGrb3ScnxVB/LLqQW5C4R8U3Z8v3KfhY1nVL+DjhC0hKNPLZ6OYCax77AwxExrnh9\nC98/DftjRHQDViD1WaxWPUPSssDPSL/cAPeR/sJtW7aNxYttdAaeBf5RSz3jgaXm4jgWj4huJV9v\nlM6MiGdIfTOnAA8Wp5PlRhTrdgWWBNYktTwavJ8aLE3qu2jSMhExHTir+FLJrPHFv0uVLLt78V6P\nAtoW06ZFxDkRUUUK+aHAHUWf2xwkXVnSqXtSHWUNLHkfBpbNW5zU53QcsAlQ/QdpHLB4dSd7maWK\n+dV1rELqb9wvIt6rYfkRZT+L7/URRsSrwIPA9zrCm8oB1ESSOpHOvTcurkR8ChwNrCNpnfLlI+Lf\npE69S4t1IbWG2gAPFOu/RwqgGk/Div/4Q4D+khavYZFHgZ0ktcTP9ybgWOo5/QKIiM9IpzXbz+3O\nJHUlnVI+Xc+iOzVgGYDrSZ22O5VMe5PUqty5oXVFxFekYO0CrFjD/ENKOnXLA7jBin6uC0mnwIcV\nk4cD08vrLU7vtwYeK153Jr3/V0bEfXNbQ+E04EBS0DcbB1DTDQRmkTqEexVfq5P+M9QWII+QThkO\nKibtQ+pU7FXytQuwraQe5esrXebfm3TKML58PqlDdRHgBknLF+ssLekiSWvP3WHOdhmpj+Gp+hYs\nat+J1IHaKJI6SKoinXZMJAVH+TJtJa0o6XJSC+GM+rZbnLacDpxQMi1IoXqapAMlLaZkFWB2X56k\nUyX1kbSQ0qX5o4Avgbcae3xz4VzgeEkdI2IS6Vgvl7SVpPbFVdg7gA9Jp70AfyG1Ck9u6s4j4l3g\nduDIpm6rlAOo6fYFro+If0fEp9VfwBXAnrU0kyFdXTi+uCqzAvCn0vUj4n5Sp+MeJetU9618RroM\nv0PUMKBTcRVnA2AG8JykyaS/ipP4fud3qS/L7gc5pqZtR8RjNe230L96fdIVsC9IfScN3c/xRb0T\nSK2skcAGEfF1+T6Ar0gdpIsAfSLilTqOrdStlPWRFbcCDCJdQfoP6TRmKHA16T82pFssri/mfUwK\n4m2Lzt+W9ndSEB9Y1Hs+cBLwR9L78FxR96YRMV3ScqQ/bP2ASWXvd2m9/cvnFRcPanImqcXXbFT7\n75GZWctyC8jMsnEAmVk2DiAzy8YBZGbZ1HaFxkp077F4LLPs8rnLsDLt26r+hSyLUaNGjouIeu+c\ndgA1wDLLLs/9jz6buwwrs2S3jrlLsFp0aq+xDVnOp2Bmlo0DyMyycQCZWTYOIDPLxgFkZtk4gMws\nGweQmWXjADKzbBxAZpaNA8jMsnEAmVk2DiAzy8YBZGbZOIDMLBsHkJll4wAys2wcQGaWjQPIzLJx\nAJlZNg4gM8vGAWRm2TiAzCwbB5CZZeMAMrNsHEBmlo0DyMyycQCZWTYOIDPLxgFkZtk4gMwsGweQ\nmWXjADKzbBxAZpaNA8jMsnEAmVk2DiAzy8YBZGbZOIDMLBsHkJll4wAys2wcQGaWjQPIzLJpl7sA\nq9vECePZc+dtAPji889o27YN3XssAcAbr41m8KFHcsqZ5wFw9Z8uZurXX/Ob409p0j5333ELPv/s\nUzp27ATAjXc8wOJL9OSsU37L8GeeAmDatKmMH/cFo8d82qR9LUi6dGjLWmv9ZPbroXfdy9ixH7Dr\nzjuy4oo/4pvp37DroN05+dTTmrSfu+68g9+fdTpvvvEGT//zeap6925q6dk4gOZzi3XvwUNPPAfA\nJeefTecuXTjo10cDsNoy3Rj29/s47Kjf0r3H4s2630uuvJ61e1XNMe3Usy+Y/f2Qa/7M66+83Kz7\nbO06derEcyNfmmPa2LEfsOFGA7j7vgf5+uuv6du7F1tvsx3rVVXVspX6rbnmWtw29G4OP+zgppac\nnU/BWrF2bduxx96D+euVl8/zfT9wz1C233nQPN9va9alSxfWXa+K994b06Tt/Hj11Vl1tdWaqaq8\n3AJq5fYefDBbb9yHg484ptZlhj/zJGedcvz3pnfq3Im7HnqixnWOP/Jg2rRpy1bbD+SIY05E0ux5\nH/5nLP8ZO5YNBmzS1PIXKNOmTaNvVS8All9xRYbeec8c88ePH8/zz43gf046dY7pkydPZrNNBtS4\nzSF/u4XV11ijZQqeD7RYAEkK4KKIOLZ4fRzQNSJOb+b9nBQR55S8/mdEbNCc+5ifLbzwIuw8aE+G\nXPNnOnbsWOMy/TfaePZpXENccuX1LLnU0kyZMplD99+Du4fewi677Tl7/oP33MHW2w+kbdu2Ta5/\nQVLTKRjAs888Tb/e69KmTRuO++2JrLHmmnPMX3jhhWtcrxK0ZAtoOrCzpD9ExLgW3M9JwOwAqqTw\nqXbAwYez3ab92XWPfWqc39gW0JJLLQ1A164Ls+POu/HyqBfmCKAH7rmTM8+7uHmKrwDVfUC1cQuo\nZcwErgaOBk4unSFpCeBKYLli0m8i4tli+i1AD+AFYCugKiLGSboXWBboCFwaEVdLOhfoJOkl4LWI\n2FPSlIjoKul24IaIeKjY5xDgAeBe4FxgE6AD8KeIuKrF3oV5oNti3dl2x10YevMQdv3lvt+b35gW\n0MyZM/lq0pd077E4M2bM4LGHH2KjjX8+e/6Yd99m0qSJrNenX3OVX/EquQXU0p3QfwL2lLRo2fRL\ngYsjog+wC3BtMf004PGIWA+4h+8CCuCAiKgCegNHSuoREScC0yKiV0TsyZxuA3YDkLQQsCnwEDAY\nmFTsuw9woKQVm+l4s/nVoUcxYcL4Jm/nv9Ons++gHdhq4z5s+7O+LLnUD9l97wNmz3/g7qFsP3DX\nOfqEbN667957WGmFZXhuxHB23nFbtt9my9wlzTVFRMts+LuWyJnADGAaRR+QpM+Bj0sWXwL4MfA0\nsFNEvF9sYwKwatECOh3YqVh+BWDLiBhRvZ8a9tsReAdYmdSSGlS0kO4E1gamFqssChwcEQ+X1X8Q\ncBDAD5dZturZF99upnfGmsuS3Wru87L8OrXXyIio9waleXEV7BJgFHB9ybQ2QP+ImFa6oGr5sypp\nE2CzYp2pkp4gnYrVKiK+KZbbktQSurV6c8ARETGsnvWvJp1CsnavqpZJabMK1+L3AUXEBGAo6dSn\n2sPA4dUvJPUqvn0GGFRM2wJYrJi+KDCxCJ8fA6UdEDMkta9l97cB+wMDgOrAGQYcWr2OpFUldZnL\nwzOzJphXNyJeCJTeqnsk0FvSaEmvA4cU088AtpA0Ctga+ASYDPwDaCdpNHAWMKJkW1cDoyXdXMN+\nHwZ+CjwaEf8tpl0LvA6MkvQqcBW+H8osixbrA5obkjoAsyJipqT+wF8iold967W0tXtVxf2PPpu7\nDCvjPqD51/zUB9QYywFDJbUB/gscmLkeM2tB81UARcQ7wLq56zCzecMfRjWzbBxAZpaNA8jMsnEA\nmVk2DiAzy8YBZGbZOIDMLBsHkJll4wAys2wcQGaWjQPIzLJxAJlZNg4gM8vGAWRm2TiAzCwbB5CZ\nZeMAMrNsHEBmlo0DyMyycQCZWTa1DkovaZG6VoyIr5q/HDOrJHU9FeM1IEiPMq5W/TpIj9AxM5tr\ntQZQRCw7Lwsxs8rToD4gSbtLOqn4fhlJVS1blplVgnoDSNIVwM+AvYtJU4ErW7IoM6sMDXky6gYR\nsZ6kFwEiYoKkhVq4LjOrAA05BZtRPKs9ACT1AL5t0arMrCI0JID+BNwFLCHpDOAZ4LwWrcrMKkK9\np2ARcaOkkcBmxaRdI+LVli3LzCpBQ/qAANoCM0inYb572syaRUOugp0M3Ar8EFgGuEXS/7R0YWa2\n4GtIC2gvoCoipgJI+j0wEvhDSxZmZgu+hpxOjWXOoGoHvNcy5ZhZJanrw6gXk/p8pgKvSRpWvN6C\ndCXMzKxJ6joFq77S9Rrw95LpI1quHDOrJHV9GPWv87IQM6s89XZCS1oJ+D2wBtCxenpErNqCdZlZ\nBWhIJ/QQ4HrSOEBbA0OB21qwJjOrEA0JoM4RMQwgIsZExCmkT8ebmTVJQ+4Dmi5JwBhJhwAfAT1b\ntiwzqwQNCaCjga7AkaS+oEWBA1qyKDOrDA35MOpzxbeT+W5QMjOzJqvrRsR7KMYAqklE7NwiFZlZ\nxVBEzRkjadO6VoyIx1qkovlQm849o8Nqg3KXYWUmvnBF7hKsFp3aa2RE9K5vubpuRKyYgDGzPDy2\nj5ll4wAys2waHECSOrRkIWZWeRoyIuL6kl4B3ileryPp8havzMwWeA1pAV0GbAeMB4iIl/FHMcys\nGTQkgNpExNiyabNaohgzqywN+SjGfyStD4SktsARwNstW5aZVYKGtIAOBY4BlgM+A/oV08zMmqQh\nnwX7HNh9HtRiZhWmISMiXkMNnwmLiINapCIzqxgN6QN6tOT7jsBOwH9aphwzqyQNOQW7vfS1pL8B\nj7RYRWZWMebmoxgrAss3dyFmVnka0gc0ke/6gNoAE4ATW7IoM6sMdQZQMRb0OqRxoAG+jdoGEDIz\na6Q6T8GKsLknImYVXw4fM2s2DekDel7Sei1eiZlVnLrGhG4XETOBjYADJY0BviY9oDAiwqFkZk1S\nVx/Q88B6wMB5VIuZVZi6AkiQnoY6j2oxswpTVwAtIemY2mZGxEUtUI+ZVZC6Aqgt6Ymomke1mFmF\nqSuAPomIM+dZJWZWceq6DO+Wj5m1qLoCqM4no5qZNVWtARQRE+ZlIWZWefxgQjPLxgFkZtk4gMws\nGweQmWXjADKzbBxAZpaNA8jMsnEAmVk2DiAzy8YBZGbZOIDMLBsHkJll4wAys2wcQGaWjQPIzLJx\nAJlZNg4gM8vGATSfmvKvyxhx24mzv5ZbqjsDqlZh2otXsM1P15q93F2XHsKAqlWaZZ8Ld+nImGFn\nc/EJu86e1r5dW644ZQ9G3/s7Xrr7FAZu2mv2vF02X5dRd53MyDtPZsg5+zVLDa3V+PHj6VvVi75V\nvVhhmSX50fJLz37dqb3oW9WLql5r8cvdd2Xq1KlN2tdbb77Jxhv1Z9EuHbj4oj820xHkUddTMSyj\nadNn0G/3c+eYtvwPe/DhpxM5YfCWPPTUq82+z9MO25anR747x7QTfrUlX0yYzNoDz0QS3RftDMBK\nyy3BcQdswc/3u4gvJ09jicW6Nns9rUmPHj14buRLAJx95ul06dqVo485DoDFu3WdPW+/vffkmquu\n5Kija33kXr0W696dCy++jAfuv7fphWfmFlArM/rtj5g05Rt+3vfHzbrddVdflp49FuHR4W/MMX3f\nHftzwXUPAxARjP/yawAO2GkDrhr6FF9OngbAFxOnNGs9C6oNNxrAe2PerX/BOvTs2ZPeffrQvn37\nZqoqH7eA5lOdOrRnxG0nAjD2o/Hsduw1s+edd+0/OO3X2/H4c2/Wuv7R+2zKbtv0+d70Z0e9y7Hn\n3znHNEmce8zODD7lBjZZf7XZ0xft2gmA0369HQOqVuH9D7/g6HPv4PMJk1ll+Z4APH790bRt04az\nr3qIR/45Z3jZnGbOnMnDw/6XzbfY6nvz9vrlbrzz1lvfm37kb45hz733mRflZTHPA0jSLOCVYt9v\nAPtGRKNOiiVdC1wUEa9LOikizimZ98+I2KBZi86gplOwas++OAaADdddqdb1L77xMS6+8bEG7evg\nQQMY9sxrfPjZl3NMb9euDcssuRjDX3qPEy68myP3+jl/OHonBp96I23btmXl5XqyxYGXsnTPxXjs\nut9Q9YtzmDRlWgOPsHJMmzaNvlWp72yDjQaw3wGDv7fMTbfcPq/Lmi/kaAFNi4heAJJuBg4BGvWc\n+Yj4VcnLk4BzSua1+vBpiPOuHcbxg7dk5qxva5zfmBZQ37VXZMN1V+KgQQPo0qkDC7Vvy5Rp0zn1\nsvv5etp07nv8ZQDufmQU+w7sD8BHn3/J86PfZ+bMbxn78Xje/uBzVl5uCUa+/u9mPtLWr1OnTrP7\ngGrjFlAeTwNrA0g6BjigmH5tRFwiqQswFFiG9Kz6syLidklPAMcBvwA6SXoJeC0i9pQ0JSK6Srod\nuCEiHiq2PwR4ALgXOBfYBOgA/CkirponR9uMHhvxJqcdth1LLbFojfMb0wLa/+QbZn+/1/Z9qVpj\nOU697H4AHnrqVX7aexWefOFtNll/Nd587xMAHvi/lxm0VW9ueuA5enTrwirL9+T9j8Y38agql1tA\n85ikdsDWwD8kVQH7A31Jj4R+TtKTwI+AjyNi22KdOf63RcSJkg6vblGVuQ3YDXhI0kKkJ70eCgwG\nJkVEH0kdgGclPRwR75fVdxBwEADt588rPOf9dRh3XnJwi+7jlEvv5a9n78sFx+3CuIlTOPj0mwB4\n5J9vsFn/1Rl118nMmhWcdMm9TJj0dYvWYsmnn37Khv16M/mrr2jTpg1XXHYJL45+nUUWWSR3aY2m\niJi3O/yuDwhSC+hYUjD0iIjfFcucBXwB/AMYRmoFPRgRTxfznwCOi4h/Vbd4SrZf3QLqCLwDrAxs\nBQwqWkh3klpd1f1OiwIHR8TDtdXcpnPP6LDaoOZ5A6zZTHzhitwlWC06tdfIiOhd33JZ+4CqSVJN\nC0bE20XraBvgD0VL5cyG7CQivimCaktSS+jW6t0BR0TEsLk9ADNrHvPLfUBPAQMldS76fXYCnpb0\nQ2BqRNwE/BFYr4Z1Z0iq7YaI20indgNILSmKfw+tXkfSqsU+zWwey90JDUBEjCo6iZ8vJl0bES9K\n2hK4QNK3wAzSqVq5q4HRkkZFxJ5l8x4GbgTuj4j/Vm8bWAEYVbS8vgAGNusBmVmDzPM+oNbIfUDz\nJ/cBzb8a2gc0v5yCmVkFcgCZWTYOIDPLxgFkZtk4gMwsGweQmWXjADKzbBxAZpaNA8jMsnEAmVk2\nDiAzy8YBZGbZOIDMLBsHkJll4wAys2wcQGaWjQPIzLJxAJlZNg4gM8vGAWRm2TiAzCwbB5CZZeMA\nMrNsHEBmlo0DyMyycQCZWTYOIDPLxgFkZtk4gMwsGweQmWXjADKzbBxAZpaNA8jMsnEAmVk2DiAz\ny8YBZGbZOIDMLBsHkJll4wAys2wcQGaWjQPIzLJxAJlZNg4gM8vGAWRm2TiAzCwbRUTuGuZ7kr4A\nxuauo5ksDozLXYTVaEH62SwfEUvUt5ADqMJI+ldE9M5dh31fJf5sfApmZtk4gMwsGwdQ5bk6dwFW\nq4r72bgPyMyycQvIzLJxAJlZNg4gM8vGAWRm2TiAzFoBScpdQ0tol7sAmz9JUkSEpKVIV0s/zl1T\npar+WRTfbwYsAjwHfBoRs7IW10RuAVmNivAZCNwK/EXSeZKWyV1XJSoJn6OAM4C+wOPA+jnrag4O\nIKuRpJ8AxwDbAc8DPwMmZS2qgklaFdg4IjYEPgD+TWoFVc9vladoDiCrzSzgQWBXYFtg94iYLGnN\nvGVVHkk9gI+B0ZKGAAOBrSPiW0n7Slo0WukdxQ4gm4OkNST9AvgvMAA4DNgnIt6TtDVwjaQlsxZZ\nQST1A/4HmAksCawMDI6ImZL2Ao4FFs5YYpP4oxg2B0kHAvtFxIaSfkPqZ3gcmAqcDJwQEQ/mrHFB\nVZxGKSK+LZm2IvAY8CvSadf5wESgLbAusGdEvJqh3GbhAKpwJVe72lZfUZF0CzA8Ii6X9CtgeaA7\ncF9EPFx6VcaaT9nVrh7A9IiYImkX4GcRcbikVUgtoR8AL0REqx4oz5fhK1TRqblORNwhqTewsaQx\nEXEvcB2wBUBEXFss3z4iZhTTHD7NqGj5/AQ4FdhVUhVwIvCBpOuAEcCOklaJiHeAd/JV27zcB1S5\n2gCfS1oY+A/QAfi1pCuAGcA2kvYuWX5mhhorQiSjgcMlbQK8RAqjz4F7SH1xKwEXSlooW6EtwAFU\noSLiTeBZUvgMjIhzgB1IfQv9gG7AvpK6Fsu71dMCJHUqeTkO2B94FXg/Ii4AjiKd/k4HVgc6z/Mi\nW5D7gCqIpM7A5hFxn6S+pCtdAv4B/D4iLpXUhtTHsCvwbkT8PV/FCzZJHUlXsR4iXd36SUT8rjjt\n6g/0iojpktoBXYAeEfFevoqbnwOowhT3kfQGvgEOjIgXJa0HPAqcEhF/LlveHc4tQNLiETFO0gDg\nSeBdUgBNL+ZfT7rK1S8ivslYaovyKViFKLlT9g+kJv3MiHgRICJGAZsBlxa3+8/m8GleSpYFzi5O\nb18H7gOWIv1hACAi9gdeA57KUug84hZQBSi51N4G6AosRrrSNSMitipZbhVghYh4JFOpFUPSIsBa\nQJeIeETSz4F7gV9GxIOS+kXECEk9I+LzvNW2HLeAFnAl4bMFcArpIxVjI2JTYCFJD0jqK+lJYHzx\nn6FVfq5oflf6vkbEV8A6wO8kbRURjwN7AXdIuhC4TtIyC3L4gO8DWuAV4bMVcCFwOHCrpHWAUyPi\n55JuJX3C+sKImFC9Tr6KF0xlNxn+EpgUEX+RNAP4bTH/fkmbAxuTrkx+mLPmecGnYAuw4pRrYeAG\n0n0lPwAuAD4CvgSOiIiJkrpFxJfucG55kn5N+ljFoOKmwupAOgC4rAihivk5uAW0ACr5Be4YEZMk\nDSZ1PJ9J6ujsDHwCfCjpjIj4EtzyaUnF6dfKwD6k0QU+lbQTsCxwE9AeGCzpsYj4Ol+l85YDaAFT\n0ufTF/izpP0i4hVJPUn3/SxGCqNhwF0RMS1nvQuy0pZM8e87RV/bbcCbpJ/DJKB7RJwu6b5KCh9w\nAC1wivDZHNiZdFfzMElbFiH0PHAzsAJweES8kLHUBVpZn88GpOB/Cbid9FmuxyNijKRDgLWL1Spu\nwDf3AS1giuEbHgL2Ly7j/g7Yj9TsH0M6BZsZEc/nq3LBVd5/I+k4YHfgC2A88AxwczG422DgUNLw\nJ612SI2mcAtowTMe+Bdp2E4i4szi/p5hwIYR8c+MtVWCdqQP81IM3LYlMCAiphXDagwA1pT0BelO\n5/0rNXzA9wG1etX3lkhaVGlozq9IV752LllsCPAhcF/1h0ut+RWnvjdKOrG49WE86cbPnwJExF2k\nUQd2jIgxwLER8Uq2gucDbgG1ckWfz/akAeQnShpBGsLzVqWnWEwlDSw/GDiC9KHGKbnqXVAVgXMm\n8DegJ7AHaeTCW4D1JU0sTntHAqsqDQA3PVvB8wm3gFqh0jtqlcYMPgnYm/T0igMj4g1gEKnV05V0\n38kPgA2Ab7+3QWsSSd1J/W5nRcTlwNVAR6AHaaQBARdLupo00NgN0cqf59Vc3AndykhagvRUhFuL\n4Tp/Shq7pwOpFfTLiHhf0goR8UGxzgbAjaS7ayu2v6ElSdqWNF5z/4j4StLNwJMRcbWkxYAVSVcf\nR0YrH0a1OfkUrPXZkPRgug7F0BptSZ9wH096VMuXRV/EIcUl3vHAWGBT/+K3nIj4u6RvgZGShgGd\nSDcYEhETSadjozKWOF9yC6iVKPoMZklqS2oBbQK8Xnye6CxSp/MvSPeU/A443oOJzXtKj05+GFgy\nIj6X1HFBHs+nqRxArYCk1UjGouh8AAAECUlEQVT9OA8DTxWj5G0NbE0KoSslnU4aU6YbcF1EDKuk\nzxTNT4qfzR9JT7JYoD/N3lQOoFZA0sbA/5HuoB0K/Ij0odLNgYVIT80cUlwR81/c+YCkHYHTSDd+\nhv8Q1MwB1EpI2oj0qOS+wC6kW/t3Il3pWhk4nTTIGFHyYDvLR1LXiPAtD3VwJ3QrERHPSNoDuBPY\noLiV/0HS86QOIj1FwcEzH3H41M8toFZG0jbA5UCf6gHESj4B7z4fa1XcAmplIuKh4nLvm5JWi4iJ\nZUM+mLUabgG1UsWNb19HxBO5azGbWw6gVs6nXdaaOYDMLBt/GNXMsnEAmVk2DiAzy8YBZA0maZak\nlyS9KukOSZ2bsK1NihspkbSDpBPrWLabpMPmYh+nF2MyN2h62TJDJP2iEftaQZKHOmkkB5A1xrSI\n6BURa5Ee8XNI6Uwljf6dioj7I+LcOhbpBjQ6gGz+5wCyufU0sHLxl/8NSX8mjXezrKQtJA2XNKpo\nKXWFNGyppDclPUPJmNWS9pN0RfH9DyTdI+nl4msD4FxgpaL1dUGx3G8lvSBptKQzSrZ1sqS3JD0K\nrFbfQUg6sNjOy5LuKmvVbSbpaUlvS9quWL6tpAtK9n1wU9/ISuYAskaT1I40FEj1gOqrATdGxLrA\n18ApwGYRsR7pCR3HSOoIXANsT3oyxJK1bP4y0kiC6wDrAa+RhjEdU7S+fitpC2AVYH2gF1Al6aeS\nqkiPwFmXFHB9GnA4d0dEn2J/b5DGzq62Auk57dsCVxbHMJj0XPc+xfYPVHoUks0FfxTDGqOTpJeK\n758G/gr8EBgbESOK6f2ANYBni6GrFwKGAz8mfWC2+nnoN5E+RFvu56THF1OMmzypGNK01BbF14vF\n666kQFoYuCciphb7uL8Bx7SWpLNJp3ldSY8vqja0+IDvO5LeK45hC2Dtkv6hRYt9v92AfVkZB5A1\nxrSI6FU6oQiZ0scJC3gkIvYoW64X0Fx3vQr4Q0RcVbaP38zFPoaQxsp+WdJ+pJEmq5VvK4p9HxER\npUGFpBUauV/Dp2DW/EYAG0paGUBSZ0mrkp6FvqKklYrl9qhl/cdITwut7m9ZBJhMat1UGwYcUNK3\ntLSknsBTwE6SOklamHS6V5+FgU8ktQf2LJu3q6Q2Rc0/At4q9n1osTySVpXUpQH7sRq4BWTNKiK+\nKFoSt0rqUEw+JSLelnQQ8HdJ40iPKF6rhk0cBVyt9NjiWcChETFc0rPFZe7/LfqBVgeGFy2wKcBe\nETFK0u2kZ7CPJZ0m1udU4Lli+VeYM+jeAp4kPdLokIj4RtK1pL6hUUo7/4I0RrfNBX8WzMyy8SmY\nmWXjADKzbBxAZpaNA8jMsnEAmVk2DiAzy8YBZGbZ/D92eH/ST/bXGQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8bb7570b00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds, labels = get_predictions(model_pt_frozen, X_data, y_data)\n",
    "\n",
    "y_trues = [np.argmax(ii) for ii in labels]\n",
    "y_preds = [np.argmax(ii) for ii in preds]\n",
    "\n",
    "cm = confusion_matrix(y_trues, y_preds)\n",
    "fpr, tpr = print_statistics(cm)\n",
    "\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm, classes=['Negative','Positive'], title='ARABIC EMBEDDINGS - FROZEN')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TP: 94\n",
      "TN: 47\n",
      "FP: 11\n",
      "FN: 373\n",
      "TPR/recall: 0.201284796574\n",
      "TNR/specificity: 0.810344827586\n",
      "PPV/precision: 0.895238095238\n",
      "NPV/negative predictive value: 0.111904761905\n",
      "FNR/false negative rate: 0.798715203426\n",
      "FPR/false positive rate: 0.189655172414\n",
      "FDR/false discovery rate: 0.104761904762\n",
      "FOR/false ommission rate: 0.888095238095\n",
      "F1 score: 0.328671328671\n",
      "Accuracy: 0.268571428571\n",
      "MCC/Matthews correlation coefficient: 0.00911420561544\n",
      "BM/Bookmaker Informedness: 0.0116296241601\n",
      "MK/Markedness: 0.00714285714286\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAASAAAAEmCAYAAADC2k2dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XecVNX9xvHPQ6+KIho7BgS7SFFR\nscSCRmMXjQVUgiVqjC36s5dYEhNbjD2GECtqrDGCJSoWNILGHhUVjdFIU2ki4Pf3x7lDhnHLLOxy\n2Z3n/XrNC+bWc3dmnz3n3HvPVURgZpaHZnkXwMwqlwPIzHLjADKz3DiAzCw3DiAzy40DyMxy4wAy\ns9w4gMok6UlJ0yS1Lpk+XNI3kmZImirpUUnrVLH+tpJC0i9KpnfNps/IXv+VdI2klkXLfChph6L3\nK0v6g6RPJU2X9Lak8yS1r2K/pdsvvPYvKn9I2r1kvSuy6Ydm7w+VNL9o/fclHV3H/XyTlXe6pNcl\nXSxp2aJtlO7jA0l/lNSjiv20KCn/pkXLdJe00AVuknaU9Pds31MkvSLpVEltsvmdJN0s6bNsmXck\nnVr68yxHyfF/K2l20fuDJJ0raW72/gtJz0nqX8V21srWv6aKeSGpe/b/c7P3+xXNb5FN61qyXmHZ\nTUuml/v5tqiiLMXHU3h9Uc7PygFUhuxDHAAEsHsVi/w6IjoAqwKfAH+oYpkhwNTs36p0yraxIdAf\nOKaasiwPPA+0BfpHREdgR6AT0K2Gw+gUER2KXncWzXunuFzZl2w/YELJNp4vrA/sC/xa0iZ12M+v\ns/J2AQ4DNgeeLQnO57PtLwvsAMwGxknaoIZjmwr8srqZ2S/m3cBtwJoR0RnYH1gNWD1b7HKgA7Bu\ntu/dqzj+shQfP/AR8KOiabdmi92ZzV8B+DtwVxWbGgxMAw5QyR++KkwFzpfUvLoFJAk4hOq/h+V8\nvtW5s+Rz71TOSg6g8gwGxgLDqT5AiIjZwEigV/F0Se1IH+gxwNqS+tawjc+BR4H1qlnkRGA6cHBE\nfJit83FEHB8Rr5Z5PKUeBLaUtFz2fmfgVeCzGso5HniL9AtbJxHxdUT8g/RL3pkURqXLzI+ICRHx\nU+Ap4NwaNvknYCNJ25TOyH7pLgPOj4gbI2Jqtv1/RcRxEfFutmg/4LaImBYR30bE2xFxd12Pra4i\nYh5wK7CqpC4lswcDZwJzgR/VsqlHgG+Ag2tYZgCwCnA8KdRa1VCuRf5868IBVJ7BpC/JrcBASStV\ntVD2l/zHwHsls/YBZpD+yo3KtlclSasAA0mBV5UdgL9ExLd1OYBafA08AByQvR8MjKhpBUn9gB7A\nS4u604iYTgrbAbUs+pdalpkFXARcWMW8nqSazj217GMscKGkwyStXcuy9SYLgcHAFFJtpzB9AKnc\nd5D+qFX7nckEcBZwjoqa7yWGkP7YFGqlu9VQrsX+fMvhAKqFpK2ANYGRETGOVC0/sGSxk7M273Rg\nK1I1t9gQUhV1PqkZ8OMqviSTs218AswkNRmq0hn4dBEOZXLW31B4lf5lGwEMzvpktgHuq2Ibm2fr\nzgBeBP4MvFuyTG37KfUfYPl6WOZ6YA1Ju5RMXyH7d0FtTtIdWdlmSSp8VseR/sAcC7wp6b0qtlWf\nBmWf92xgGLBvVhsqGAL8LSKmkb4zu0hasaYNRsQDwCTgJ6Xzslr4fqRa3lzS96u0Nl/O51vj8RS9\n/l7OSg6g2g0BRkfE5Oz9bXz3g/tN1ubtSvpC9SzMkLQ6sB3pyw1wP9AG2LVkGytk22gHPEuqUldl\nCrDyIhzHChHRqej1VvHMiHiG1DdzJvBQ1pwsNTZbtwPwPWB9Us2j7P1UYVVSn8RiLRMRc4ALspeK\nZk3J/l25aNkDsp/1eKB5Nm12RFwUEX1IIT8SuCvrc1uIpOuKOltPr6Xs1RmZlWEl4HWgT9H225LC\n4tasbM+T+pJK//BV5UzgDNJ3rNhewDzg4ez9raRQK272lfP51ng8Ra/tylnJAVSD7IswCNgmOzvy\nGXACsLGkjUuXj4iPSO3rK7N1IdWGmgEPZuu/T/pyVFmlzn7xhwP9Ja1QxSKPAXtJaojP7hbgJGpp\nfgFExH9JzZra+iaqJakDqUk5ppZF9ypjGYA/kjqQ9yqa9japVrl3ueWKiK9Iv3jtgbWqmH9UUWdr\nub+g1e1rMnAkcK6kQkjuBSwDXFP0vVuV2pthRMSjpC6An5bMGkLqZP8o295dQEtSl0FV21nsz7cc\nDqCa7QnMJ3UI98pe65J+GaoLkEdJTYYjskmDgfOK1u9F6hPaVVLn0vWzsx2HkJoMU0rnkzpUlwH+\nJGnNbJ1VJV0maaNFO8wFriKdUXu6tgWzsu8FvFHXnUhqLakPqZk3jRQcpcs0z05D/w7YlvQzrFHW\nhDkXOLVoWpBC9RxJwyQtp2RtUu2jsL+zJPWT1Erp1PzxwBfAv+p6fHUVEW+T+gYLl2gMAW4mnREt\nfGe2BHpJ2rCMTZ5RtC0krQpsT+rzKWxvY+BXVHNSpYbPt7WkNkWvxcoQB1DNhgB/jIiPIuKzwgu4\nGjioqmsiMpcCv8jOynQFfl+8ftZWf4+F//oU2t7/JZ2G3z2qGKwpO4uzBenMyAuSpgOPA1/y3c7v\nYl+UXKdxYlXbjojHq9pvpn9hfdIZkkmkvpNy9/OLrLxTSbWsccAWETGzdB/AV8CTpLDtFxGv1XBs\nxW6npI8suxRgEOkM0cfAZFIT6wb+d/o7SEE4mfQHZEdg14iYUeZ+F9elwBHZH5XtgStKvjPjSM3y\nas/CFkTEs6Q+nIJDgFciYnTJ9/gq0tnDwiUO5Xy+M0jdDIXXD7Lp+5d87jNq67MCUPXfNTOzhuUa\nkJnlxgFkZrlxAJlZbhxAZpab6s7iWJHlO68Qq62xZt7FsBItm6n2hSwX48ePmxwRpfe2fYcDqAyr\nrbEmDz/xXN7FsBIrdKztBnHLS9uWmljOcm6CmVluHEBmlhsHkJnlxgFkZrlxAJlZbhxAZpYbB5CZ\n5cYBZGa5cQCZWW4cQGaWGweQmeXGAWRmuXEAmVluHEBmlhsHkJnlxgFkZrlxAJlZbhxAZpYbB5CZ\n5cYBZGa5cQCZWW4cQGaWGweQmeXGAWRmuXEAmVluHEBmlhsHkJnlxgFkZrlxAJlZbhxAZpYbB5CZ\n5cYBZGa5cQCZWW4cQGaWGweQmeXGAWRmuXEAmVluHEBmlhsHkJnlxgFkZrlxAJlZbhxAZpYbB9BS\nbtrUKQzcelMGbr0pvddZk77rf3/B+9WXb8P5Z566YNnrfnc5l11yQb3t+7AD92H7LXoveH/04Qcv\n2Hf/jXswcOtN621fjVX71s3ZrE+vBa+JH37I0089yUqdl2XzvpvQa8N1ufCC8xZ7P/fcfRe9N16f\ndq2aMe6llxZMnzJlCgN32I4VOnXg5z87drH3s6S1yLsAVrPllu/MqKdfBOCySy6gXfsOHHXcCQB0\nX3lZHnnoPo494RSW77xCve73bw/eR/v27Readu3Ntyz4//lnnsoyyyxTr/tsjNq2bcsL415ZaNrE\niR+y5VYD+Mv9DzFz5kw269uLXX64G7379Fnk/ay//gbcMfIvHPvTIxea3qZNG84+9wLefON13njj\n9UXefl5cA2rEmrdowYFDhnLjtVfV63ZnzpjBjddcyc9O+r8q50cED913N3vss3+97rcpat++PZv0\n7sP7709YrO2ss+669OjZs8rtb7nVVrRp02axtp8X14AauSFDj2KnAX05+mcnVbvMc2Oe5LwzfvGd\n6W3btuO+UU9+Z/qlF53HsGN+Ttt2bavc3gvPP8MKK67EWt26L3K5m4rZs2ezWZ9eAKy51lqMvPve\nheZPmTKFF18Yy/+dftZC06dPn84O2w6ocpvD/3wb6663XsMUeCnTYAEkKYDLIuKk7P3JQIeIOLee\n93N6RFxU9P65iNiiPvexNOu4zDLss/9B3Hz972nTpurA2GLAtguacbV547V/MvGDCZx70aV8/NGH\nVS5z/z0j2WPvQYta5CalqiYYwLPPjGHzvpvQrFkzTj7lNNZbf/2F5nfs2LHK9SpNQ9aA5gB7S7o4\nIiY34H5OBxYEUCWFT8HQo4/jh9tuzqADB1c5vy41oHH/eIFX//ky/Tfuwbx585ky+XP2+9GO3PXg\nowDMmzePRx66n4efeK7ej6MpKfQBVcc1oKQhA2gecANwAnBG8QxJXYDrgDWyST+PiGez6bcBnYF/\nADsDfSJisqT7gNWBNsCVEXGDpEuAtpJeAd6IiIMkzYiIDpLuBP4UEQ9n+xwOPAjcB1wCbAu0Bn4f\nEdc32E9hCVhuueXZbc99ueOW4ex/0JDvzK9LDWjw4Ucw+PAjAPj4ow859IC9F4QPwJgnn6Db2j1Y\nedXV6qfwFco1oKShO6F/DxwkadmS6VcCl0dEP2Af4KZs+jnAExHRG7iX/wUUwOER0QfoC/xMUueI\nOA2YHRG9IuKgkn3cAewPIKkVsD3wMDAU+DLbdz9gmKS16ul4c3PEMcczdeqUBt/PA/eOdOdzDu6/\n7166dV2NF8Y+z9577MqPfjhwwbye3bty6ikncsuI4XTruhpvvflmjiWtG0VEw2z4fzWR84G5wGyy\nPiBJnwP/KVq8C7AOMAbYKyI+yLYxFeiR1YDOBfbKlu8KDIyIsYX9VLHfNsC7QHdSTWpQVkO6G9gI\nmJWtsixwZESMLin/EcARAKuutnqfsa++W08/GasvK3RsnXcRrBptW2pcRPStbbklcRbsCmA88Mei\nac2A/hExu3hBSapqA5K2BXbI1pkl6UlSU6xaEfF1ttxAUk3o9sLmgOMiYlQt699AakKy0SZ9Gial\nzSpcg18HFBFTgZGkpk/BaGDBZZuSemX/fQYYlE3bCVgum74sMC0Ln3WAzYu2NVdSy2p2fwdwGDAA\nKATOKODowjqSekhqX836ZtaAltSFiL8Fii/V/RnQV9Krkt4EjsqmnwfsJGk8sAvwKTAdeARoIelV\n4AJgbNG2bgBelXRrFfsdDWwNPBYR32TTbgLeBMZLeh24Hl8PZZaLBusDWhSSWgPzI2KepP7AtRHR\nq7b1GtpGm/QJn3Ze+rgPaOm1NPUB1cUawEhJzYBvgGE5l8fMGtBSFUAR8S6wSd7lMLMlwzejmllu\nHEBmlhsHkJnlxgFkZrlxAJlZbhxAZpYbB5CZ5cYBZGa5cQCZWW4cQGaWGweQmeXGAWRmuXEAmVlu\nHEBmlhsHkJnlxgFkZrlxAJlZbhxAZpYbB5CZ5cYBZGa5qXZQeknL1LRiRHxV/8Uxs0pS01Mx3gCC\n9CjjgsL7ID1Cx8xskVUbQBGx+pIsiJlVnrL6gCQdIOn07P+rSerTsMUys0pQawBJuhrYDjgkmzQL\nuK4hC2VmlaGcJ6NuERG9Jb0MEBFTJbVq4HKZWQUopwk2N3tWewBI6gx826ClMrOKUE4A/R64B+gi\n6TzgGeBXDVoqM6sItTbBImKEpHHADtmk/SLi9YYtlplVgnL6gACaA3NJzTBfPW1m9aKcs2BnALcD\nqwCrAbdJ+r+GLpiZNX3l1IAOBvpExCwASRcC44CLG7JgZtb0ldOcmsjCQdUCeL9himNmlaSmm1Ev\nJ/X5zALekDQqe78T6UyYmdliqakJVjjT9Qbw16LpYxuuOGZWSWq6GfUPS7IgZlZ5au2EltQNuBBY\nD2hTmB4RPRqwXGZWAcrphB4O/JE0DtAuwEjgjgYsk5lViHICqF1EjAKIiAkRcSbp7ngzs8VSznVA\ncyQJmCDpKOATYMWGLZaZVYJyAugEoAPwM1Jf0LLA4Q1ZKDOrDOXcjPpC9t/p/G9QMjOzxVbThYj3\nko0BVJWI2LtBSmRmFaOmGtDVS6wUS7nX//Uxa//gpLyLYSWevufCvItgi6mmCxEfX5IFMbPK47F9\nzCw3DiAzy03ZASSpdUMWxMwqTzkjIm4q6TXg3ez9xpJ+1+AlM7Mmr5wa0FXAbsAUgIj4J74Vw8zq\nQTkB1CwiJpZMm98QhTGzylLOrRgfS9oUCEnNgeOAdxq2WGZWCcqpAR0NnAisAfwX2DybZma2WMq5\nF+xz4IAlUBYzqzDljIh4I1XcExYRRzRIicysYpTTB/RY0f/bAHsBHzdMccyskpTTBLuz+L2kPwOP\nNliJzKxiLMqtGGsBa9Z3Qcys8pTTBzSN//UBNQOmAqc1ZKHMrDLUGEDZWNAbk8aBBvg2IqodpMzM\nrC5qbIJlYXNvRMzPXg4fM6s35fQBvSipd4OXxMwqTk1jQreIiHnAVsAwSROAmaQHFEZEOJTMbLHU\n1Af0ItAb2HMJlcXMKkxNASRIT0NdQmUxswpTUwB1kXRidTMj4rIGKI+ZVZCaAqg56YmoWkJlMbMK\nU1MAfRoR5y+xkphZxanpNLxrPmbWoGoKoO2XWCnMrCJVG0ARMXVJFsTMKo8fTGhmuXEAmVluHEBm\nlhsHkJnlxgFkZrlxAJlZbhxAZpYbB5CZ5cYBZGa5cQCZWW4cQGaWGweQmeXGAWRmuXEAmVluHEBm\nlhsHkJnlxgFkZrmpaVB6y9GMl67i9ff+s+D9oBNuYM1VOjP6puPZ5/jrePjp1wG458qjuGLE44wZ\n9+4i72uNlZfj9t8Mo3nzZrRs0Zxr73iKm+5+hg7tWvPYzScsWG7VFTtxx8P/4JTf3MNP9t2KIwdt\nzfxvv2XmrDkc88vbefv9zxb9gBuhL6ZN5ZhDdgdgyqTPad68OZ2W7wzAu2+9ztrrbsD8efPp2r0H\n5156LW3atlvkfX315RdccOoxfPLRB7Rq3YazLrmabj3XWzB//vz5DNlzW7qstAqX33Tn4h3YEuQA\nWkrNnjOXzQ+4ZKFpa67SmX9/No1Thw5cEED14dNJX7HdoZfxzdx5tG/binF3n8Ffn3qNTyd9uVAZ\nnr31F9z3xCsA3Pm3l7jp7mcA2HWbDfnViXuzx7HX1FuZGoNOyy3PrQ+ln8ENV15Mu3YdOHjYcQBs\ns+GqC+addcIw7rntZg4aeuwi72v4Nb+lx3obcul1t/LhhHf49Tknc80tDyyYf8fwa+narSczZ0xf\njCNa8twEa2RefecTvpzxNT/YbJ162+bcefP5Zu48AFq3akkzffeBKN3W6MKKy3fk2fHpQbnTZ369\nYF77tq0Iot7K09T06teff0/8YLG28cF7/6LfFtsA0LVbDz795COmTP4cgP9++gnP/n00eww6ZLHL\nuqS5BrSUatu6JWPvOA2AiZ9MYf+Tblww71c3PcI5x+zGEy+8Xe36Jwzenv1/2O87058d/x4n/fru\n70xfbaVO/OWqo+m2ehdOv+I+Pp305ULzB+3ch7tHj19o2pGDtuZnB29Hq5Yt2PnIq+p0fJVi3rx5\nPP/Uo2y+9Q7fmXf6cYcx8YPvNp0PPPwYdt37xwtNW3vdDfj7qAfp1bc/b/xzHJ998jGff/ofOq+w\nIpf/8v847tTzmTWzcdV+IIcAkjQfeC3b91vAkIiYVcdt3ARcFhFvSjo9Ii4qmvdcRGxRr4XOQVVN\nsIJnX061kC036Vbt+pePeJzLRzxe9v7+/d8v2HT/i1m5y7KMvGwY9z72Mp9P/d8Xer+BfRh65oiF\n1rl+5NNcP/Jp9t+5L6f9ZGeGnf3nsvfX1M35ejYH7bYVAL369meP/b5bO7nod38se3uDj/w5l11w\nGgftthXde65Hj/U2onmL5ox54hGW69yFdTfsxbixY+qt/EtKHjWg2RHRC0DSrcBRQJ2eMx8RPyl6\nezpwUdG8Rh8+5fjVTaP4xdCBzJv/bZXz61oDKvh00pe8OeEztuzdjXsfS/09G/ZYlRbNm/PyWx9X\nuc7IUeO48vT9F+Eomq7Wbdou6AOqTl1qQB06LsPZv059bBHBnttsxCqrrcmjD/2FMY//jeeeHM2c\nOXOYOWM6Z594BOdfdkP9HUwDyrsJNgbYCEDSicDh2fSbIuIKSe2BkcBqpGfVXxARd0p6EjgZ2Bdo\nK+kV4I2IOEjSjIjoIOlO4E8R8XC2/eHAg8B9wCXAtkBr4PcRcf0SOdp69PjYtznnp7uxcpdlq5xf\nlxrQqit2YsqXM/l6zlw6dWxL/17f56pbnlgwf9DOfRj5yEsLrdNtjS5M+GgSALsMWJ/3Pp60iEdS\nuepSA5r+1Re0adOOlq1acf+dI+jVbws6dFyGY045h2NOOQeAcWPHcMtNVzea8IEcA0hSC2AX4BFJ\nfYDDgM1Ij4R+QdJTwPeB/0TErtk6C/22RcRpko4t1KhK3AHsDzwsqRXpSa9HA0OBLyOin6TWwLOS\nRkfEQr2Eko4AjgCgZYf6Oux69as/jOLuK45c7O30XOt7XHLiXgSBEFeMeJw3ii4B2GfH3ux53LUL\nrXP0/luz3WbrMHfefL74ahbDzhpRulmrRx+89w7nnXwUzZo3Z63uPTnzkqvzLlK9UMSSPXtR1AcE\nqQZ0EikYOkfE2dkyFwCTgEeAUaRa0EMRMSab/yRwckS8VKjxFG2/UANqA7wLdAd2BgZlNaS7SbWu\nQr/TssCRETG6ujI3a7ditO45qH5+AFZvnr7nwryLYNXYtFuncRHRt7blcu0DKpCqOO8LRMQ7We3o\nh8DFWU3l/HJ2EhFfZ0E1kFQTur2wO+C4iBi1qAdgZvVjabkO6GlgT0ntsn6fvYAxklYBZkXELcBv\ngN5VrDtXUstqtnsHqWk3gFSTIvv36MI6knpk+zSzJSzvTmgAImJ81kn8Yjbppoh4WdJA4FJJ3wJz\nSU21UjcAr0oaHxEHlcwbDYwAHoiIbwrbBroC47Oa1yRgz3o9IDMryxLvA2qM3Ae0dHIf0NKr3D6g\npaUJZmYVyAFkZrlxAJlZbhxAZpYbB5CZ5cYBZGa5cQCZWW4cQGaWGweQmeXGAWRmuXEAmVluHEBm\nlhsHkJnlxgFkZrlxAJlZbhxAZpYbB5CZ5cYBZGa5cQCZWW4cQGaWGweQmeXGAWRmuXEAmVluHEBm\nlhsHkJnlxgFkZrlxAJlZbhxAZpYbB5CZ5cYBZGa5cQCZWW4cQGaWGweQmeXGAWRmuXEAmVluHEBm\nlhsHkJnlxgFkZrlxAJlZbhxAZpYbB5CZ5cYBZGa5cQCZWW4cQGaWGweQmeVGEZF3GZZ6kiYBE/Mu\nRz1ZAZicdyGsSk3ps1kzIrrUtpADqMJIeiki+uZdDvuuSvxs3AQzs9w4gMwsNw6gynND3gWwalXc\nZ+M+IDPLjWtAZpYbB5CZ5cYBZGa5cQCZWW4cQGaNgCTlXYaG0CLvAtjSSZIiIiStTDpb+p+8y1Sp\nCp9F9v8dgGWAF4DPImJ+roVbTK4BWZWy8NkTuB24VtKvJK2Wd7kqUVH4HA+cB2wGPAFsmme56oMD\nyKokaUPgRGA34EVgO+DLXAtVwST1ALaJiC2BD4GPSLWgwvxG2URzAFl15gMPAfsBuwIHRMR0Sevn\nW6zKI6kz8B/gVUnDgT2BXSLiW0lDJC0bjfSKYgeQLUTSepL2Bb4BBgA/BQZHxPuSdgFulPS9XAtZ\nQSRtDvwfMA/4HtAdGBoR8yQdDJwEdMyxiIvFt2LYQiQNAw6NiC0l/ZzUz/AEMAs4Azg1Ih7Ks4xN\nVdaMUkR8WzRtLeBx4CekZtevgWlAc2AT4KCIeD2H4tYLB1CFKzrb1bxwRkXSbcDzEfE7ST8B1gSW\nB+6PiNHFZ2Ws/pSc7eoMzImIGZL2AbaLiGMlrU2qCa0E/CMiGvVAeT4NX6GyTs2NI+IuSX2BbSRN\niIj7gJuBnQAi4qZs+ZYRMTeb5vCpR1nNZ0PgLGA/SX2A04APJd0MjAX2kLR2RLwLvJtfaeuX+4Aq\nVzPgc0kdgY+B1sAxkq4G5gI/lHRI0fLzcihjRYjkVeBYSdsCr5DC6HPgXlJfXDfgt5Ja5VbQBuAA\nqlAR8TbwLCl89oyIi4DdSX0LmwOdgCGSOmTLu9bTACS1LXo7GTgMeB34ICIuBY4nNX/nAOsC7ZZ4\nIRuQ+4AqiKR2wI4Rcb+kzUhnugQ8AlwYEVdKakbqY9gPeC8i/ppfiZs2SW1IZ7EeJp3d2jAizs6a\nXf2BXhExR1ILoD3QOSLez6/E9c8BVGGy60j6Al8DwyLiZUm9gceAMyPimpLl3eHcACStEBGTJQ0A\nngLeIwXQnGz+H0lnuTaPiK9zLGqDchOsQhRdKXsxqUo/LyJeBoiI8cAOwJXZ5f4LOHzql5LVgV9m\nzds3gfuBlUl/GACIiMOAN4CncynoEuIaUAUoOtXeDOgALEc60zU3InYuWm5toGtEPJpTUSuGpGWA\nDYD2EfGopB8A9wEHRsRDkjaPiLGSVoyIz/MtbcNxDaiJKwqfnYAzSbdUTIyI7YFWkh6UtJmkp4Ap\n2S9Do7yvaGlX/HONiK+AjYGzJe0cEU8ABwN3SfotcLOk1Zpy+ICvA2rysvDZGfgtcCxwu6SNgbMi\n4geSbifdYf3biJhaWCe/EjdNJRcZHgh8GRHXSpoLnJLNf0DSjsA2pDOT/86zzEuCm2BNWNbk6gj8\niXRdyUrApcAnwBfAcRExTVKniPjCHc4NT9IxpNsqBmUXFRYC6XDgqiyEKuZzcA2oCSr6AreJiC8l\nDSV1PJ9P6uhsB3wK/FvSeRHxBbjm05Cy5ld3YDBpdIHPJO0FrA7cArQEhkp6PCJm5lfSJcsB1MQU\n9flsBlwj6dCIeE3SiqTrfpYjhdEo4J6ImJ1neZuy4ppM9u+7WV/bHcDbpM/hS2D5iDhX0v2VFD7g\nAGpysvDZEdibdFXzKEkDsxB6EbgV6AocGxH/yLGoTVpJn88WpOB/BbiTdC/XExExQdJRwEbZahU3\n4Jv7gJqYbPiGh4HDstO4ZwOHkqr9E0hNsHkR8WJ+pWy6SvtvJJ0MHABMAqYAzwC3ZoO7DQWOJg1/\n0miH1FgcrgE1PVOAl0jDdhIR52fX94wCtoyI53IsWyVoQbqZl2zgtoHAgIiYnQ2rMQBYX9Ik0pXO\nh1Vq+ICvA2r0CteWSFpWaWjOr0hnvvYuWmw48G/g/sLNpVb/sqbvCEmnZZc+TCFd+Lk1QETcQxp1\nYI+ImACcFBGv5VbgpYBrQI1c1ufzI9IA8tMkjSUN4Xm70lMsZpEGlh8KHEe6qXFGXuVtqrLAOR/4\nM7Ai8GPSyIW3AZtKmpY1e8ccXTz9AAAFIUlEQVQBPZQGgJuTW4GXEq4BNULFV9QqjRl8OnAI6ekV\nwyLiLWAQqdbTgXTdyUrAFsC339mgLRZJy5P63S6IiN8BNwBtgM6kkQYEXC7pBtJAY3+KRv48r/ri\nTuhGRlIX0lMRbs+G69yaNHZPa1It6MCI+EBS14j4MFtnC2AE6eraiu1vaEiSdiWN19w/Ir6SdCvw\nVETcIGk5YC3S2cdx0ciHUa1PboI1PluSHkzXOhtaoznpDvcppEe1fJH1RRyVneKdAkwEtvcXv+FE\nxF8lfQuMkzQKaEu6wJCImEZqjo3PsYhLJdeAGomsz2C+pOakGtC2wJvZ/UQXkDqd9yVdU3I28AsP\nJrbkKT06eTTwvYj4XFKbpjyez+JyADUCknqS+nFGA09no+TtAuxCCqHrJJ1LGlOmE3BzRIyqpHuK\nlibZZ/Mb0pMsmvTd7IvLAdQISNoG+DvpCtqRwPdJN5XuCLQiPTVzeHZGzH9xlwKS9gDOIV34Gf5D\nUDUHUCMhaSvSo5I3A/YhXdq/F+lMV3fgXNIgY0TRg+0sP5I6RIQveaiBO6EbiYh4RtKPgbuBLbJL\n+R8iPU/qCNJTFBw8SxGHT+1cA2pkJP0Q+B3QrzCAWNEd8O7zsUbFNaBGJiIezk73vi2pZ0RMKxny\nwazRcA2okcoufJsZEU/mXRazReUAauTc7LLGzAFkZrnxzahmlhsHkJnlxgFkZrlxAFnZJM2X9Iqk\n1yXdJandYmxr2+xCSiTtLum0GpbtJOmni7CPc7MxmcuaXrLMcEn71mFfXSV5qJM6cgBZXcyOiF4R\nsQHpET9HFc9UUufvVEQ8EBGX1LBIJ6DOAWRLPweQLaoxQPfsL/9bkq4hjXezuqSdJD0vaXxWU+oA\nadhSSW9LeoaiMaslHSrp6uz/K0m6V9I/s9cWwCVAt6z2dWm23CmS/iHpVUnnFW3rDEn/kvQY0LO2\ng5A0LNvOPyXdU1Kr20HSGEnvSNotW765pEuL9n3k4v4gK5kDyOpMUgvSUCCFAdV7AiMiYhNgJnAm\nsENE9CY9oeNESW2AG4EfkZ4M8b1qNn8VaSTBjYHewBukYUwnZLWvUyTtBKwNbAr0AvpI2lpSH9Ij\ncDYhBVy/Mg7nLxHRL9vfW6Sxswu6kp7TvitwXXYMQ0nPde+XbX+Y0qOQbBH4Vgyri7aSXsn+Pwb4\nA7AKMDEixmbTNwfWA57Nhq5uBTwPrEO6YbbwPPRbSDfRlvoB6fHFZOMmf5kNaVpsp+z1cva+AymQ\nOgL3RsSsbB8PlHFMG0j6JamZ14H0+KKCkdkNvu9Kej87hp2AjYr6h5bN9v1OGfuyEg4gq4vZEdGr\neEIWMsWPExbwaET8uGS5XkB9XfUq4OKIuL5kHz9fhH0MJ42V/U9Jh5JGmiwo3VZk+z4uIoqDCkld\n67hfw00wq39jgS0ldQeQ1E5SD9Kz0NeS1C1b7sfVrP846Wmhhf6WZYDppNpNwSjg8KK+pVUlrQg8\nDewlqa2kjqTmXm06Ap9KagkcVDJvP0nNsjJ/H/hXtu+js+WR1ENS+zL2Y1VwDcjqVURMymoSt0tq\nnU0+MyLekXQE8FdJk0mPKN6gik0cD9yg9Nji+cDREfG8pGez09x/y/qB1gWez2pgM4CDI2K8pDtJ\nz2CfSGom1uYs4IVs+ddYOOj+BTxFeqTRURHxtaSbSH1D45V2Pok0RrctAt8LZma5cRPMzHLjADKz\n3DiAzCw3DiAzy40DyMxy4wAys9w4gMwsN/8PRuPmLIX4QNIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8bb406f7f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds, labels = get_predictions(model_pt_trainable, X_data, y_data)\n",
    "\n",
    "y_trues = [np.argmax(ii) for ii in labels]\n",
    "y_preds = [np.argmax(ii) for ii in preds]\n",
    "\n",
    "cm = confusion_matrix(y_trues, y_preds)\n",
    "fpr, tpr = print_statistics(cm)\n",
    "\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm, classes=['Negative','Positive'], title='ARABIC EMBEDDINGS - TRAINABLE')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TP: 304\n",
      "TN: 19\n",
      "FP: 39\n",
      "FN: 163\n",
      "TPR/recall: 0.65096359743\n",
      "TNR/specificity: 0.327586206897\n",
      "PPV/precision: 0.886297376093\n",
      "NPV/negative predictive value: 0.104395604396\n",
      "FNR/false negative rate: 0.34903640257\n",
      "FPR/false positive rate: 0.672413793103\n",
      "FDR/false discovery rate: 0.113702623907\n",
      "FOR/false ommission rate: 0.895604395604\n",
      "F1 score: 0.750617283951\n",
      "Accuracy: 0.615238095238\n",
      "MCC/Matthews correlation coefficient: -0.0141293095955\n",
      "BM/Bookmaker Informedness: -0.021450195673\n",
      "MK/Markedness: -0.0093070195111\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAASAAAAEmCAYAAADC2k2dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XeYFeX5xvHvDUuvUjTSxAJ2QYog\nWBB778YEe4smllhiD/YWo4k9liix9xb1J7ZgFMSGYkUMKhFRgRWULuX5/TGz5LDsLrtsmd099+e6\n9nLPzJx5n7Mr977zzsw7igjMzLLQIOsCzCx/OYDMLDMOIDPLjAPIzDLjADKzzDiAzCwzDiAzy4wD\nKM9J+krS95Ja5Cw7RtKonNeS9AdJn0uaL+m/kq6U1KSM/Y6StEDSnJyvf6brhkgKSY8Xe0+vdHlu\n2yFpbvr+GZIekNS2Au0szVk+RdLDkvoXaze3jUJJL0v6ZQmf55hi9d9UbJvXJR2R83pNSbdLmpru\n+wtJIyRtUOYvJY84gAygADiljPXXA8cBhwGtgF2BocDDK9nviRHRMudrz5x104FBktrnLDscmFjC\nfnpFREtgHWA14MIKtDM1fW8rYCAwAXhN0valtLE+MAK4UdIFZXy2ucBhkrqXtDL9XGOA5sDWaft9\ngFeBHcvYb15xABnA1cAZuT2LIpJ6AL8FhkXEGxGxOCI+BvYHdpE0dBXb/Bl4Ejg4bachcBBwX2lv\niIifgKeBjSraWCSmRMRw4A7gqlK2mxER9wAnAOcUC8hcs0iCqrSQOhX4CTg0Iial7c+KiLsi4oaK\n1l9fOYAM4B1gFHBGCeu2B6ZExFu5CyPia2AslftrfjdJrwpgZ+BjYGppG0taDdgnbbcyHgf65B52\nluApkp7hFmVscxmwv6T1S1i3A/BERCxd9TLrPweQFRkOnCSpY7HlHYBvS3nPt+n60lwvaVbO1yW5\nKyNiDNAu/Qd8GEkglWScpFnADKAbcGtF2inBVEDACj2+nNoWpe21K2Ob74C/AReXsLoD8F3RC0l7\npbXNlvTCSurLGw4gAyAiPgKeAc4utmoGsGYpb1szXV+akyOibc7XH0vY5h7gRGA74IlS9tMnItoC\nTYFbSMZwmlawnVydgSA5jCqRpEZAR+CHlezrKmBnSb2KLS8k5+cWEU+nn+FUoPFK9pk3HECW6wLg\nWJJ/oEVeAbpKWu5QRFJXkkHdlyvZ5j0kY0zPRcS8sjZMeyV3AGsDm1SizX2BcRExt4xt9gYWA2+V\nsQ0RUQj8FSje63oZ2EeS/42VwT8cWyYi/gM8BJycs2wiyWHGfZIGSmooaWPgMeCliHipkm1+CWwL\nnLeybdOB6iOB+cAXFWknvZSgc3pm6xjg3FK2aydpGHATcFUaMCtzLTAI2LDYstWAeyStm7bfCuhd\nkbrrOweQFXcxUHxw9kSSnse9wBzgeZJB6/1Xsq8bi12f825JG0XE6xFR6uAzMF7SHGAmyan6fSMi\n99CorHY6pe+dA7wNbAoMiYji4zBFbfyHJKBOTc+YrVR6du5P5IwXRcQMkh7iAuB1YDbwPsnp+BPK\ns998IE9IZmZZcQ/IzDLjADKzzDiAzCwzDiAzy0xB1gXUBR06dIi11uqedRlWzBKfQKm1xr83bkZE\nFL+qfgUOoHJYa63ujH7znazLsGJ+mr8o6xKsFGu0bjy5PNv5EMzMMuMAMrPMOIDMLDMOIDPLjAPI\nzDLjADKzzDiAzCwzDiAzy4wDyMwy4wAys8w4gMwsMw4gM8uMA8jMMuMAMrPMOIDMLDMOIDPLjAPI\nzDLjADKzzDiAzCwzDiAzy4wDyMwy4wAys8w4gMwsMw4gM8uMA8jMMuMAMrPMOIDMLDMOIDPLjAPI\nzDLjADKzzDiAzCwzDiAzy4wDyMwy4wAys8w4gMwsMw4gM8uMA8jMMuMAMrPMOIDMLDMOIDPLjAPI\nzDLjADKzzDiAarnCwkIG9O3NgL696d7lF6yzVudlr5s1Emf94fRl2/7l2j9z6cUXVrrNC/54Huut\n3ZUObVsut3zy5MnsutP29N98M3bafghTpkypdFt13ZptmzJ0cL9lX/+d/BWjX3uV9bp0YPut+rNV\nv0358xWXVLqdKy+5gCFb9mHo4H4ctPdufPftVABmzZzJEb8+gCFb9mHnIYP49JOPKt1WTXIA1XLt\n27fnzXff58133+eY447npFNOXfa6SZMmPPXk48yYMaNK29xt9z15bcxbKyw/56wzGHbIYbz93gec\ne/5whp93TpW2Wxc1bdaMV0a/s+yr21rdARiw5Va8/PrbjHx1LI8+9ADj3xtXqXZ+d8rpjHpjHK+M\nfocdd9mNa666DIDrrrmKTTbtxag3xnHjbXdy/lmnr2RPtYsDqA4rKCjg6GOO44br/lKl+x0wcCBr\nrrnmCssnfPoJQ4ZuD8C2Q7bjmX8+VaXt1kctWrRgs96b89WXkyq1n1atWy/7ft68uUgCYOKET9l6\nyFAAevTcgK8nT2batO8r1VZNKsi6AKuc35zwO/r32YzTzjiz1G1eHfUvzjz91BWWN2venFGvjSl3\nW5tu1osnH3+ME08+haeefILZs2dTWFhI+/btV6n2+mDB/PkMHdwPgG5rdWfE/Y8ut/6HwkLGvfMW\np5117nLL58yezV67bFfiPm/5+92sv8FGKyy//OI/8sgD99GqdWsef/ZFADbadFOeffpJBmw5mHHv\nvM2Uryfz7TffsPrqa1TFx6t21RZAkgK4NiJOT1+fAbSMiAuruJ1zI+LynNdjImJQVbZRm7Vu3Zph\nhxzGzTdeT9NmzUrcZtsh2/Hmu+9Xuq0rrvozp55yIvfePYLBW29Dp86dKSjI779hRYdgxb35xuts\nv1V/GjRowEmn/oENNtx4ufUtW7Uq8X1lOXf4JZw7/BKuu+Yq7rz1Zs487wJOPvVMzj/rNIYO7seG\nG23Cppv1pqCgYaU+U02qzv97FgL7SboiIqp2kGJ55wLLAiifwqfIiSf/ni236MNhhx9Z4vqq6gF1\n6tSJhx55HIA5c+bw5BOP0aZNm1Urup4bsOVW3PfIk6WuX5UeUJH9DjyYYQfuzZnnXUCr1q257pY7\nAIgI+m/ak25rrV254mtQdQbQYuA24FTgvNwVkjoCfwO6pYt+HxGj0+X3A+2Bt4FdgL4RMUPSk0BX\noClwXUTcJulKoJmk94GPI2KYpDkR0VLSQ8A/IuK5tM0RwD+BJ4ErgSFAE+CmiLi12n4KNaBdu3bs\nf8BBjLjr7xx+xFErrK+qHtCMGTNo164dDRo04OqrriixLSufivaAvvjP56yzXg8ARj73DD16rg/A\nj7Nm0ax5cxo3bsy9/7iTgYO2Wm68qLar7kHom4Bhkor/mbwO+EtE9Af2B+5Il18AvBIRfYAn+F9A\nARwVEX2BfsDJktpHxNnA/IjoHRHDirXxIPBLAEmNge2B54CjgR/TtvsDx0qqO38ySnHKqadTWEVn\nw849+0zW7d6FefPmsW73LstO7f/71VFstvH6bLpRT6Z9/z1nnXNe2TuyKnPpheexzYDeDNmyD6Ne\neZFLr7oWgImfTWCbLXozuO8mvPLi88uW1xWKiOrZ8f96IhcDi4D5pGNAkqYBU3M27whsALwG7BsR\nX6b7+AHomfaALgT2TbfvDuwcEWOL2imh3abA58B6JD2pg9Ie0qPAZsC89C1tgN9ExAvF6j8OOA6g\na7dufSdOmlxFPxmrKj/NX5R1CVaKNVo3fjci+q1su5oYQfwrMA64K2dZA2DLiJifu6GKzi0WI2kI\nsEP6nnmSRpEcipUqIhak2+1M0hN6oGh3wEkRMXIl77+N5BCSvn37VU9Km+W5ar8OKCJ+AB4mOfQp\n8gJwYtELSb3Tb18HDkqX7QSsli5vA8xMw2cDYGDOvhZJalRK8w8CRwJbA0WBMxI4oeg9knpKarGK\nH8/MKqGmLkS8BuiQ8/pkoJ+kDyR9AhyfLr8I2EnSOGBX4FtgNvA8UCDpA+ASYGzOvm4DPpB0Xwnt\nvgBsA7wUET+ny+4APgHGSfoIuBVfD2WWiWobA1oVkpoASyJisaQtgVsiovfK3lfd+vbtF6PfrNg1\nG1b9PAZUe9WmMaCK6AY8LKkB8DNwbMb1mFk1qlUBFBGfA5tnXYeZ1QzfjGpmmXEAmVlmHEBmlhkH\nkJllxgFkZplxAJlZZhxAZpYZB5CZZcYBZGaZcQCZWWYcQGaWGQeQmWXGAWRmmXEAmVlmHEBmlhkH\nkJllxgFkZplxAJlZZhxAZpYZB5CZZabUSekllfmE+4j4qerLMbN8UtZTMT4GguRRxkWKXgfJI3TM\nzFZZqQEUEV1rshAzyz/lGgOSdLCkc9Pvu0jqW71lmVk+WGkASboR2A44NF00D/hbdRZlZvmhPE9G\nHRQRfSS9BxARP0hqXM11mVkeKM8h2KL0We0BIKk9sLRaqzKzvFCeALoJeAzoKOki4HXgqmqtyszy\nwkoPwSLibknvAjukiw6MiI+qtywzywflGQMCaAgsIjkM89XTZlYlynMW7DzgAaAT0AW4X9I51V2Y\nmdV/5ekBHQL0jYh5AJIuA94FrqjOwsys/ivP4dRklg+qAuCL6inHzPJJWTej/oVkzGce8LGkkenr\nnUjOhJmZVUpZh2BFZ7o+Bp7NWT62+soxs3xS1s2of6/JQsws/6x0EFrSusBlwEZA06LlEdGzGusy\nszxQnkHoEcBdJPMA7Qo8DDxYjTWZWZ4oTwA1j4iRABExKSLOJ7k73sysUspzHdBCSQImSToe+AZY\nvXrLMrN8UJ4AOhVoCZxMMhbUBjiqOosys/xQnptR30y/nc3/JiUzM6u0si5EfIJ0DqCSRMR+1VKR\nmeWNsnpAN9ZYFbXc17MWcNpTn2RdhhVz16U3Z12CVVJZFyK+XJOFmFn+8dw+ZpYZB5CZZabcASSp\nSXUWYmb5pzwzIm4h6UPg8/R1L0k3VHtlZlbvlacHdD2wB1AIEBHj8a0YZlYFyhNADSJicrFlS6qj\nGDPLL+W5FeNrSVsAIakhcBIwsXrLMrN8UJ4e0AnAaUA34HtgYLrMzKxSynMv2DTg4BqoxczyTHlm\nRLydEu4Ji4jjqqUiM8sb5RkDeinn+6bAvsDX1VOOmeWT8hyCPZT7WtI9wIvVVpGZ5Y1VuRVjbWCt\nqi7EzPJPecaAZvK/MaAGwA/A2dVZlJnlhzIDKJ0LuhfJPNAASyOi1EnKzMwqosxDsDRsnoiIJemX\nw8fMqkx5xoDektSn2isxs7xT1pzQBRGxGNgKOFbSJGAuyQMKIyIcSmZWKWWNAb0F9AH2qaFazCzP\nlBVAguRpqDVUi5nlmbICqKOk00pbGRHXVkM9ZpZHygqghiRPRFUN1WJmeaasAPo2Ii6usUrMLO+U\ndRrePR8zq1ZlBdD2NVaFmeWlUgMoIn6oyULMLP/4wYRmlhkHkJllxgFkZplxAJlZZhxAZpYZB5CZ\nZcYBZGaZcQCZWWYcQGaWGQeQmWXGAWRmmXEAmVlmHEBmlhkHkJllxgFkZplxAJlZZhxAZpaZsial\ntwzduN+GfPPjwmWvb33ja9o3b8Sp23bnljH/5cNv5wBwwqCuvDSxkM9nzKtUe78b3I212zVjUuE8\nbhnz9XLr9tq4I5t3bk0E/PuLmYya9AObrdmSPTdenaUBSyN4dPx3TCqcX6ka6oJ2bVrw3K0nAbBG\n+9YsXbqU6TOT30Wv9bsw/rMpFDRswGdffscxw+9h/oJFq9zWHkM2ZfgJe7A0gsVLlnLm1Y8y5v0v\nABi25wDOPmZnAK68YyT3/fPN5d77yF9/w9qd29PvwMtXuf2a4ACqpX5eElzx8hfLLWvfvBEz5y1i\nl/U7LAugqvLSxEIaF4it1l5tueUD12rDas0acfELkwigZZOGAHw2bS4ffJvU17l1E44e2IWLX6j/\nz7D84ce5DDz4SgDO+81uzJ23kL/e8zIA00dfs2zdXZcdzrEHbM31976yym39683PeGbUhwBs0qMT\n9151FL33u5TVWjfnvON2ZfCwPxERjLn/LJ4d9QGzZid/APYe2ou58xaWtetaw4dgdcyUHxcwf/FS\nNli9RZXu97Ppc1mwaOkKy7dZpx3PfTqdSF/PWbgEgIVLYtk2jQsaQKzw1rw2+r1JrNO1Q6X2MXf+\nz8u+b9GsCZH+jHcctCEvj53AzJ/mMWv2fF4eO4GdBm+UbteYkw8ZypV3PF+ptmuKe0C1VOOG4pzt\n1wGgcO7P3DZ2yrJ1//fpDPbauCMTps0t9f079GxP/65tVlj+nxlzeWT89+Wuo0OLRvTt0oZenVsx\nZ+ESHh7/HdPnJP8wenVqxd6brE6rJgXcPPq/5d5nfdewYQN2GrwRL475ZIV191x5JD26r7HC8uvv\nfYX7n3lrheV7bbcZF5+0Fx3btWK/k/8GQKeObZny/cxl23wzbRadOrYF4ILf7sF197zMvJzwqs1q\nPIAkLQE+TNv+FDg8Iio0gCHpDuDaiPhE0rkRcXnOujERMahKi85ASYdgRSYVJj+udds3L/X9L00s\n5KWJhZWuo6BhAxYtXcpVr3xJ706tOLRvJ6599SsAxk+dzfips1mvQ3P23Lgj17+W3yHUrEkjxj54\nNgBj3pvEiCfeWGGbQ8++q0L7fPpfH/D0vz5gcJ91Gf7b3dn9+BtRCU/sC4LNenZmna4dOfOax+m2\nZrtV+gw1LYse0PyI6A0g6T7geKBCz5mPiGNyXp4LXJ6zrs6HT3k8P2EGu2zQgaVR8rFPVfWAZs1f\nxHvfzAbg/amzObRfpxL2OY8OLRrTonFD5v68pNz7rm/mL1y0bAyoNBXtARUZPW4S63TpQPu2Lfhm\n2iy27ttj2brOq7fltXc/Z0CvtemzUTcmPHsRBQ0b0LFdK0befgo7H3vdqn+oapb1IdhrwGYAkk4D\njkqX3xERf5XUAngY6ELyrPpLIuIhSaOAM4ADgGaS3gc+johhkuZEREtJDwH/iIjn0v2PAP4JPAlc\nCQwBmgA3RcStNfJpq9Cn0+ayx8ar07Zpyb/CquoBjZ86m/U7tuCNybPo0aE502YnXfuOLRoxfW5y\nhqdr26YUNFBeh095VaQHtE7XDnzx9QwAem/QhcaNCiicNZcXx3zKRSfuSdtWzQDYYcsNGH7D08z8\naR63P/I6AN3WbMfj1x9fq8MHMgwgSQXArsDzkvoCRwIDSB4J/aakV4F1gKkRsXv6nuX+pEfE2ZJO\nLOpRFfMg8EvgOUmNSZ70egJwNPBjRPSX1AQYLemFiPiyWH3HAccBtOywZpV97qr0/ITpnDCoW5Xs\n67Rtu7NGq8Y0KWjAZbv24N5xU/n0+7m88NkMjuzfmaE92rFw8VLuHTcVgN6dWzNgrTYsWQqLlizl\n729OWUkLVlH7bt+bX+8xgEWLl7Bg4SIOPetOAGb+NI8rbn+e1+89E4DLb3uemT9V7jKMrChK6cJX\nW4P/GwOCpAd0OkkwtI+I4ek2lwDTgeeBkSS9oGci4rV0/SjgjIh4p6jHk7P/oh5QU+BzYD1gF+Cg\ntIf0KEmvq+g31gb4TUS8UFrNq6+7SRzwp4er5gdgVeauS2/OugQrxYL3b3o3IvqtbLtMx4CKSCUN\nq0FETEx7R7sBV6Q9lYvL00hELEiDameSntADRc0BJ0XEyFX9AGZWNWrLdUD/BvaR1Dwd99kXeE1S\nJ2BeRNwL/BnoU8J7F0lqVMp+HyQ5tNuapCdF+t8Tit4jqWfappnVsKwHoQGIiHHpIHHRaYA7IuI9\nSTsDV0taCiwiOVQr7jbgA0njImJYsXUvAHcDT0dE0YURdwDdgXFpz2s6sE+VfiAzK5caHwOqizwG\nVDt5DKj2Ku8YUG05BDOzPOQAMrPMOIDMLDMOIDPLjAPIzDLjADKzzDiAzCwzDiAzy4wDyMwy4wAy\ns8w4gMwsMw4gM8uMA8jMMuMAMrPMOIDMLDMOIDPLjAPIzDLjADKzzDiAzCwzDiAzy4wDyMwy4wAy\ns8w4gMwsMw4gM8uMA8jMMuMAMrPMOIDMLDMOIDPLjAPIzDLjADKzzDiAzCwzDiAzy4wDyMwy4wAy\ns8w4gMwsMw4gM8uMA8jMMuMAMrPMOIDMLDMOIDPLjAPIzDLjADKzzDiAzCwzDiAzy4wDyMwyo4jI\nuoZaT9J0YHLWdVSRDsCMrIuwEtWn381aEdFxZRs5gPKMpHciol/WddiK8vF340MwM8uMA8jMMuMA\nyj+3ZV2AlSrvfjceAzKzzLgHZGaZcQCZWWYcQGaWGQeQmWXGAWRWB0hS1jVUh4KsC7DaSZIiIiSt\nSXK2dGrWNeWrot9F+v0OQGvgTeC7iFiSaXGV5B6QlSgNn32AB4BbJF0lqUvWdeWjnPA5BbgIGAC8\nAmyRZV1VwQFkJZK0KXAasAfwFrAd8GOmReUxST2BbSNiMPAV8F+SXlDR+jp5iOYAstIsAZ4BDgR2\nBw6OiNmSNs62rPwjqT0wFfhA0ghgH2DXiFgq6XBJbaKOXlHsALLlSNpI0gHAz8DWwG+BwyLiC0m7\nArdL+kWmReYRSQOBc4DFwC+A9YCjI2KxpEOA04FWGZZYKb4Vw5Yj6VjgiIgYLOn3JOMMrwDzgPOA\nsyLimSxrrK/SwyhFxNKcZWsDLwPHkBx2/QmYCTQENgeGRcRHGZRbJRxAeS7nbFfDojMqku4H3oiI\nGyQdA6wFtAOeiogXcs/KWNUpdrarPbAwIuZI2h/YLiJOlNSDpCe0BvB2RNTpifJ8Gj5PpYOavSLi\nEUn9gG0lTYqIJ4E7gZ0AIuKOdPtGEbEoXebwqUJpz2dT4I/AgZL6AmcDX0m6ExgL7C2pR0R8Dnye\nXbVVy2NA+asBME1SK+BroAnwO0k3AouA3SQdmrP94gxqzAuR+AA4UdIQ4H2SMJoGPEEyFrcucI2k\nxpkVWg0cQHkqIiYAo0nCZ5+IuBzYi2RsYSDQFjhcUst0e/d6qoGkZjkvZwBHAh8BX0bE1cApJIe/\nC4ENgeY1XmQ18hhQHpHUHNgxIp6SNIDkTJeA54HLIuI6SQ1IxhgOBP4TEc9mV3H9JqkpyVms50jO\nbm0aEcPTw64tgd4RsVBSAdACaB8RX2RXcdVzAOWZ9DqSfsAC4NiIeE9SH+Al4PyIuLnY9h5wrgaS\nOkTEDElbA68C/yEJoIXp+rtIznINjIgFGZZarXwIlidyrpS9gqRLvzgi3gOIiHHADsB16eX+yzh8\nqpYSXYFL08PbT4CngDVJ/jAAEBFHAh8D/86k0BriHlAeyDnV3gBoCaxGcqZrUUTskrNdD6B7RLyY\nUal5Q1JrYBOgRUS8KGko8CTw64h4RtLAiBgrafWImJZttdXHPaB6Lid8dgLOJ7mlYnJEbA80lvRP\nSQMkvQoUpv8Y6uR9RbVd7s81In4CegHDJe0SEa8AhwCPSLoGuFNSl/ocPuDrgOq9NHx2Aa4BTgQe\nkNQL+GNEDJX0AMkd1tdExA9F78mu4vqp2EWGvwZ+jIhbJC0C/pCuf1rSjsC2JGcmp2RZc03wIVg9\nlh5ytQL+QXJdyRrA1cA3wCzgpIiYKaltRMzygHP1k/Q7ktsqDkovKiwKpKOA69MQypvfg3tA9VDO\n/8BNI+JHSUeTDDxfTDLQ2Rz4Fpgi6aKImAXu+VSn9PBrPeAwktkFvpO0L9AVuBdoBBwt6eWImJtd\npTXLAVTP5Iz5DABulnRERHwoaXWS635WIwmjkcBjETE/y3rrs9yeTPrfz9OxtgeBCSS/hx+BdhFx\noaSn8il8wAFU76ThsyOwH8lVzSMl7ZyG0FvAfUB34MSIeDvDUuu1YmM+g0iC/33gIZJ7uV6JiEmS\njgc2S9+WdxO+eQyonkmnb3gOODI9jTscOIKk2z+J5BBscUS8lV2V9Vfx8RtJZwAHA9OBQuB14L50\ncrejgRNIpj+ps1NqVIZ7QPVPIfAOybSdRMTF6fU9I4HBETEmw9ryQQHJzbykE7ftDGwdEfPTaTW2\nBjaWNJ3kSucj8zV8wNcB1XlF15ZIaqNkas6fSM587Zez2QhgCvBU0c2lVvXSQ9+7JZ2dXvpQSHLh\n5zYAEfEYyawDe0fEJOD0iPgws4JrAfeA6rh0zGdPkgnkZ0oaSzKF5wNKnmIxj2Ri+aOBk0huapyT\nVb31VRo4FwP3AKsDvyKZufB+YAtJM9PD3neBnkomgFuYWcG1hHtAdVDuFbVK5gw+FziU5OkVx0bE\np8BBJL2eliTXnawBDAKWrrBDqxRJ7UjG3S6JiBuA24CmQHuSmQYE/EXSbSQTjf0j6vjzvKqKB6Hr\nGEkdSZ6K8EA6Xec2JHP3NCHpBf06Ir6U1D0ivkrfMwi4m+Tq2rwdb6hOknYnma95y4j4SdJ9wKsR\ncZuk1YC1Sc4+vht1fBrVquRDsLpnMMmD6ZqkU2s0JLnDvZDkUS2z0rGI49NTvIXAZGB7/49ffSLi\nWUlLgXcljQSakVxgSETMJDkcG5dhibWSe0B1RDpmsERSQ5Ie0BDgk/R+oktIBp0PILmmZDhwpicT\nq3lKHp38AvCLiJgmqWl9ns+nshxAdYCk9UnGcV4A/p3OkrcrsCtJCP1N0oUkc8q0Be6MiJH5dE9R\nbZL+bv5M8iSLen03e2U5gOoASdsC/yK5gvZhYB2Sm0p3BBqTPDVzRHpGzH9xawFJewMXkFz4Gf5D\nUDIHUB0haSuSRyUPAPYnubR/X5IzXesBF5JMMkbkPNjOsiOpZUT4kocyeBC6joiI1yX9CngUGJRe\nyv8MyfOkjiN5ioKDpxZx+Kyce0B1jKTdgBuA/kUTiOXcAe8xH6tT3AOqYyLiufR07wRJ60fEzGJT\nPpjVGe4B1VHphW9zI2JU1rWYrSoHUB3nwy6ryxxAZpYZ34xqZplxAJlZZhxAZpYZB5CVm6Qlkt6X\n9JGkRyQ1r8S+hqQXUiJpL0lnl7FtW0m/XYU2LkznZC7X8mLbjJB0QAXa6i7JU51UkAPIKmJ+RPSO\niE1IHvFzfO5KJSr8/1REPB0RV5axSVugwgFktZ8DyFbVa8B66V/+TyXdTDLfTVdJO0l6Q9K4tKfU\nEpJpSyVNkPQ6OXNWSzpC0o3p92tIekLS+PRrEHAlsG7a+7o63e4Pkt6W9IGki3L2dZ6kzyS9BKy/\nsg8h6dh0P+MlPVasV7eDpNfWc8saAAACTUlEQVQkTZS0R7p9Q0lX57T9m8r+IPOZA8gqTFIByVQg\nRROqrw/cHRGbA3OB84EdIqIPyRM6TpPUFLgd2JPkyRC/KGX315PMJNgL6AN8TDKN6aS09/UHSTsB\nPYAtgN5AX0nbSOpL8giczUkCrn85Ps7jEdE/be9Tkrmzi3QneU777sDf0s9wNMlz3fun+z9WyaOQ\nbBX4VgyriGaS3k+/fw34O9AJmBwRY9PlA4GNgNHp1NWNgTeADUhumC16Hvq9JDfRFjeU5PHFpPMm\n/5hOaZprp/TrvfR1S5JAagU8ERHz0jaeLsdn2kTSpSSHeS1JHl9U5OH0Bt/PJX2RfoadgM1yxofa\npG1PLEdbVowDyCpifkT0zl2Qhkzu44QFvBgRvyq2XW+gqq56FXBFRNxarI3fr0IbI0jmyh4v6QiS\nmSaLFN9XpG2fFBG5QYWk7hVs1/AhmFW9scBgSesBSGouqSfJs9DXlrRuut2vSnn/yyRPCy0ab2kN\nzCbp3RQZCRyVM7bUWdLqwL+BfSU1k9SK5HBvZVoB30pqBAwrtu5ASQ3SmtcBPkvbPiHdHkk9JbUo\nRztWAveArEpFxPS0J/GApCbp4vMjYqKk44BnJc0geUTxJiXs4hTgNiWPLV4CnBARb0ganZ7m/r90\nHGhD4I20BzYHOCQixkl6iOQZ7JNJDhNX5o/Am+n2H7J80H0GvErySKPjI2KBpDtIxobGKWl8Oskc\n3bYKfC+YmWXGh2BmlhkHkJllxgFkZplxAJlZZhxAZpYZB5CZZcYBZGaZ+X+6Ip7BKxKMxwAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8bb4036470>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds, labels = get_predictions(model_no_embedding, X_data_enc, y_data)\n",
    "\n",
    "y_trues = [np.argmax(ii) for ii in labels]\n",
    "y_preds = [np.argmax(ii) for ii in preds]\n",
    "\n",
    "cm = confusion_matrix(y_trues, y_preds)\n",
    "fpr, tpr = print_statistics(cm)\n",
    "\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm, classes=['Negative','Positive'], title='NO EMBEDDING')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TP: 451\n",
      "TN: 3\n",
      "FP: 55\n",
      "FN: 16\n",
      "TPR/recall: 0.96573875803\n",
      "TNR/specificity: 0.051724137931\n",
      "PPV/precision: 0.891304347826\n",
      "NPV/negative predictive value: 0.157894736842\n",
      "FNR/false negative rate: 0.03426124197\n",
      "FPR/false positive rate: 0.948275862069\n",
      "FDR/false discovery rate: 0.108695652174\n",
      "FOR/false ommission rate: 0.842105263158\n",
      "F1 score: 0.927029804728\n",
      "Accuracy: 0.864761904762\n",
      "MCC/Matthews correlation coefficient: 0.0293114055777\n",
      "BM/Bookmaker Informedness: 0.017462895961\n",
      "MK/Markedness: 0.0491990846682\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVMAAAEmCAYAAADfpHMGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xe8FNX5x/HPF1BAqoIdAXsXBAu2\niIoFY481FlSsiSUqxhpj7yVqYpRoxK7EriGi0R8qKDbsvZeoUVGJNKU8vz/OLC7LvZe9MJdl7/2+\nX6993bszZ+Y8szP77JkzTRGBmZnNnWaVDsDMrDFwMjUzy4GTqZlZDpxMzcxy4GRqZpYDJ1Mzsxw4\nmZqZ5SEicn0BHwGTgPHAd8A/gWWKxg8Bzs7+7w5EVnY88F/gQWDLGubZL/t/f2BaVv5/wMvAdkVl\nS+dZeO1RVGY9YBjwPfAt8CxwALB3UflJwPTieZTGkr3vAtwCjAUmZPPariT+AF4FmhUNOxsYUlKu\nTVbXsFo+1361fe5zsJ5mV1dhHX6ZrbO2NZQ7PVu29UqGF6+j8cAHwOE1rKMWs1sntcReOv/Ca6mi\n+H8COpdM91JWb/eibfGnbNofgBeATetZz6Rs2u+Bp4DDStZzoY4fstdrwHlAh5J6RpZ8/v8F2hQN\nOwgYUfRewBHAK8DEbD2NAPYsKrM68DDpe/h9tnzbzsG2sknRsk9g1u9X16zuydn7b4C7gSVrWXcB\n7F4yvC/wWdH7wvyKc0c/4KMa5jkiW8aWJcPLWb8ja1nm4uUpvB6o63NqqJbp9hHRFliStFFcOZvy\nHbPyPYBHgHsk7V9H+aez8h2Bq4DbJXWsaZ5FrzsAJG0APAY8DqwAdAIOB/pHxC2F8kB/4PPieZQG\nIWkRYCRpha0OdAYuA26VtGtJ8aWAPWfzOewK/AhsJWnJ2ZSdW7Orq7AOewJrAycVj5QkYF9S4htQ\nw/RPF31uuwIXSlq7pkDqWid1xP90yfptGxGfF43/ENirqI41gdY1zOfCLMYOwF+BuyU1r0c920dE\nO6AbcD5wAnBdDXW0AxYl/Wj3AUZJalPH8rUAjq5j/BXA74DjSJ/X0sCpwDZFZR4gfZ8WBxYDjiI1\nQOolIp4sWperZ4OLv1+fZMOOyMqsALQFLq5hdgOofZspNQH4Q10FJHUnJfsAdqihyOzWb12OKFnv\n29dVuEF38yNiMnAnsFqZ5b+MiMtJLZ4LJNUZX0RMB24itbJWLDOsi4AbIuKCiPgmkhciYvcypy92\nDOkXa2AW+6SIuA04B7gkSzgFFwJnSGpRx/wGAFeTWht7z0E89VFWXRHxJTCclFSLbUL6gTga2FPS\ngnXMYwzwJrBqLUXyXCcFNwH7Fb0fANxYR4zTgVuBRUjJp14iYlxE3A/sAQyQtEYNZSZHxHOkL30n\nUmKtzUXAoBoaCUhaCfgNqRX6SLbdTYuIkRGxf1amM7As8LeI+Cl7jYqIkfVdtvqKiO+BeynZZiR1\nAzYFDgG2ljS7z/kKYC9JK9RRZj9gNKkVWmuCntv1W44GTaaSFiJtXKPrOendpF/SlWcz/+akDXIK\n8HGZ8WxASvB52BK4K1tRxYaSdn1WKhp2N6lVsH8tsXUl7erckr32q6lcHupTl6QupBbieyWjBpBa\nPndk77erYx7rkj6L52sYl/c6KRgNtJe0arad7AHcXEeMzUmfw4ekvak5EhHPAp+RfmxqK/MDqcVY\naxnSZzUCGFTDuM2BTyNils+zyFjSOrtZ0k5lJK7cSOoE7MKs28x+wPMRcRfpx3V2DYb/AH8jNa5q\nsx8/b8e1Jui81m9dGiqZ3ivpe1Ly2JL0K1sfhd2oRWoZ3yeb/2TSrsQ+EfFVSZlvJH1f9FoVWJi0\nzF/UM57adK5lXl8UjS8I0i7LaZJa1jDNfsArEfEGcBuwem27xTkop657Jf0AfAp8BfyxMCJLgLsB\nt0bEFFIiLG0V9Mk+9/Gk/s+bgHdriGVO10mfkvX7fg1lCq3TLYG3SF/OUoOybWkC8CfgDxExrZ71\nlPqc2rfd+pQ5DThS0qIlwzuT+khnkPRZFt9kSd0idfxtRup/vQT4QtITksrdg5sTV0gaR+oz7Qwc\nWTJ+P1LrkOxvObv65wHbS1q9dISkjUndK0Mj4gXgfeDXJcVmt35ntzzF6/6sugo3VDLdKSI6Ai1J\nneSPS1qiHtMvnf39tpbxo7P5LwzcT82/8J0jomPR601SJ/V0Ul9uHr6pZV5LFo2fISKGAZ+QdnNK\nFX5hyfrkHqe8jW0mkl6XND571dbyKaeunbJ+vr7AKsz8w7AzMJV0wIhsXv1LvvSjs8+9LbAEqa/t\n3BpimdN1Mrpk/S5fQ5mbSF+u/al9F//ibFtqDawDXCSpuK+2nHpKLU3t227ZZSLiNdIB2RNLRo2l\n5POKiC6kddSSdHCKiPgsIo7IYu5GSig1fg5lbjezc1REdADWIn03uxTNfyNSt8Pt2aBbgTUllXYf\nzSQivgb+DJxZw+gBwMMRUfie1ZSgZ7d+Z7c8xeu+zv7bhu4znRYRd5OOiG5cj0l3JrWG3p7N/MeT\n+o72LacVFxETgaeBX9Ujlrr8G/hVDX27u5NadO/UMM2pwCnAQoUBkjYk9fmeJOlLSV8C65P6i+rq\nY51FRKxe1GH+ZOn4+tYVEY+T+qOKDyYMIB1g+CSb/h/AAhQd8CmZx3+Bu4BZOvAbYJ0Uz/tj0m7d\ntqRulrrKRpa8RgG/nNM6sy6NpUkHJmsr05Z0ZHqW9VODPwIH83MDA9LBui6S1ik3roj4FPgLMEtf\nbja+zu2mPiLiVdLZKn8pOm4wgJTkX8q2mWey4eV0Z11EamX3LgyQ1Jr0Pdu0aDs+BughqUcNMeWy\nfuvS0H2mkrQj6VfqzTLKLy7pCNIGdFINfZGziIixwLWkXaJy/B7YX9LxWd8OknpIun0209XkMqA9\ncJ2kJSS1krQXKVken+1qlcY7gnSaVPEv6ABSH9pqpE77nqSNfiFmPqK9QFZH4VWvRFvPuor9CdhS\nUk9JSwNbkPpIC9P3AC6glpZ09jnvDLxey/zzXCelBgKbR8SE2RWUtArpR7+2OOuatr2k7Ugtr5uz\nhFJapqWk3qSDM98B189uvhHxHqlf+qiiYW8D15DOYtlSUuusT3DDoroWlnSGpBUkNVM6IHUg9T9+\nMaduIB332EFSK1LiO4Sft5mepG6AvWe3HWcHtC4hbScFO5EaacXb8aqkH6gaE3Qt61cl36lW9V3Q\n4kBzfTHzOYqF8+r2Lho/hJrPM51Aao0OA7apYZ7F55mOLBnfhXSaz1rUfp7psUXl1wP+BYwj7Wo9\nA+xXMs++FJ33VlMs2fuupH7Hb7NleA7YsWSaAFYoer9+NmwI0Ir0xdq+hrquAu4sqjdKXmfXc93U\np65+JeP/Smpdngi8UMP0S5EOBK7BrOdnfpV9RouVrPfS80zrXCdFZUvnX3itW1v82fAW1H6e6QRS\nF8y5ZOeJlllP4TzTcaQW9m+B5iXbe+E80wmkL/IFpFOLajzfsTR+YBnS8YERRcNESrCvZjF8Qequ\n2Z3USGpDSmgf8fP5wrcBS8/l93uWdRc/n5d5UMmwE0gH0vbM4lughu3xG9IPc19mPc/0oKL3bbPt\n6KPs/UPAJTXEt3u2rC3KXL+l36nIph3BrOeZzrLdF7+UzdTMzOaCLyc1M8uBk6mZWQ6cTM3McuBk\namaWgzk5tabJ6dy5c3Tr1r3SYViJKdN88HR+9erLY76JiNIrtxo1J9MydOvWnVHP1HUZtFXCf8dN\nrnQIVovunVvP9l4ZjY13883McuBkamaWAydTM7McOJmameXAydTMLAdOpmZmOXAyNTPLgZOpmVkO\nnEzNzHLgZGpmlgMnUzOzHDiZmpnlwMnUzCwHTqZmZjlwMjUzy4GTqZlZDpxMzcxy4GRqZpYDJ1Mz\nsxw4mZqZ5cDJ1MwsB06mZmY5cDI1M8uBk6mZWQ6cTM3McuBkamaWAydTM7McOJmameXAydTMLAdO\npmZmOXAyNTPLgZOpmVkOnEzNzHLgZGpmlgMnUzOzHDiZmpnlwMnUzCwHTqZmZjlwMjUzy4GTqZlZ\nDpxMzcxy4GRqZpYDJ9MqMnbsWNbv3ZP1e/eke5clWK7b0jPet15AnHD8cTPKXnbpxZx95ulzXecO\nv9yG9Xr1oFeP1TnyN4cxbdq0uZ5nY7LcYm3o33f9Ga9PP/mYp0c+wZrLLs62m/Vhiw168qcLz5nr\nei674GzWX2O5GfX83yMPAfDpJx+zcpeFZww/+bgj57oumzMtKh2Ala9Tp04888JLAJx95um0aduW\nY44dBEDHtq247967Of6Ek+jcuXNudd5821Dat29PRLDXHrty153/YPc99sxt/tWuVevW/GvEMzMN\n++yTj1m3z0b8/ba7mThhAtv2XZ8tturPmj17zVVdAw87kkOOOGaW4d26LzdLDDbvuWXaSLRo0YKB\nBx3ClZdflut827dvD8DUqVOZ8tNPSMp1/o3dQm3asEaPtfn4ow8qHYo1MLdMG5FDD/8t6/Zai2MH\n/b7WMo+P+D9+f9ysrZvWCy3EiCefqnGa7bfdmuefe5attunPLr/aNbd4G4PJkybRv+/6ACzTtRuD\nbxw60/jvvh3Liy88y1GDTppp+PgffmC37fvVOM8rrhnCiiuvOsvwG667mruH3sqaPXtx6pnn06Hj\nwgB8+slHbLtZH9q2bcegk//IehtsnMeiWT01WDKVFMClEXFc9n4Q0DYiTs+5npMj4tyi909FxIZ5\n1lEt2rdvz9777MdVf76CVq1b11hm076bzegqKNcDw4YzefJk9t9vb0b832Ns0W/LPMJtFGrazQd4\nbvQott2sD83UjMOPGsRKq6w20/i27drVa9d8nwMO5qhBJyGJS847g7NPO5GLrriGxRZfgqdeeoeF\nF+nEqy+N4ZD9dufhUWNo1679XC+b1U9Dtkx/BHaRdF5EfNOA9ZwMzEimTTWRFhxx1O/YYL1e7Dfg\ngBrHz0nLFKBVq1Zst90OPHD/fU6mZSj0mdamvi3TRRdbfMb/e+57IAN/vQsALVu2pGXLlgCs2bMX\nXbsvx4fvvctaa/ee20WwemrIZDoVGAwcA5xSPELSosDVQNds0O8iYlQ2/FagE/AcsA3QOyK+kXQv\nsAzQCrg8IgZLOh9oLekl4PWI2FvS+IhoK+kO4IaIGJbVOQR4ALgXOB/oC7QE/hIR1zTYpzCPLbLI\nIvxq190Zcv11DNj/wFnG16dlOn78eH744QeWXHJJpk6dykMPDWOjjTbJO+Qmqb4t06++/ILFllgS\ngOH/vG9GS3fsN1/TceFFaN68OZ989CEfffAeXbsv2yAxW90aus/0L8Arki4sGX45cFlEjJTUFRgO\nrAr8EXgsIs6TtA1wSNE0B0bEt5JaA89JuisiTpR0RET0rKHu24E9gGGSFgS2AA4HBgLjImJdSS2B\nUZIejogP81zwSjr6mOO4+qo/z/V8JkyYwK4778BPP/7ItOnT2LTv5hx86GE5RGj1dd4Zp/DGa68g\niS7LdOPcS64E4NmnR3Lp+WfRvEULmjdrzjkXX0nHhRepcLRNkyKiYWb8cwvxTGAKMImsz1TSV8Dn\nRcUXBVYBngR2LiQ2Sd8CK2Ut09OBnbPy3YGtI2J0oZ4a6m0FvAusQGrh7p61XO8E1gImZpN0AA6N\niIdL4j+ELJkv07Vr73fe/zinT8by8t9xkysdgtWie+fWL0TEOpWOY16aF0fz/wSMAa4vGtYM2CAi\nJhUXVC3n3UjqC/TLppkoaQRpd79WETE5K7c1qYV6W2F2wJERMXw20w8mdVPQu/c6DfOLY2aNRoOf\nZxoR3wJDSbvXBQ8DRxTeSCrspo8Eds+GbQUsnA3vAHyXJdJVgD5F85oiaYFaqr8dOADYhNSVQPb3\n8MI0klaS1GYOF8/MDJh3J+1fAhRflnMUsI6kVyS9ARQ64s4AtpI0BugPfAH8ADwEtJD0CnAWMLpo\nXoNJ/bK31FDvw8AvgH9HxE/ZsGuBN4Axkl4DrsHn25rZXGqwPtM5kR0QmhYRUyVtAPy1loNL81Tv\n3uvEqGeer3QYVsJ9pvMv95lWXldgqKRmwE/AwRWOx8ysLPNVMo2Id4G1Kx2HmVl9+UYnZmY5cDI1\nM8uBk6mZWQ6cTM3McuBkamaWAydTM7McOJmameXAydTMLAdOpmZmOXAyNTPLgZOpmVkOnEzNzHLg\nZGpmlgMnUzOzHDiZmpnlwMnUzCwHTqZmZjlwMjUzy4GTqZlZDpxMzcxyUOsD9SS1r2vCiPhf/uGY\nmVWnup5O+joQgIqGFd4H6bHMZmZGHck0IpaZl4GYmVWzsvpMJe0p6eTs/y6SejdsWGZm1WW2yVTS\nn4HNgH2zQROBqxsyKDOzalNXn2nBhhHRS9KLABHxraQFGzguM7OqUs5u/hRJzUgHnZDUCZjeoFGZ\nmVWZcpLpX4C7gEUlnQGMBC5o0KjMzKrMbHfzI+JGSS8A/bJBu0XEaw0blplZdSmnzxSgOTCFtKvv\nq6bMzEqUczT/FOA2YCmgC3CrpJMaOjAzs2pSTst0H6B3REwEkHQO8AJwXkMGZmZWTcrZZf+YmZNu\nC+CDhgnHzKw61XWjk8tIfaQTgdclDc/eb0U6om9mZpm6dvMLR+xfB/5ZNHx0w4VjZlad6rrRyXXz\nMhAzs2o22wNQkpYHzgFWA1oVhkfESg0Yl5lZVSnnANQQ4HrSfUz7A0OB2xswJjOzqlNOMl0oIoYD\nRMT7EXEq6S5SZmaWKec80x8lCXhf0mHAf4DFGjYsM7PqUk4yPQZoCxxF6jvtABzYkEGZmVWbcm50\n8kz27w/8fINoMzMrUtdJ+/eQ3cO0JhGxS4NEZGZWhepqmf55nkUxn5sO/DTV98Oe36zSb1ClQzCb\noa6T9h+dl4GYmVUz35vUzCwHTqZmZjkoO5lKatmQgZiZVbNy7rS/nqRXgXez9z0kXdngkZmZVZFy\nWqZXANsBYwEi4mV8OamZ2UzKSabNIuLjkmHTGiIYM7NqVc7lpJ9KWg8ISc2BI4F3GjYsM7PqUk7L\n9HDgWKAr8F+gTzbMzMwy5Vyb/xWw5zyIxcysapVzp/2/UcM1+hFxSINEZGZWhcrpM/130f+tgJ2B\nTxsmHDOz6lTObv4dxe8l3QQ80mARmZlVoTm5nHRZoFvegZiZVbNy+ky/4+c+02bAt8CJDRmUmVm1\nqTOZZs9+6kF67hPA9Iio9YbRZmZNVZ27+VnivCcipmUvJ1IzsxqU02f6rKReDR6JmVkVq+sZUC0i\nYiqwMXCwpPeBCYBIjVYnWDOzTF19ps8CvYCd5lEsZmZVq65kKoCIeH8exWJmVrXqSqaLSjq2tpER\ncWkDxGNmVpXqSqbNgbZkLVQzM6tdXcn0i4g4c55FYmZWxeo6NcotUjOzMtWVTLeYZ1GYmVW5WpNp\nRHw7LwMxM6tmc3LXKDMzK+FkamaWAydTM7McOJmameXAydTMLAdOpmZmOXAyNTPLgZOpmVkOnEzN\nzHLgZGpmlgMnUzOzHDiZmpnlwMnUzCwHTqZmZjlwMjUzy4GTqZlZDpxMzcxyUNcD9Ww+sXCbBVh9\njTVnvL9l6N188vFHbLf1Ftx+5730/+X2AOy+y/Yc+bvj2OQXfeeqvl126M/zzz5Dnw03YujdD8wY\nHhGcdfofuPfuO2nevDkDDz6Mw3575FzVVW0W6dCGYdekZV68U3umT5/O19+NB6DHyl14+e3PaNG8\nGW9/+CUHnXYTkyZPmes6e6/WlcdvHMS+J/6de/79EgDjn7+C1977HIBPv/yO3X53DQCH7fELjvj1\nZizfdVG6bHYCY7+fMNf1W3mcTKtA69atGfnMmJmGffLxRyy9dBcuvuC8Gck0L0cdM4hJEydy/XWD\nZxp+y01D+M9nn/L8y2/QrFkzvv7qq1zrrQbfjptAnz3PB+CUQ7dlwsQf+dNNjwLw9ahLZoy7/pwB\nHLzrJlxx82NzVV+zZuLso3fkkaffnGn4pB+nzKir2NMvfcCwJ17j4WuPnqt6rf68m1/F1lhrLdp3\naM9jjz6S63z7brYFbdu1m2X4dYOv4fcn/4FmzdJms+hii+Vab2My6sX3WW6ZznM9n9/suSn3Pvoy\nX3/7Q1nlX377Mz75wo9vqwS3TKvApEmT2Hj9XgB069adW4bePWPc8SecwtlnnMbmW2xZ6/SXX3ox\n/7jj1lmGb7jRJlx46eVlx/Hhh+9z951DefD+e+ncuTMXXnI5y6+wYj2WpGlo3rwZW220Go889cYs\n4246/wBW7L74LMOvuPkxbn3w2ZmGLbVoB3bYvAfbHHIFvVffe6ZxrRZswchbfs+0qdO4+PpHeGDE\nK/kuhNXbPE+mkqYBr2Z1vwkMiIiJ9ZzHtcClEfGGpJMj4tyicU9FxIa5Bl1hNe3mF2y48SYAPDXy\nyVqnP/rYQRx97KC5juOnH3+kVctWPD7qWe6/925+e+hBPPTo43M938aidcsFGH37iQA89eL7DLnn\n6VnK7Hvi9WXP76Ljf8Wpl9/H9Okxy7iVtj2NL74eR/elO/HQ4KN47b3P+fCzb+Y8eJtrlWiZToqI\nngCSbgEOAy6tzwwi4qCitycD5xaNa1SJtBzHnXASF194Li1a1Lw682qZLrV0F3bYeRcAtt9xZ357\n6MA5C7iRqq0fs1h9Wqa9VuvKjecfAECnjm3ZeuPVmTp1Og+MeIUvvh4HwEf/GcsTz79Lz1W6OJlW\nWKV3858E1gKQdCxwYDb82oj4k6Q2wFCgC9AcOCsi7pA0AhgE7Aq0lvQS8HpE7C1pfES0lXQHcENE\nDMvmPwR4ALgXOB/oC7QE/hIR18yTpW0gW/TbinPO+CNffvF5jePzapn+cvsdeWLEY+w74EBGPvk4\ny6+w0lzPs6mpT8t01e1On/H/4DP24V9PvsYDI16hY7vWTJw8hZ+mTKVTxzZs0HM5Lr3h3w0QrdVH\nxZKppBZAf+AhSb2BA4D1AQHPSHocWA74PCJ+mU3ToXgeEXGipCMKLd0StwN7AMMkLQhsARwODATG\nRcS6kloCoyQ9HBEflsR3CHAIwDLLdM1tuRvKoBNOYq/dds5lXttssSnvvPMWE8aPZ9Xlu3Ll1X+j\n35Zbc8ygEzj4gH246srLadOmLVf+dfDsZ2a5W2W5JbjylL2YHtNppmZcfP0jvPXBlwD8Zq9NOXZA\nPxbv1J7nhp7MQyNf5zdnzrpXYvlTxKz9MQ1a4c99ppBapseRklyniDgtK3MW8DXwEDCc1Dp9MCKe\nzMaPAAZFxPOFlmjR/Ast01bAu8AKwDbA7lnL9U5Sa7jQT9sBODQiHq4t5rV7rxOPj3q2ttFWIYtv\ncFSlQ7BaTH7pLy9ExDqVjmNeqmifaYEk1VQwIt7JWq3bAudlLcgzy6kkIiZnSXdrUgv1tkJ1wJER\nMXxOF8DMrNT8cp7pE8BOkhbK+kl3Bp6UtBQwMSJuBi4GetUw7RRJC9Qy39tJ3QebkFq4ZH8PL0wj\naaWsTjOzOVbpA1AARMSY7ABRYV/62oh4UdLWwEWSpgNTSN0BpQYDr0gaExF7l4x7GLgRuD8ifirM\nG+gOjMlaxF8DO+W6QGbW5MzzPtNq5D7T+ZP7TOdfTbHPdH7ZzTczq2pOpmZmOXAyNTPLgZOpmVkO\nnEzNzHLgZGpmlgMnUzOzHDiZmpnlwMnUzCwHTqZmZjlwMjUzy4GTqZlZDpxMzcxy4GRqZpYDJ1Mz\nsxw4mZqZ5cDJ1MwsB06mZmY5cDI1M8uBk6mZWQ6cTM3McuBkamaWAydTM7McOJmameXAydTMLAdO\npmZmOXAyNTPLgZOpmVkOnEzNzHLgZGpmlgMnUzOzHDiZmpnlwMnUzCwHTqZmZjlwMjUzy4GTqZlZ\nDpxMzcxy4GRqZpYDJ1Mzsxw4mZqZ5cDJ1MwsB06mZmY5cDI1M8uBk6mZWQ6cTM3McqCIqHQM8z1J\nXwMfVzqOnHQGvql0EFajxrRuukXEopUOYl5yMm1iJD0fEetUOg6blddNdfNuvplZDpxMzcxy4GTa\n9AyudABWK6+bKuY+UzOzHLhlamaWAydTM7McOJmameXAydTMLAdOpmZVQJIqHYPVrUWlA7D5kyRF\nREhaknTWx+eVjqmpKqyL7P9+QHvgGeDLiJhW0eBsBrdMrUZZIt0JuA34q6QLJHWpdFxNUVEiPRo4\nA1gfeAxYr5Jx2cycTK1GktYEjgW2A54FNgPGVTSoJkzSSsCmEbER8BHwCal1WhjvboAKczK12kwD\nHgR2A34J7BkRP0havbJhNT2SOgGfA69IGgLsBPSPiOmSBkjqEL76puKcTG0mklaTtCvwE7AJ8Btg\nv4j4QFJ/4G+SlqhokE2IpD7AScBUYAlgBWBgREyVtA9wHNCugiFaxpeT2kwkHQzsHxEbSfodqV/u\nMWAicApwQkQ8WMkYG6tsV10RMb1o2LLAo8BBpF37C4HvgObA2sDeEfFaBcK1Ek6mTVzRUfvmhSPD\nkm4Fno6IKyUdBHQDFgHui4iHi48uW35Kjtp3An6MiPGSfgVsFhFHSFqR1EJdHHguIhrLTcurnk+N\naqKyAxo9IuIfktYBNpX0fkTcC/wd2AogIq7Nyi8QEVOyYU6kOcpapGsCfwB2k9QbOBH4SNLfgdHA\njpJWjIh3gXcrF63Vxn2mTVcz4CtJ7YBPgZbAbyX9GZgCbCtp36LyUysQY5MQySvAEZL6Ai+REutX\nwD2kvuvlgUskLVixQK1OTqZNVES8BYwiJdKdIuJcYAdSX1wfoCMwQFLbrLxbow1AUuuit98ABwCv\nAR9GxEXA0aQulh+BVYGF5nmQVhb3mTYhkhYCtoyI+yStTzpiL+Ah4JyIuFxSM1Kf3G7AexHxz8pF\n3LhJakU6Gj+MdJR+zYg4Ldu13wDoGRE/SmoBtAE6RcQHlYvY6uJk2sRk5ymuA0wGDo6IFyX1Av4N\nnBoRV5WU98GmBiCpc0R8I2kT4HHgPVIy/TEbfz3paH2fiJhcwVCtTN7NbyKKrpA5j7TbODUiXgSI\niDFAP+Dy7JLFGZxI86VkGeDsrAvlDeA+YEnSjxwAEXEA8DrwREUCtXpzy7QJKDr9qRnQFliYdMR+\nSkRsU1RuRaB7RDxSoVCbDEnQp1TBAAAHsklEQVTtgTWANhHxiKTNgXuBX0fEg5L6RMRoSYtFxFeV\njdbK4ZZpI1eUSLcCTiVdFvpxRGwBLCjpAUnrS3ocGJt9sX2ddwMo/lwj4n9AD+A0SdtExGPAPsA/\nJF0C/F1SFyfS6uHzTBu5LJFuA1wCHAHcJqkH8IeI2FzSbaQ7EV0SEd8WpqlcxI1TyQn5vwbGRcRf\nJU0Bjs/G3y9pS2BT0hkWn1UyZqsf7+Y3YtlufTvgBtJ5i4sDFwH/Ab4HjoyI7yR1jIjvfbCp4Un6\nLenS0N2zE/ALyfVA4IosoXo9VCG3TBuhoi9jq4gYJ2kg6aDTmaSDHAsBXwCfSTojIr4Ht0gbUraL\nvwKwH+kuXF9K2hlYBrgZWAAYKOnRiJhQuUhtTjmZNjJFfaTrA1dJ2j8iXpW0GOm80oVJiXU4cFdE\nTKpkvI1ZcQsz+/tu1jd9O/AWaT2MAxaJiNMl3edEWr2cTBuZLJFuCexCupppuKSts4T6LHAL0B04\nIiKeq2CojVpJH+mGpB+xl4A7SNfWPxYR70s6DFgrm8w3365i7jNtZLJbtg0DDshOrTkN2J+0a/k+\naTd/akQ8W7koG6/S/k5Jg4A9ga+BscBI4JbsRtsDgcNJtzz0bfSqnFumjc9Y4HnSoy2IiDOz80eH\nAxtFxFMVjK0paEG6UQzZTbS3BjaJiEnZrfQ2AVaX9DXpCqcDnEgbB59nWuUK5y5K6qD0+Ir/kY7g\n71JUbAjwGXBf4cYllr+se+VGSSdmp6ONJV0k8QuAiLiLdHeuHSPifeC4iHi1YgFbrtwyrXJZH+n2\npIfffSdpNOkxF7cpPU10IumheAOBI0k3zBhfqXgbqyx5ngncBCwG7EW6I/6twHqSvsu6Vl4AVlK6\nGfePFQvYcueWaRUqvpJG6RlBJwP7kp4ienBEvAnsTmqNtiWd17g4sCEwfZYZ2lyRtAipn/qsiLgS\nGAy0AjqR7sgl4DJJg0k3fb4h/Lz7RscHoKqMpEVJT6e8LXukxS9I9x5tSWqd/joiPpTUPSI+yqbZ\nELiRdFWN++cagKRfkp7PtEFE/E/SLcDjETFY0sLAsqSzKF4IP2qkUfJufvXZCFgfaJndTq856U5Q\nY0mP//0+67s7LDvtZizwMbCFv8QNJyL+KWk68IKk4UBr0sn4RMR3pF3+MRUM0RqYW6ZVIutjmyap\nOall2hd4I7u++yzSAaddSecsngb83jd2nvck9QMeBpaIiK8ktfL9SJsGJ9MqIGllUr/nw8AT2d3X\n+wP9SQn1akmnk+6J2RH4e0QM9zXelZGtm4tJTxT1XZ+aCCfTKiBpU+D/SFfODAWWI92wZEtgQeBz\nYEh2ZN8tofmApB2BP5Iukgj/qDV+TqZVQtLGwIOk/tJfkS5P3Jl0xH4F4HTSDZ+JCB+xnw9IahsR\nPg2tifABqCoRESMl7QXcCWyYXY74IOl564eQnmbpJDofcSJtWtwyrTKStgWuBNYt3My56E5R7iM1\nqxC3TKtMRAzLTsF5S9LKEfFdyW3ezKwC3DKtUtlJ4hMiYkSlYzEzJ9Oq5117s/mDk6mZWQ58oxMz\nsxw4mZqZ5cDJ1MwsB06mVjZJ0yS9JOk1Sf+QtNBczKtvdtEBknaQdGIdZTtK+s0c1HF69gymsoaX\nlBkiadd61NVdkm9v2IQ5mVp9TIqInhGxBumx0YcVj1RS720qIu6PiPPrKNIRqHcyNZuXnExtTj0J\nrJC1yN6UdBXpfp3LSNpK0tOSxmQt2LaQHu0h6S1JIyl6RpWk/SX9Oft/cUn3SHo5e20InA8sn7WK\nL8rKHS/pOUmvSDqjaF6nSHpb0r+BlWe3EJIOzubzsqS7Slrb/SQ9KekdSdtl5ZtLuqio7kPn9oO0\nxsHJ1OpNUgvS7f8KD4NbGbgxItYGJgCnAv0iohfpSanHSmoF/A3YnvSEziVqmf0VpDvU9wB6Aa+T\nHvXxftYqPl7SVsCKwHpAT6C3pF9I6k16rPLapGS9bhmLc3dErJvV9ybpWVkF3YFNSY/JvjpbhoHA\nuIhYN5v/wUqP17YmzpeTWn20lvRS9v+TwHXAUsDHETE6G94HWA0YlT2qakHgaWAV0s1Y3gWQdDPp\nBi2lNgf2A8iekzQue+xHsa2y14vZ+7ak5NoOuCciJmZ13F/GMq0h6WxSV0Jb0iOxC4ZmN495V9IH\n2TJsBaxV1J/aIav7nTLqskbMydTqY1JE9CwekCXMCcWDgEciYq+Scj2BvK4QEXBeRFxTUsfv5qCO\nIaRnY70saX/SEwwKSucVWd1HRkRx0kVS93rWa42Md/Mtb6OBjSStACBpIUkrAW8By0paPiu3Vy3T\nPwocnk3bXFJ74AdSq7NgOHBgUV/s0pIWA54AdpbUWlI7UpfC7LQDvpC0ALB3ybjdJDXLYl4OeDur\n+/CsPJJWktSmjHqskXPL1HIVEV9nLbzbJLXMBp8aEe9IOgT4p6RvgJHAGjXM4mhgsKSBwDTg8Ih4\nWtKo7NSjf2X9pqsCT2ct4/HAPhExRtIdwEukhwg+WUbIfwCeycq/ysxJ+23gcdJjsg+LiMmSriX1\npY5Rqvxr0jO5rInztflmZjnwbr6ZWQ6cTM3McuBkamaWAydTM7McOJmameXAydTMLAdOpmZmOfh/\nL7VC5hZ4UP0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8bb405a9e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds, labels = get_predictions(model_bidirectional, X_data, y_data)\n",
    "\n",
    "y_trues = [np.argmax(ii) for ii in labels]\n",
    "y_preds = [np.argmax(ii) for ii in preds]\n",
    "\n",
    "cm = confusion_matrix(y_trues, y_preds)\n",
    "fpr, tpr = print_statistics(cm)\n",
    "\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm, classes=['Negative','Positive'], title='BIDIRECTIONAL - ARABIC EMBEDDINGS - TRAINABLE')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TP: 5\n",
      "TN: 89\n",
      "FP: 11\n",
      "FN: 95\n",
      "TPR/recall: 0.05\n",
      "TNR/specificity: 0.89\n",
      "PPV/precision: 0.3125\n",
      "NPV/negative predictive value: 0.483695652174\n",
      "FNR/false negative rate: 0.95\n",
      "FPR/false positive rate: 0.11\n",
      "FDR/false discovery rate: 0.6875\n",
      "FOR/false ommission rate: 0.516304347826\n",
      "F1 score: 0.0862068965517\n",
      "Accuracy: 0.47\n",
      "MCC/Matthews correlation coefficient: -0.110581467116\n",
      "BM/Bookmaker Informedness: -0.06\n",
      "MK/Markedness: -0.203804347826\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAASAAAAEmCAYAAADC2k2dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XeYFeXdxvHvTW+CCooxGiHBigoC\nioIksYMlltix997Na9QYNbHHxJZEsQR57WIsUQJEfbEjCip2kShWoiAqCCgLv/ePmSWHdXfZZffs\nw+65P9e1l3tm5sz8zq578zzPzDyjiMDMLIVmqQsws9LlADKzZBxAZpaMA8jMknEAmVkyDiAzS8YB\nZGbJOICsWpL2lfS8pG8kfZZ/f5wkFWxzvqSQtFmF954v6bZK9hmSeuTf95Q0VtIsSV9Kmihpx4Jt\nz5b0nqQ5kj6SdHfBunGSjqiw7+6SFkn6SxXHfVVSs4Jlv5c0vJrPP07SfElrFizbVtL7Ba/flzQv\nr3GWpEcKt7eqOYCsSpJOB64GrgBWA7oCxwADgVb5NgIOBL4ADl6Gw/wD+Fe+71WBk4Cv830fnO97\n24joAPQDHlvK/g4CZgH7SmpdyfrVgX1rWeM3wG+Wss0ueY0/AP4DXFvLY5QkB5BVSlIn4ELguIgY\nGRGzI/NSRAyNiG/zTQeR/VGfTPZH36oWx+gCdAdujIjv8q9nIuLpfJNNgTERMRUgIqZHxLCl7PYg\n4FxgAbBLJesvBy6Q1KKmdQLXAPuVt9qqExHzgZHABrXYf8lyAFlVtgBaAw8uZbuDyVox5V2jnWtx\njJnAu8BtknaT1LXC+vHAQZLOlNRPUvPqdiZpELAGcBdwD1kYVfR3shbWIbWo82PgRuD8pW0oqR2w\nT167LYUDyKrSBZgREWXlCyQ9m4/TzJP00/yPbS/gjohYQPYvf427YZHdiLgV8D5wJfCppCclrZ2v\nvw04EdgBeAL4TNJZ1ezyYOCfETELuAMYImnViocl606dV0UXrSqXALtI6lnF+gckfUkWbtuRdVtt\nKRxAVpWZQJfCrkpEDIiIFfN1zYDdgTJgVL7J7WR/9Kvkr8uAloU7lVT+ekG+z48i4oSI+AmwFtl4\ny4iCY94eEdsCK5KNP10oaYeKxUpqSxaGt+fvew74ANi/4rYRMSpfd1SFfVyfDyTPkXR2hfd8DlxH\n1i2tzG75z6Y1cALwhKTVqtjWcg4gq8pzwLfArtVsczDQAfhA0nTgXrLA2S9f/wHQrcJ7ugMLybo1\nS4iID4E/AxtWsm5BRNwLTK5sPVkYdgT+Iml6Xs8PqbwbBtk40TlAu4JjHBMRHfKviyt5zxVkLba+\nVeyTiFgYEX/PP+OWVW1nGQeQVSoivgQuIPuD3lNSB0nNJPUG2pP9cW9DNubTO//qBVzGf7tho4F1\nJR0oqaWklYGLgZERUSZpJUkXSOqR77sLcBj5+ImkQyTtJGmFfP0QoCfwfCUlHwzcAmxUUM9AoLek\njSr5fOOAV6ldl/FLsq7ir6raRpldgZWAN2u675IVEf7yV5VfwFBgAjAX+Jzsj/8o4DxgYiXbr07W\nvdowfz0AeJrs1PgnwM3ASvm69sCtZGNAc4DpwJ3AD/P1ewDP5O/9miwwDik41jjgCLIwLAM2qqSe\nUcAf8u8D6FGwrn++bHg1n38ccETB6w7AZ8D7BcveB+bln2E28BowNPXvrjF8Kf8Bmpk1OHfBzCwZ\nB5CZJeMAMrNkHEBmlkxt7ocpWWrVPtRmxdRlWAUb9/B1fsurV16aNCMiVlnadg6gGlCbFWnd7/jU\nZVgFYx88M3UJVoWuHVtNq8l27oKZWTIOIDNLxgFkZsk4gMwsGQeQmSXjADKzZBxAZpaMA8jMknEA\nmVkyDiAzS8YBZGbJOIDMLBkHkJkl4wAys2QcQGaWjAPIzJJxAJlZMg4gM0vGAWRmyTiAzCwZB5CZ\nJeMAMrNkHEBmlowDyMyScQCZWTIOIDNLxgFkZsk4gMwsGQeQmSXjADKzZBxAZpaMA8jMknEAmVky\nDiAzS8YBZGbJOIDMLBkHkJkl4wAys2QcQGaWjAPIzJJxAJlZMg4gM0umReoCrHord2zLqD/sD0DX\nlduzaFHw+ZdzAejVoytX3/M8Z13/GACn7N2f9m1bcdGtT9XpmHtvvQFn7j+ACPh05mwOu/ghZn49\nj41+vCrXnjqY9m1bMe0/X3HoRQ8ye+53dfuAjdwPVmzD+j03XPx6+B0j+fCDaRy83y9Zq1t3vp0/\nn91+uTdn/Po3dTrOQ/eP5A+X/I533n6L0f/3LL379AXgi5kzOfygfXl50ovsu/9BXHLl1XU6TkNz\nAC3nvvh6HpsfdTMA5xw8iG/mfcdV9zwPwKzRv2LXQetyxR3PMvPrefVyvObNxBXHb0efQ4cx8+t5\nXHTUVhyzez8uuvUp/nrGjpx1/eM8PfkDDhq8MafuszkX/u3JejluY9WmbVsef+bFJZZ9+ME0+m+x\nJbff+wDffPMN2wzclO0G70SvTfos83HW26Ant9x+D2eefPwSy1u3acNZ557PW2+8zltvvL7M+0/F\nXbBGrGzhIm5++CVO3HOzetunJCTRvm1LAFZo15pPZ8wGYO01O/P05A8AeHzie+w2aL16O25T1b59\nezbuvQnvvze1TvtZZ9316bH2upXuv/8WA2ndpk2d9p+KW0CN3A0PTuSFm47gj3ePr3Kbn/Zei8uP\n2/Z7y+d9u4CtThyxxLKyhYs4+arRvHDTkXwzfwFTP/6CU64ZA8Ab73/OzgPW5uFnp7DHz9ZnjVVX\nqN8P0wjNnzePrQf2A+BHa3Vj+B0jl1j/xcyZTHpxAqf9z9lLLJ8zeza/GLxVpfv8680jWHe9DYpT\n8HKmaAEkKYA/RsTp+eszgA4RcX49H+fsiLi44PWzETGgPo+xPJs99ztuH/sax+3ej/nflVW6zZMv\nT1vcjVuaFs2bceQv+rD50Tfz3idf8qeTtufM/Qdw2W3PcPTlj3Dlidvx64O25JFnp/DdgoX1+VEa\npcq6YADPP/c022y5Kc2aNePEU89kvfV7LrG+wworVPq+UlPMFtC3wB6SLomIGUU8ztnA4gAqpfAp\nd919E3juhsMYMXpypetr0wLq1aMrAO998iUAI8e9yRn7bQHAOx/OZJdf3QVAjzVWZsjmPertMzQ1\n5WNAVXELKFPMACoDhgGnAucUrpC0CnA98KN80SkR8Uy+/A6gM/ACMBjoGxEzJD0ArAm0Aa6OiGGS\nLgXaSnoZeD0ihkqaExEdJN0N3BoRo/JjDgf+ATwAXAr8HGgN/DkibijaT6EBzJo9n/vGvckhQ3px\nayUhVJsW0CczZrPeWl3o0qkdM76ayzZ9u/P2tJkArLJiOz7/ci4SnHXAQG58aFK9fo5S4hZQptiD\n0H8GhkrqVGH51cCfImJT4JfATfny3wKPR0Qf4H7+G1AAh0VEX6AfcJKkzhFxFjAvInpHxNAKx7gL\n2AdAUitgG2AUcDjwVX7sTYEjJXWvp8+bzNX3TqBzp3Z13s+nM+dw8Yin+NdVBzDhxiPYuEdXLr/j\nWQD23ronk289mleGH82nM2dX2eKy+jfqHw/Qe73uvDhhPEP32pV9dttp8bp+G67Nb399JnfdMYLe\n63Xn7bfeSFhp7SgiirPj/7ZELgQWAPPIx4AkfQZ8UrD5KsB6wFPA7hHxXr6PL4B18hbQ+cDu+fbd\ngB0iYnz5cSo5bhtgCtCDrCW1d95CGglsDMzN39IJODoixlao/yjgKABad+rbZsCv6ucHY/Vm2oNn\npi7BqtC1Y6uJEdFvads1xFmwq4BJwN8KljUDtoiIJS5ekaTKdiDp58C2+XvmShpH1hWrUkTMz7fb\ngawldGf57oATI2LMUt4/jKwLSbOOPyxOSpuVuKJfBxQRXwD3kHV9yo0FTih/Ial3/u3TwN75su2B\nlfLlnYBZefisB2xesK8FklpWcfi7gEOBQUB54IwBji1/j6R1JLVfxo9nZnXQUBciXgl0KXh9EtBP\n0mRJbwDH5MsvALaXNAkYAnwKzAZGAy0kTQZ+BxRe9DIMmCzp9kqOOxb4KfBoRJTfM3AT8AYwSdJr\nwA34eiizJIo2BrQsJLUGFkZEmaQtgL9GRO+lva/YmnX8YbTud/zSN7QG5TGg5dfyNAZUGz8C7pHU\nDPgOODJxPWZWRMtVAEXEFGCT1HWYWcPwzahmlowDyMyScQCZWTIOIDNLxgFkZsk4gMwsGQeQmSXj\nADKzZBxAZpaMA8jMknEAmVkyDiAzS8YBZGbJOIDMLBkHkJkl4wAys2QcQGaWjAPIzJJxAJlZMg4g\nM0umyknpJXWs7o0R8XX9l2NmpaS6p2K8DgTZo4zLlb8OskfomJktsyoDKCLWbMhCzKz01GgMSNK+\nks7Ov19DUt/ilmVmpWCpASTpOmAr4MB80Vzg+mIWZWaloSZPRh0QEX0kvQQQEV9IalXkusysBNSk\nC7Ygf1Z7AEjqDCwqalVmVhJqEkB/Bu4DVpF0AfA0cFlRqzKzkrDULlhEjJA0Edg2X7RXRLxW3LLM\nrBTUZAwIoDmwgKwb5qunzaxe1OQs2DnAncDqwBrAHZJ+XezCzKzpq0kL6ACgb0TMBZB0ETARuKSY\nhZlZ01eT7tQ0lgyqFsC/i1OOmZWS6m5G/RPZmM9c4HVJY/LX25OdCTMzq5PqumDlZ7peBx4pWD6+\neOWYWSmp7mbUmxuyEDMrPUsdhJb0E+AiYAOgTfnyiFiniHWZWQmoySD0cOBvZPMADQHuAe4qYk1m\nViJqEkDtImIMQERMjYhzye6ONzOrk5pcB/StJAFTJR0DfAysWtyyzKwU1CSATgU6ACeRjQV1Ag4r\nZlFmVhpqcjPq8/m3s/nvpGRmZnVW3YWI95PPAVSZiNijKBWZWcmorgV0XYNVsbwrWwCzPkldhVXQ\nsW3L1CVYHVV3IeJjDVmImZUez+1jZsk4gMwsmRoHkKTWxSzEzEpPTWZE3EzSq8CU/HUvSdcWvTIz\na/Jq0gK6BtgZmAkQEa/gWzHMrB7UJICaRcS0CssWFqMYMystNbkV40NJmwEhqTlwIvBOccsys1JQ\nkxbQscBpwI+A/wCb58vMzOqkJveCfQbs2wC1mFmJqcmMiDdSyT1hEXFUUSoys5JRkzGgRwu+bwPs\nDnxYnHLMrJTUpAt2d+FrSf8L/KtoFZlZyViWWzG6A2vVdyFmVnpqMgY0i/+OATUDvgDOKmZRZlYa\nqg2gfC7oXmTzQAMsiogqJykzM6uNartgedjcHxEL8y+Hj5nVm5qMAU2Q1KfolZhZyaluTugWEVEG\nbAkcKWkq8A3ZAwojIhxKZlYn1Y0BTQD6ALs1UC1mVmKqCyBB9jTUBqrFzEpMdQG0iqTTqloZEX8s\nQj1mVkKqC6DmZE9EVQPVYmYlproA+jQiLmywSsys5FR3Gt4tHzMrquoCaJsGq8LMSlKVARQRXzRk\nIWZWevxgQjNLxgFkZsk4gMwsGQeQmSXjADKzZBxAZpaMA8jMknEAmVkyDiAzS8YBZGbJOIDMLBkH\nkJkl4wAys2QcQGaWjAPIzJJxAJlZMg4gM0umuknpbTkx58VreO3dTxa/3vvUYay1emfG3nQyvzz5\nekY9+RoA9119DFeNeIynJk6p0/F+f9KuDB7UE4BLbxzNyLGTABh2wQEM6tuDr+bMB+Co8/6Xye98\nXKdjNRUzZ85kx+2zWYz/85/pNGvenFW6rALA5MmvsPHGvShbWMa6663PTbfcSrt27Zb5WE8+MY69\n9tiVbt26A7Dr7ntw9rnn1f1DJOAAagTmfbuAzfe9dIlla63emY+mz+J/Dt9hcQDVh8Fb9qT3+mvS\nf99Lad2yBWNvPoUxz7zB7G+y0Dn7qge4/9GX6+14TUXnzp15fmL2c/n9hefTvkMHTj3tDAC6rNhh\n8bpDDhzKjTdcz8mnVvnIvRoZuOUg/v7gw3UrejngLlgjNvmdj/lqzny27r9eve1z/R+vxlMTp7Bw\n4SLmzv+OV9/5iO0HrF9v+y91A7ccxL+nvpu6jOWGW0CNQNvWLRl/11kATPt4JvucfuPidZfdNJrf\nHr8zjz//VpXvP/Wgbdhnx02/t/yZSe9y+uUjl1g2+Z2POefoIVxz2+O0a9OKn/Vbh7f+PX3x+vOP\n34VfHzmEcRPe5txrHuK7BWV1/Xglo6ysjLFj/sl22w/+3roD9t+HKW+//b3lJ51yGkMPPOh7y58f\n/xyb9enFD1ZfnUsu+wMb9OxZlJqLrcEDSNJC4NX82G8CB0fE3Fru4ybgjxHxhqSzI+LignXPRsSA\nei06scq6YOWeeWkqAAM3+UmV7//TiMf404jHanSsx8a/Rd+ea/F/w09nxqw5PD/5PcrKFgFw3rUP\nMX3G17Rq2YI//2Y/Tj90Wy4ZNrqWn6b0zJs3j/59ewMwYMtBHHLY4d/b5rY77q7x/npv0oe3p06j\nQ4cOjP7nKPbeczdee7Nu436ppGgBzYuI3gCSbgeOAWr1nPmIOKLg5dnAxQXrmlT41MRlN43hV4fv\nQNnCRZWur00LCODym8dw+c1jABh+8SG8++FnAEyf8TUA3y0oY8SD4znlID86ribatm27eAyoKrVp\nAXXs2HHx94OH7MjJJx7HjBkz6NKlS/0U3IBSd8GeAjYGkHQacFi+/KaIuEpSe+AeYA2yZ9X/LiLu\nljQOOAPYE2gr6WXg9YgYKmlORHSQdDdwa0SMyvc/HPgH8ABwKfBzoDXw54i4oUE+bZE8Nv4tfnvc\nzvxglU6Vrq9NC6hZM7HiCu344qtv2HDt1dlw7dV59Lmse7dal46LQ+gXW23MG1M/qW5XVgu1aQFN\nnz6drl27IokXJkxg0aJFdO7cuYjVFU+yAJLUAhgCjJbUFzgU6E/2SOjnJT0B/Bj4JCJ2yt+zxF9Y\nRJwl6YTyFlUFdwH7AKMktSJ70uuxwOHAVxGxqaTWwDOSxkbEexXqOwo4CoCWHerrYxfNZTePYeRV\nR9d5Py1bNOfRW04BYPac+Rx2zq0szFtWf7voYLqstAISTH77I0686K46H89q7/77RnLjsL/SonkL\n2rRty4jb7kJqnE9SV0Q07AH/OwYEWQvodLJg6BwR5+Xb/A74HBgNjCFrBT0cEU/l68cBZ0TEi+Ut\nnoL9l7eA2gBTgB7AYGDvvIU0kqzVVT7u1Ak4OiLGVlVzs3arRut1966fH4DVm1kvXJe6BKtC25aa\nGBH9lrZd0jGgcqoiviPinbx1tCNwSd5SubAmB4mI+XlQ7UDWErqz/HDAiRExZlk/gJnVj+XlOqAn\ngd0ktcvHfXYHnpK0OjA3Im4D/gD0qeS9CyS1rGK/d5F17QaRtaTI/3ts+XskrZMf08waWOpBaAAi\nYlI+SDwhX3RTRLwkaQfgCkmLgAVkXbWKhgGTJU2KiKEV1o0FRgAPRcR35fsGugGT8pbX58Bu9fqB\nzKxGGnwMqDHyGNDyyWNAy6+ajgEtL10wMytBDiAzS8YBZGbJOIDMLBkHkJkl4wAys2QcQGaWjAPI\nzJJxAJlZMg4gM0vGAWRmyTiAzCwZB5CZJeMAMrNkHEBmlowDyMyScQCZWTIOIDNLxgFkZsk4gMws\nGQeQmSXjADKzZBxAZpaMA8jMknEAmVkyDiAzS8YBZGbJOIDMLBkHkJkl4wAys2QcQGaWjAPIzJJx\nAJlZMg4gM0vGAWRmyTiAzCwZB5CZJeMAMrNkHEBmlowDyMyScQCZWTIOIDNLxgFkZsk4gMwsGQeQ\nmSWjiEhdw3JP0ufAtNR11JMuwIzURVilmtLvZq2IWGVpGzmASoykFyOiX+o67PtK8XfjLpiZJeMA\nMrNkHEClZ1jqAqxKJfe78RiQmSXjFpCZJeMAMrNkHEBmlowDyMyScQCZNQKSlLqGYmiRugBbPklS\nRISkH5CdLf0kdU2lqvx3kX+/LdAReB6YHhELkxZXR24BWaXy8NkNuBP4q6TLJK2Ruq5SVBA+JwMX\nAP2Bx4HNUtZVHxxAVilJGwGnATsDE4CtgK+SFlXCJK0D/CwiBgLvAx+QtYLK1zfKLpoDyKqyEHgY\n2AvYCdg3ImZL6pm2rNIjqTPwCTBZ0nBgN2BIRCySdLCkTtFIryh2ANkSJG0gaU/gO2AQcBxwUET8\nW9IQ4EZJqyUtsoRI2hz4NVAGrAb0AA6PiDJJBwCnAyskLLFOfCuGLUHSkcAhETFQ0ilk4wyPA3OB\nc4D/iYiHU9bYVOXdKEXEooJl3YHHgCPIul2XA7OA5sAmwNCIeC1BufXCAVTiCs52NS8/oyLpDuC5\niLhW0hHAWsDKwIMRMbbwrIzVnwpnuzoD30bEHEm/BLaKiBMkrU3WEuoKvBARjXqiPJ+GL1H5oGav\niLhXUj/gZ5KmRsQDwC3A9gARcVO+fcuIWJAvc/jUo7zlsxHwG2AvSX2Bs4D3Jd0CjAd2lbR2REwB\npqSrtn55DKh0NQM+k7QC8CHQGjhe0nXAAmBHSQcWbF+WoMaSEJnJwAmSfg68TBZGnwH3k43F/QS4\nUlKrZIUWgQOoREXEW8AzZOGzW0RcDPyCbGxhc2BF4GBJHfLt3eopAkltC17OAA4FXgPei4grgJPJ\nur/fAusD7Rq8yCLyGFAJkdQO2C4iHpTUn+xMl4DRwEURcbWkZmRjDHsB70bEI+kqbtoktSE7izWK\n7OzWRhFxXt7t2gLoHRHfSmoBtAc6R8S/01Vc/xxAJSa/jqQfMB84MiJektQHeBQ4NyL+UmF7DzgX\ngaQuETFD0iDgCeBdsgD6Nl//N7KzXJtHxPyEpRaVu2AlouBK2UvImvRlEfESQERMArYFrs4v91/M\n4VO/lFkT+H3evX0DeBD4Adk/DABExKHA68CTSQptIG4BlYCCU+3NgA7ASmRnuhZExOCC7dYGukXE\nvxKVWjIkdQQ2BNpHxL8kbQ08AOwfEQ9L2jwixktaNSI+S1tt8bgF1MQVhM/2wLlkt1RMi4htgFaS\n/iGpv6QngJn5H0OjvK9oeVf4c42Ir4FewHmSBkfE48ABwL2SrgRukbRGUw4f8HVATV4ePoOBK4ET\ngDsl9QJ+ExFbS7qT7A7rKyPii/L3pKu4aapwkeH+wFcR8VdJC4Az8/UPSdoO+BnZmcmPUtbcENwF\na8LyLtcKwK1k15V0Ba4APga+BE6MiFmSVoyILz3gXHySjie7rWLv/KLC8kA6DLgmD6GS+T24BdQE\nFfwP3CYivpJ0ONnA84VkA53tgE+BjyRdEBFfgls+xZR3v3oAB5HNLjBd0u7AmsBtQEvgcEmPRcQ3\n6SptWA6gJqZgzKc/8BdJh0TEq5JWJbvuZyWyMBoD3BcR81LW25QVtmTy/07Jx9ruAt4i+z18Bawc\nEedLerCUwgccQE1OHj7bAXuQXdU8RtIOeQhNAG4HugEnRMQLCUtt0iqM+QwgC/6XgbvJ7uV6PCKm\nSjoG2Dh/W8lN+OYxoCYmn75hFHBofhr3POAQsmb/VLIuWFlETEhXZdNVcfxG0hnAvsDnwEzgaeD2\nfHK3w4FjyaY/abRTatSFW0BNz0zgRbJpO4mIC/Pre8YAAyPi2YS1lYIWZDfzkk/ctgMwKCLm5dNq\nDAJ6Svqc7ErnQ0s1fMDXATV65deWSOqkbGrOr8nOfO1RsNlw4CPgwfKbS63+5V3fEZLOyi99mEl2\n4edPASLiPrJZB3aNiKnA6RHxarKClwNuATVy+ZjPLmQTyM+SNJ5sCs87lT3FYi7ZxPKHAyeS3dQ4\nJ1W9TVUeOBcC/wusCuxHNnPhHcBmkmbl3d6JwDrKJoD7NlnBywm3gBqhwitqlc0ZfDZwINnTK46M\niDeBvclaPR3IrjvpCgwAFn1vh1YnklYmG3f7XURcCwwD2gCdyWYaEPAnScPIJhq7NRr587zqiweh\nGxlJq5A9FeHOfLrOn5LN3dOarBW0f0S8J6lbRLyfv2cAMILs6tqSHW8oJkk7kc3XvEVEfC3pduCJ\niBgmaSWgO9nZx4nRyKdRrU/ugjU+A8keTNc6n1qjOdkd7jPJHtXyZT4WcUx+incmMA3Yxv/jF09E\nPCJpETBR0higLdkFhkTELLLu2KSEJS6X3AJqJPIxg4WSmpO1gH4OvJHfT/Q7skHnPcmuKTkP+JUn\nE2t4yh6dPBZYLSI+k9SmKc/nU1cOoEZA0rpk4zhjgSfzWfKGAEPIQuh6SeeTzSmzInBLRIwppXuK\nlif57+YPZE+yaNJ3s9eVA6gRkPQz4P/IrqC9B/gx2U2l2wGtyJ6aOTw/I+Z/cZcDknYFfkt24Wf4\nH4LKOYAaCUlbkj0quT/wS7JL+3cnO9PVAzifbJIxouDBdpaOpA4R4UsequFB6EYiIp6WtB8wEhiQ\nX8r/MNnzpI4ie4qCg2c54vBZOreAGhlJOwLXApuWTyBWcAe8x3ysUXELqJGJiFH56d63JK0bEbMq\nTPlg1mi4BdRI5Re+fRMR41LXYrasHECNnLtd1pg5gMwsGd+MambJOIDMLBkHkJkl4wCyGpO0UNLL\nkl6TdK+kdnXY18/zCymR9AtJZ1Wz7YqSjluGY5yfz8lco+UVthkuac9aHKubJE91UksOIKuNeRHR\nOyI2JHvEzzGFK5Wp9f9TEfFQRFxazSYrArUOIFv+OYBsWT0F9Mj/5X9T0l/I5rtZU9L2kp6TNClv\nKXWAbNpSSW9JepqCOaslHSLpuvz7rpLul/RK/jUAuBT4Sd76uiLf7kxJL0iaLOmCgn2dI+ltSY8C\n6y7tQ0g6Mt/PK5Luq9Cq21bSU5LekbRzvn1zSVcUHPvouv4gS5kDyGpNUguyqUDKJ1RfFxgREZsA\n3wDnAttGRB+yJ3ScJqkNcCOwC9mTIVarYvfXkM0k2AvoA7xONo3p1Lz1daak7YG1gc2A3kBfST+V\n1JfsETibkAXcpjX4OH+PiE3z471JNnd2uW5kz2nfCbg+/wyHkz3XfdN8/0cqexSSLQPfimG10VbS\ny/n3TwE3A6sD0yJifL58c2AD4Jl86upWwHPAemQ3zJY/D/02sptoK9qa7PHF5PMmf5VPaVpo+/zr\npfx1B7JAWgG4PyLm5sd4qAafaUNJvyfr5nUge3xRuXvyG3ynSPp3/hm2BzYuGB/qlB/7nRocyypw\nAFltzIuI3oUL8pApfJywgH/UTNdOAAABNElEQVRFxH4VtusN1NdVrwIuiYgbKhzjlGU4xnCyubJf\nkXQI2UyT5SruK/JjnxgRhUGFpG61PK7hLpjVv/HAQEk9ACS1k7QO2bPQu0v6Sb7dflW8/zGyp4WW\nj7d0BGaTtW7KjQEOKxhb+qGkVYEngd0ltZW0All3b2lWAD6V1BIYWmHdXpKa5TX/GHg7P/ax+fZI\nWkdS+xocxyrhFpDVq4j4PG9J3Cmpdb743Ih4R9JRwCOSZpA9onjDSnZxMjBM2WOLFwLHRsRzkp7J\nT3P/Mx8HWh94Lm+BzQEOiIhJku4mewb7NLJu4tL8Bng+3/5Vlgy6t4EnyB5pdExEzJd0E9nY0CRl\nB/+cbI5uWwa+F8zMknEXzMyScQCZWTIOIDNLxgFkZsk4gMwsGQeQmSXjADKzZP4fEr2qQNAw4MIA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8ad1fc4b38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TP: 26\n",
      "TN: 74\n",
      "FP: 26\n",
      "FN: 74\n",
      "TPR/recall: 0.26\n",
      "TNR/specificity: 0.74\n",
      "PPV/precision: 0.5\n",
      "NPV/negative predictive value: 0.5\n",
      "FNR/false negative rate: 0.74\n",
      "FPR/false positive rate: 0.26\n",
      "FDR/false discovery rate: 0.5\n",
      "FOR/false ommission rate: 0.5\n",
      "F1 score: 0.342105263158\n",
      "Accuracy: 0.5\n",
      "MCC/Matthews correlation coefficient: 0.0\n",
      "BM/Bookmaker Informedness: 0.0\n",
      "MK/Markedness: 0.0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAASAAAAEmCAYAAADC2k2dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XeYFeXdxvHvTa8KokYjKvYSlbIo\nWFCjAmLMGxvYK/Zo7LH3GHuLJm/EhtiJxhJFQTEqoFhAxRJLUEk0+kYRFQQU8Pf+MbPrYd1dFnbP\nPuye+3Nde7EzZ848v7OwN888M/OMIgIzsxSapS7AzEqXA8jMknEAmVkyDiAzS8YBZGbJOIDMLBkH\nkJkl4wAqMZI+lPSdpOUrrX9VUkjqJulpSYdWen1bSR8VLIektSttc7akWfnXXEkLCpZfk9SivI18\n+9/ly7sW7KNNvq5rwbqt8ppmSfpK0kOS1i94fYf8PSMr1dM7X/9kvrxQ+wXbHZqv363S+h0kfVjL\nn2t5DddWWj9R0n4F7RT+TN6XdHht9t9UOYBK0wfA3uULkjYG2tZ1pxFxYUR0iIgOwDHAuPLliOhe\nzdu+AC6UVOW/RUn9gMeB+4CVgDWBt4AJlYLk/4BtJHUqWHcA8G4tSj8wr+PAWmxbk5nAIZJWrWGb\ncQU/oyHAVfnPvyQ5gErT7WS/nOUOBEYkquXR/M+9q3n9MuCWiLg+ImZFxPSIOB2YDJxTsN1c4G/A\nnpD1doA9gLtqalzSmsCWwBHAIEkrLPEnyULsjkp1VSsiXiYLyA3q0Gaj5gAqTROBZSRtIKk52S/t\nHYlq+Z7sF/a8PDQqSOoI9AH+UsX7RgL9K60bwQ/BuhPwClnPqCYHAhMj4j5gKtUHYW39Dtir8uFp\nVST1BdYCJtWxzUbLAVS6yntB/YG3gY9TFRIRfwW+Bg6u9FIXQMAnVbztE2D5SuvGAStLWovss9XY\nq5MkYH9+6CXdRR0PwyLiY+BG4PxqNtlK0peSZgHPA7cA79elzcbMAVS6bgf2AQ7ix7+o84GWlda1\nBOYVsZ6zgLOB1gXrvgACWLmK7VcGPi9cEdmd1XcAxwH9gIcW0ebWwKpkvSnIAqiXpI1qepOkNQsG\nkr+sYpOLgZ2r2c/4iOiUjwGtDPQCLlhEnU2WA6hERcQ0ssHonYC/Vnr5X0C3SuvWAKYVsZ7H8naP\nKFj3NfAiMLiKtwwBxlaxfgTwa+DhiJi7iGYPJPsdmCLpU2ACWeAdUNObIuL9gsH1TlW8/hlwHYsI\nloj4lOxn/8tF1NlktVj0JtaEDQU6R8Q3lcZf7gVul3Q/8BKwDnACcG2l97eS1KZgeV5ELKhDPWfy\n4zA8FRgl6R3gNqAVcArQm4KwKhcR/5S0LfDPmhqS1I5skHoo2Vm2cnsCp0o6/YdNF/qMAN/Gouex\nuYLs0Kra7fJLIXYB3lzEvpos94BKWERMzc/EVF4/GjgNuBX4ChhF9ss/rNKmbwJzCr4qj+Esbj3P\nUGlANl83iKzH8ynwIbARsGVEVDl2EhHjIqKqcaNCu5GdNr8jIj4t/yIbv2nLDwPcq7HwZ5wDrF6L\nz/IlWQgtV+mlfuWHb2SXE3xCdshYkuQJycwsFfeAzCwZB5CZJeMAMrNkHEBmloxPw9eCWrQNteqY\nugyrpOcGq6UuwaoxefKkzyNikffVOYBqQa060nq9IanLsEomvHB96hKsGm1bqlYXrfoQzMyScQCZ\nWTIOIDNLxgFkZsk4gMwsGQeQmSXjADKzZBxAZpaMA8jMknEAmVkyDiAzS8YBZGbJOIDMLBkHkJkl\n4wAys2QcQGaWjAPIzJJxAJlZMg4gM0vGAWRmyTiAzCwZB5CZJeMAMrNkHEBmlowDyMyScQCZWTIO\nIDNLxgFkZsk4gMwsGQeQmSXjADKzZBxAZpaMA8jMknEAmVkyDiAzS8YBZGbJOIDMLBkHkJkl4wAy\ns2QcQGaWjAPIzJJxAJlZMi1SF2A1W27Z9oy64VgAftJlGb7//ns+mzELgO7rdeXa28dy2lUPAHD8\n/tvTvl1rLrph1BK316Fda5685YSK5VVW7MQ9o17ilCvur1i36w49uOvyQ9ly38uY/Na/lritpqB9\n6+ZstNHGFcsj73+QadM+ZPBuv2KNNdZk7rdzGTxkL848+9w6tXP6qacw6tG/0aplK9ZYay2G3XQr\nnTp1AuD1KVM45ugjmDnza5qpGeMnvkSbNm3q1F5DcQAt5b746hv67nUJAGcesRPfzP6Wa24fC8CM\niVfzq+26c/ktY5j+5Tf10t6s2d9WtAcw4c7f8uBTr1Ysd2jXmqP33pYXp3xQL+01dm3btuWFSa8u\ntG7atA/Zcqt+/PWhR/jmm2/o07sHg3bamV5lZUvczvY79OfCiy6mRYsWnHn6qVx+6cVcdPGlzJ8/\nn0MO3I+bh9/OJt27M336dFq2bFnXj9VgfAjWiM1f8D03//U5jt13u6Lsf63VVmDF5ToyYfLUinXn\nHr0zVw1/krnfzS9Km01N+/bt6dmrjPffn7rojWuwQ/8BtGiR9Rc269OXjz/6CIAnnxjDRhtvwibd\nuwPQpUsXmjdvXreiG5B7QI3cDfc+y0sjT+eq256odpute6/DZSfv/qP1c+Z+x88Puqra9w3ZsYz7\nxkyuWO6+Xle6rtSZx8a9wfEHbF+3wpuIOXPm0KesBwCrr7EGI+97YKHXp0+fzosvTOT0M85eaP3M\nmTPZYdt+Ve5z+O13scGGG1bb5ojht7DH4D0BeO/dd5HEL3cayOeffcYee+7FSSf/ti4fqUEVLYAk\nBXBVRJyUL58MdIiI8+q5nTMi4vcFy89FxBb12cbSbOY3c7nzkRc5eu9tmTt3XpXbPPvyewsdVtXW\n4IFlDD1rBACSuOzk3TnsnNvrVG9TU9UhGMCE8ePo27snzZo14+RTTmPDn/1sodc7duxY5fsW5dKL\nL6J5ixbstc++AMxfMJ/nnhvP+Odfol27dgwasD29epXx8+0ax38QxewBfQvsJuniiPi8iO2cAVQE\nUCmFT7nr7/w7z999KiMemljl60vSA9p43VVo0bw5r/zj3wB0bN+aDddamTE3HQdkA+L3XXMEexx/\nQ8kPRFelfAyoOkvSA7pjxG2MevQRHhszFkkArLJKV/r124bll18egB0H7cQrr0x2AAHzgWHACcCZ\nhS9IWgH4M7Bavur4iJiQr78L6AK8BOwIlEXE55IeBFYF2gDXRsQwSZcAbSW9CrwZEftKmhURHSTd\nC9wWEaPyNocDfwMeBC4BtgVaA3+MiBuK9lNoADO+ns39YyZz0C6bc1sVIbQkPaAhO5Yx8vGXK5a/\nnjWXVbc7rWJ59I3HcfrVDzh8ltDi9oDGjH6cK6+4lDFjn6Fdu3YV6/sPGMjVV1zG7NmzadWqFeOe\nfYZjjzuhhj0tXYo9CP1HYF9Jy1Zafy1wdURsCuwO3JSvPxd4KiJ6AQ/wQ0ABHBIRZUBv4DeSukTE\nacCciOgREftWauMeYE8ASa2A7YFRwFDgq7ztTYHDJK1RT583mWtvf4ounTrU2/5279+LkY9Pqrf9\nWd2ccNwxzJw5k5137E+fsh4ce/SRAHTu3JnfHH8iW22+KX1696BHz14M2ukXiautPUVEcXb8Q0/k\nAmAeMId8DEjSf4H/FGy+ArA+MA7YNSI+yPfxBbBu3gM6D9g1374bMDAiJpa3U0W7bYD3gLXJelJD\n8h7SfcAmwOz8LcsCR0TEmEr1Hw4cDkDLDmVtfnZg/fxgrN7MeOn61CVYNdq21KSI6L2o7RriLNg1\nwGTg1oJ1zYDNI2JO4YYqP7CtRNK2wA75e2ZLeprsUKxaETE3324gWU/o7vLdAcdGxOhFvH8Y2SEk\nzdqtWJyUNitxRb8OKCK+AEaSHfqUGwMcU74gqUf+7XhgSL5uANA5X78sMCMPn/WBvgX7miepuiuv\n7gEOBvoB5YEzGjiq/D2S1pXUfgk/npnVQUNdiHglsHzB8m+A3pKmSHoLODJffz4wQNJkYBDwCTAT\neBxoIWkKcCFQONI6DJgi6c4q2h0DbA08GRHf5etuAt4CJkt6A7gBXw9llkTRxoCWhKTWwIKImC9p\nc+B/I6LHot5XbM3arRit1xuSugyrxGNAS6+laQxocawGjJTUDPgOOCxxPWZWREtVAEXEe0DP1HWY\nWcPwzahmlowDyMyScQCZWTIOIDNLxgFkZsk4gMwsGQeQmSXjADKzZBxAZpaMA8jMknEAmVkyDiAz\nS8YBZGbJOIDMLBkHkJkl4wAys2QcQGaWjAPIzJJxAJlZMg4gM0um2knpJS1T0xsj4uv6L8fMSklN\nT8V4EwiyRxmXK18OskfomJktsWoDKCJWbchCzKz01GoMSNJeks7Iv+8qqay4ZZlZKVhkAEm6Hvg5\nsH++ajbw52IWZWaloTZPRt0iInpJegUgIr6Q1KrIdZlZCajNIdi8/FntASCpC/B9Uasys5JQmwD6\nI3A/sIKk84HxwKVFrcrMSsIiD8EiYoSkScAO+arBEfFGccsys1JQmzEggObAPLLDMF89bWb1ojZn\nwc4E7gZ+CnQF7pJ0erELM7OmrzY9oP2AsoiYDSDpImAScHExCzOzpq82h1PTWDioWgDvF6ccMysl\nNd2MejXZmM9s4E1Jo/PlAWRnwszM6qSmQ7DyM11vAo8WrJ9YvHLMrJTUdDPqzQ1ZiJmVnkUOQkta\nC7gI2BBoU74+ItYtYl1mVgJqMwg9HLiVbB6gQcBI4J4i1mRmJaI2AdQuIkYDRMTUiDiL7O54M7M6\nqc11QN9KEjBV0pHAx8CKxS3LzEpBbQLoBKAD8BuysaBlgUOKWZSZlYba3Iz6Qv7tTH6YlMzMrM5q\nuhDxAfI5gKoSEbsVpSIzKxk19YCub7AqzKwk1XQh4tiGLMTMSo/n9jGzZBxAZpZMrQNIUutiFmJm\npac2MyJuJul14L18ubuk64pemZk1ebXpAf0B2BmYDhARr+FbMcysHtQmgJpFxLRK6xYUoxgzKy21\nuRXj35I2A0JSc+BY4N3ilmVmpaA2PaCjgBOB1YD/A/rm68zM6qQ294L9F9irAWoxsxJTmxkRb6SK\ne8Ii4vCiVGRmJaM2Y0BPFnzfBtgV+HdxyjGzUlKbQ7B7C5cl3Q48UbSKzKxkLMmtGGsAq9d3IWZW\nemozBjSDH8aAmgFfAKcVsygzKw01BlA+F3R3snmgAb6PiGonKTMzWxw1HoLlYfNARCzIvxw+ZlZv\najMG9KKkXkWvxMxKTk1zQreIiPnAVsBhkqYC35A9oDAiwqFkZnVS0xjQi0AvYJcGqsXMSkxNASTI\nnobaQLWYWYmpKYBWkHRidS9GxFVFqMfMSkhNAdSc7ImoaqBazKzE1BRAn0TEBQ1WiZmVnJpOw7vn\nY2ZFVVMAbd9gVZhZSao2gCLii4YsxMxKjx9MaGbJOIDMLBkHkJkl4wAys2QcQGaWjAPIzJJxAJlZ\nMg4gM0vGAWRmyTiAzCwZB5CZJeMAMrNkHEBmlowDyMyScQCZWTIOIDNLxgFkZsk4gBqBWS//gYn3\nnFbxtdrKy9GvbB3mvHI9O229UcV29197JP3K1qlTW1v3XmehtmZMvJpfbrvJQttcdepgPptwZZ3a\naQqmT59On7Ie9CnrQbeuK7Hm6qtULLdtKfqU9aCsx0bss9dgZs+eXae27r7rTjbtuQmb9tyEbftt\nwZTXXqt47csvv2TvPfeg+0br02PjDZj4/PN1/WgNpqanYthSYs638+i71yULrVv9p1346NMZnDp0\nIKOefaPe2nr25fcq2uq8TDveePhcnpz4j4rXe224Gst2aFtv7TVmXbp04YVJrwLwuwvOo32HDpxw\n4skALN+pQ8VrB+2/Lzfe8GeOO6Hax+wtUrduazDmqWfo3Lkzox9/jF8fdTjjnnsBgJNPOI4BA3bk\n7nvv47vvvqtz2DUk94AasSnvfsxXs+ayXZ/1i7L/XXfoyZgJbzFn7jwAmjUTvz9+F8689sGitNdU\nbblVP96f+s867WPzLbagc+fOAGzWpy8ff/wRAF9//TXjxz/LQYcMBaBVq1Z06tSpbgU3IPeAGoG2\nrVsy8Z7TAJj28XT2POnGitcuvelxzv31zjz1wtvVvv+EA7Znz502/dH6CZP/yUmX3Vft+wYP7MUf\n7vh7xfJRe27Do8+8zqeff70kH6MkzZ8/nzGjH6P/gB1/9Np+++zJe++886P1vzn+RPbd/4Bq9zn8\n1psZOHAQAB+8/z7LL78Chw89mNenvEbPXmVccfW1tG/fvv4+RBE1eABJWgC8nrf9D+DAiFisPqOk\nm4CrIuItSWdExO8LXnsuIrao16ITq+oQrNyEV6YCsGXPtap9/9UjxnL1iLGL1eZKyy/Dz9b5KU88\n/xYAK6+wLLv178mAw65drP2Uqjlz5tCnrAcAW2zVr6KHUuiOu+5d7P0+8/Tfue3Wmxn79HggC7hX\nX5nMVddcx2Z9+nDSCcdxxWWXcO75F9btAzSQFD2gORHRA0DSncCRwGI9Zz4iDi1YPAP4fcFrTSp8\nauPSm0bz26EDmb/g+ypfX5Ie0O79e/HwU1OYPz/bZ/f1urLmqivw5sPnAtCuTUveeOhcNvrV+fX0\nKZqWtm3bVowBVWdxe0CvT5nCUUccykN/e4wuXboAsErXrqzStSub9ekDwK6778GVl1X9n9XSKPUh\n2DhgEwBJJwKH5OtviohrJLUHRgJdyZ5Vf2FE3CvpaeBkYA+graRXgTcjYl9JsyKig6R7gdsiYlS+\n/+HA34AHgUuAbYHWwB8j4oYG+bRFMnbi25x79M6svMKyVb6+JD2gITuWcfZ1D1csPz7+Tdbof0bF\n8mcTrnT41NHi9ID+9a9/sdeQ3bj51ttZZ911K9avtNJKdO26Ku++8w7rrrceTz81lvU32LAY5RZF\nsgCS1AIYBDwuqQw4GOhD9kjoFyQ9A6wJ/CcifpG/Z6HfsIg4TdIx5T2qSu4B9gRGSWpF9qTXo4Ch\nwFcRsamk1sAESWMi4oNK9R0OHA5Ayw719bGL5tKbR3PfNUfUy75WW3k5uq7UmXGT6jZwavXn4t9d\nwBfTp3P8sUcD0KJFCya88DIAV11zHQcfsC/fffcd3dZck2E33Zqy1MWiiGjYBn8YA4KsB3QSWTB0\niYhz8m0uBD4DHgdGk/WCHomIcfnrTwMnR8TL5T2egv2X94DaAO8BawM7AkPyHtJ9ZL2u8nGnZYEj\nImJMdTU3a7ditF5vSP38AKzezHjp+tQlWDXattSkiOi9qO2SjgGVk6SqNoyId/Pe0U7AxXlP5YLa\nNBIRc/OgGkjWE7q7vDng2IgYvaQfwMzqx9JyHdCzwC6S2uXjPrsC4yT9FJgdEXcAVwC9qnjvPEkt\nq9nvPWSHdv3IelLkfx5V/h5J6+ZtmlkDSz0IDUBETM4HiV/MV90UEa9IGghcLul7YB7ZoVplw4Ap\nkiZHxL6VXhsDjAAejojvyvcNdAMm5z2vz4Bd6vUDmVmtNPgYUGPkMaClk8eAll61HQNaWg7BzKwE\nOYDMLBkHkJkl4wAys2QcQGaWjAPIzJJxAJlZMg4gM0vGAWRmyTiAzCwZB5CZJeMAMrNkHEBmlowD\nyMyScQCZWTIOIDNLxgFkZsk4gMwsGQeQmSXjADKzZBxAZpaMA8jMknEAmVkyDiAzS8YBZGbJOIDM\nLBkHkJkl4wAys2QcQGaWjAPIzJJxAJlZMg4gM0vGAWRmyTiAzCwZB5CZJeMAMrNkHEBmlowDyMyS\ncQCZWTIOIDNLxgFkZsk4gMwsGQeQmSXjADKzZBxAZpaMIiJ1DUs9SZ8B01LXUU+WBz5PXYRVqSn9\n3aweESssaiMHUImR9HJE9E5dh/1YKf7d+BDMzJJxAJlZMg6g0jMsdQFWrZL7u/EYkJkl4x6QmSXj\nADKzZBxAZpaMA8jMknEAmTUCkpS6hmJokboAWzpJUkSEpJXJzpb+J3VNpar87yL/fgdgGeAF4NOI\nWJC0uDpyD8iqlIfPLsDdwP9KulRS19R1laKC8DkOOB/oAzwFbJayrvrgALIqSdoYOBHYGXgR+Dnw\nVdKiSpikdYFtImJL4EPgX2S9oPLXG+UhmgPIqrMAeAQYDPwC2CsiZkr6WdqySo+kLsB/gCmShgO7\nAIMi4ntJB0paNhrpFcUOIFuIpA0l7QF8B/QDjgYOiIj3JQ0CbpS0UtIiS4ikvsDpwHxgJWBtYGhE\nzJe0H3AS0DFhiXXiWzFsIZIOAw6KiC0lHU82zvAUMBs4Ezg1Ih5JWWNTlR9GKSK+L1i3BjAWOJTs\nsOsyYAbQHOgJ7BsRbyQot144gEpcwdmu5uVnVCTdBTwfEddJOhRYHVgOeCgixhSelbH6U+lsVxfg\n24iYJWl34OcRcYykdch6Qj8BXoqIRj1Rnk/Dl6h8ULN7RPxFUm9gG0lTI+JB4BZgAEBE3JRv3zIi\n5uXrHD71KO/5bAycDQyWVAacBnwo6RZgIvArSetExHvAe+mqrV8eAypdzYD/SuoI/BtoDfxa0vXA\nPGAnSfsXbD8/QY0lITJTgGMkbQu8ShZG/wUeIBuLWwu4UlKrZIUWgQOoREXE28AEsvDZJSJ+D/wP\n2dhCX6ATcKCkDvn27vUUgaS2BYufAwcDbwAfRMTlwHFkh7/fAhsA7Rq8yCLyGFAJkdQO6B8RD0nq\nQ3amS8DjwEURca2kZmRjDIOBf0bEo+kqbtoktSE7izWK7OzWxhFxTn7YtTnQIyK+ldQCaA90iYj3\n01Vc/xxAJSa/jqQ3MBc4LCJekdQLeBI4KyL+VGl7DzgXgaTlI+JzSf2AZ4B/kgXQt/nrt5Kd5eob\nEXMTllpUPgQrEQVXyl5M1qWfHxGvAETEZGAH4Nr8cv8KDp/6pcyqwO/yw9u3gIeAlcn+YwAgIg4G\n3gSeTVJoA3EPqAQUnGpvBnQAOpOd6ZoXETsWbLcO0C0inkhUasmQtAywEdA+Ip6QtB3wILBPRDwi\nqW9ETJS0YkT8N221xeMeUBNXED4DgLPIbqmYFhHbA60k/U1SH0nPANPzX4ZGeV/R0q7w5xoRXwPd\ngXMk7RgRTwH7AX+RdCVwi6SuTTl8wNcBNXl5+OwIXAkcA9wtqTtwdkRsJ+lusjusr4yIL8rfk67i\npqnSRYb7AF9FxP9Kmgeckr/+sKT+wDZkZyY/SllzQ/AhWBOWH3J1BG4ju67kJ8DlwMfAl8CxETFD\nUqeI+NIDzsUn6ddkt1UMyS8qLA+kQ4A/5CFUMn8P7gE1QQX/gNtExFeShpINPF9ANtDZDvgE+EjS\n+RHxJbjnU0z54dfawAFkswt8KmlXYFXgDqAlMFTS2Ij4Jl2lDcsB1MQUjPn0Af4k6aCIeF3SimTX\n/XQmC6PRwP0RMSdlvU1ZYU8m//O9fKztHuBtsr+Hr4DlIuI8SQ+VUviAA6jJycOnP7Ab2VXNoyUN\nzEPoReBOoBtwTES8lLDUJq3SmM8WZMH/KnAv2b1cT0XEVElHApvkbyu5Cd88BtTE5NM3jAIOzk/j\nngMcRNbtn0p2CDY/Il5MV2XTVXn8RtLJwF7AZ8B0YDxwZz6521DgKLLpTxrtlBp14R5Q0zMdeJls\n2k4i4oL8+p7RwJYR8VzC2kpBC7KbecknbhsI9IuIOfm0Gv2An0n6jOxK54NLNXzA1wE1euXXlkha\nVtnUnF+TnfnarWCz4cBHwEPlN5da/csPfUdIOi2/9GE62YWfWwNExP1ksw78KiKmAidFxOvJCl4K\nuAfUyOVjPr8km0B+hqSJZFN43q3sKRazySaWHwocS3ZT46xU9TZVeeBcANwOrAjsTTZz4V3AZpJm\n5Ie9k4B1lU0A922ygpcS7gE1QoVX1CqbM/gMYH+yp1ccFhH/AIaQ9Xo6kF138hNgC+D7H+3Q6kTS\ncmTjbhdGxHXAMKAN0IVspgEBV0saRjbR2G3RyJ/nVV88CN3ISFqB7KkId+fTdW5NNndPa7Je0D4R\n8YGkbhHxYf6eLYARZFfXlux4QzFJ+gXZfM2bR8TXku4EnomIYZI6A2uQnX2cFI18GtX65EOwxmdL\nsgfTtc6n1mhOdof7dLJHtXyZj0UcmZ/inQ5MA7b3P/ziiYhHJX0PTJI0GmhLdoEhETGD7HBscsIS\nl0ruATUS+ZjBAknNyXpA2wJv5fcTXUg26LwH2TUl5wC/9WRiDU/Zo5PHACtFxH8ltWnK8/nUlQOo\nEZC0Htk4zhjg2XyWvEHAILIQ+rOk88jmlOkE3BIRo0vpnqKlSf53cwXZkyya9N3sdeUAagQkbQP8\nnewK2pHAmmQ3lfYHWpE9NXN4fkbM/+MuBST9CjiX7MLP8H8EVXMANRKStiJ7VHIfYHeyS/t3JTvT\ntTZwHtkkY0TBg+0sHUkdIsKXPNTAg9CNRESMl7Q3cB+wRX4p/yNkz5M6nOwpCg6epYjDZ9HcA2pk\nJO0EXAdsWj6BWMEd8B7zsUbFPaBGJiJG5ad735a0XkTMqDTlg1mj4R5QI5Vf+PZNRDyduhazJeUA\nauR82GWNmQPIzJLxzahmlowDyMyScQCZWTIOIKs1SQskvSrpDUl/kdSuDvvaNr+QEkn/I+m0Grbt\nJOnoJWjjvHxO5lqtr7TNcEl7LEZb3SR5qpPF5ACyxTEnInpExEZkj/g5svBFZRb731REPBwRl9Sw\nSSdgsQPIln4OIFtS44C18//5/yHpT2Tz3awqaYCk5yVNzntKHSCbtlTS25LGUzBntaSDJF2ff/8T\nSQ9Iei3/2gK4BFgr731dnm93iqSXJE2RdH7Bvs6U9I6kJ4H1FvUhJB2W7+c1SfdX6tXtIGmcpHcl\n7Zxv31zS5QVtH1HXH2QpcwDZYpPUgmwqkPIJ1dcDRkRET+Ab4Cxgh4joRfaEjhMltQFuBH5J9mSI\nlarZ/R/IZhLsDvQC3iSbxnRq3vs6RdIAYB1gM6AHUCZpa0llZI/A6UkWcJvW4uP8NSI2zdv7B9nc\n2eW6kT2n/RfAn/PPMJTsue6b5vs/TNmjkGwJ+FYMWxxtJb2afz8OuBn4KTAtIibm6/sCGwIT8qmr\nWwHPA+uT3TBb/jz0O8huoq1sO7LHF5PPm/xVPqVpoQH51yv5cgeyQOoIPBARs/M2Hq7FZ9pI0u/I\nDvM6kD2+qNzI/Abf9yS9n39bRPbnAAABVUlEQVSGAcAmBeNDy+Ztv1uLtqwSB5AtjjkR0aNwRR4y\nhY8TFvBEROxdabseQH1d9Srg4oi4oVIbxy9BG8PJ5sp+TdJBZDNNlqu8r8jbPjYiCoMKSd0Ws13D\nh2BW/yYCW0paG0BSO0nrkj0LfQ1Ja+Xb7V3N+8eSPS20fLxlGWAmWe+m3GjgkIKxpVUkrQg8C+wq\nqa2kjmSHe4vSEfhEUktg30qvDZbULK95TeCdvO2j8u2RtK6k9rVox6rgHpDVq4j4LO9J3C2pdb76\nrIh4V9LhwKOSPid7RPFGVeziOGCYsscWLwCOiojnJU3IT3M/lo8DbQA8n/fAZgH7RcRkSfeSPYN9\nGtlh4qKcDbyQb/86CwfdO8AzZI80OjIi5kq6iWxsaLKyxj8jm6PbloDvBTOzZHwIZmbJOIDMLBkH\nkJkl4wAys2QcQGaWjAPIzJJxAJlZMv8Pg4jiOCirxTkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8ad1f7b518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# requires label shape of (m,)\n",
    "y_train_p = [np.argmax(ii) for ii in y_train]\n",
    "y_train_p = np.array(y_train_p)\n",
    "y_valid_p = [np.argmax(ii) for ii in y_valid]\n",
    "y_valid_p = np.array(y_valid_p)\n",
    "from sklearn.naive_bayes import GaussianNB, MultinomialNB\n",
    "gnb = GaussianNB()\n",
    "y_preds = gnb.fit(X_train, y_train_p).predict(X_valid)\n",
    "y_trues = y_valid_p\n",
    "cm = confusion_matrix(y_trues, y_preds)\n",
    "fpr, tpr = print_statistics(cm)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm, classes=['Negative','Positive'], title='GAUSSIAN-NB')\n",
    "plt.show()\n",
    "#print(\"GaussianNB: Number of mislabeled points out of a total %d points : %d\" % (X_valid.shape[0],(y_valid != y_pred).sum()))\n",
    "clf = MultinomialNB()\n",
    "y_preds = clf.fit(X_train, y_train_p).predict(X_valid)\n",
    "y_trues = y_valid_p\n",
    "cm = confusion_matrix(y_trues, y_preds)\n",
    "fpr, tpr = print_statistics(cm)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm, classes=['Negative','Positive'], title='MULTINOMIAL-NB')\n",
    "plt.show()\n",
    "#print(\"MultinomialNB: Number of mislabeled points out of a total %d points : %d\" % (X_valid.shape[0],(y_valid != y_pred).sum()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}