topicnet/dataset_manager/DemoDataset.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/evgenyegorov/miniconda3/envs/artm10/lib/python3.6/site-packages/artm/master_component.py:161: DeprecationWarning: PyUnicode_AsEncodedObject() is deprecated; use PyUnicode_AsEncodedString() to encode from str to bytes or PyCodec_Encode() for generic encoding\n",
      "  master_config.topic_name.append(topic_name)\n"
     ]
    }
   ],
   "source": [
    "from topicnet.dataset_manager import api"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Markdown, display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Измеряемые характеристики датасетов.\n",
       "* D -- число документов\n",
       "* <имя модальности> W -- размер словаря данной модальности (количество уникальных токенов)\n",
       "* <имя модальности> len D -- средняя длина документа в терминах этой модальности (количество слов)\n",
       "* <имя модальности> len D uniq -- средняя длина документа в уникальных терминах этой модальности (количество уникальных слов)\n",
       "\n",
       "Вторая таблица выводит более подробную информацию о длинах документов в расчёте на модальность.\n",
       "\n",
       "## 20 NG\n",
       "**Скачать** `api.load_dataset('20NG')`\n",
       "\n",
       "|       |    D |   @lemmatized W |   @lemmatized len D |   @lemmatized len D uniq |   @bigram W |   @bigram len D |   @bigram len D uniq |\n",
       "|:------|------------:|-----------------------:|---------------------------:|--------------------------------:|-------------------:|-----------------------:|----------------------------:|\n",
       "| value |       11301 |             1.0614e+06 |                    93.9204 |                         60.5687 |             213701 |                18.9099 |                     15.0068 |\n",
       "\n",
       "\n",
       "|      |   len_total@lemmatized |   len_total@bigram |   len_uniq@lemmatized |   len_uniq@bigram |\n",
       "|:-----|-----------------------:|-------------------:|----------------------:|------------------:|\n",
       "| mean |                93.9204 |            18.9099 |               60.5687 |           15.0068 |\n",
       "| std  |               276.901  |            66.4278 |              104.23   |           39.1756 |\n",
       "| min  |                 0      |             0      |                0      |            0      |\n",
       "| 25%  |                20      |             3      |               19      |            3      |\n",
       "| 50%  |                42      |             8      |               35      |            8      |\n",
       "| 75%  |                83      |            16      |               65      |           15      |\n",
       "| max  |              6497      |          1528      |             1875      |          831      |\n",
       "\n",
       "**Метаданные:** есть метки классов (20 штук)\n",
       "\n",
       "## МКБ-10\n",
       "**Скачать** `api.load_dataset('MKB10')`\n",
       "\n",
       "Вместо лемматизации через pymorphy был проведён стемминг.\n",
       "\n",
       "**Метаданные:** есть метки классов (25 штук). См. https://ru.wikipedia.org/wiki/%D0%9C%D0%9A%D0%91-10#%D0%9F%D0%B5%D1%80%D0%B5%D1%87%D0%B5%D0%BD%D1%8C_%D0%BA%D0%BB%D0%B0%D1%81%D1%81%D0%BE%D0%B2_%D0%9C%D0%9A%D0%91-10\n",
       "\n",
       "|       |    D |   @text W |   @text len D |   @text len D uniq |   @letter W |   @letter len D |   @letter len D uniq |\n",
       "|:------|------------:|-----------------:|---------------------:|--------------------------:|-------------------:|-----------------------:|----------------------------:|\n",
       "| value |        1733 |           953168 |               550.01 |                    550.01 |               1733 |                      1 |                           1 |\n",
       "\n",
       "|      |   len_total@text |   len_total@letter |   len_uniq@text |   len_uniq@letter |\n",
       "|:-----|-----------------:|-------------------:|----------------:|------------------:|\n",
       "| mean |          550.01  |                  1 |         550.01  |                 1 |\n",
       "| std  |          736.858 |                  0 |         736.858 |                 0 |\n",
       "| min  |            7     |                  1 |           7     |                 1 |\n",
       "| 25%  |          148     |                  1 |         148     |                 1 |\n",
       "| 50%  |          300     |                  1 |         300     |                 1 |\n",
       "| 75%  |          640     |                  1 |         640     |                 1 |\n",
       "| max  |         8042     |                  1 |        8042     |                 1 |\n",
       "\n",
       "\n",
       "## Stanford triplets\n",
       "**Скачать** `api.load_dataset('stanford_triplets')`\n",
       "\n",
       "|       |    D |   @word W |   @word len D |   @word len D uniq |\n",
       "|:------|------------:|-----------------:|---------------------:|--------------------------:|\n",
       "| value |      963564 |      3.55391e+09 |               3688.3 |                   962.404 |\n",
       "\n",
       "\n",
       "|      |   len_total@word |   len_uniq@word |\n",
       "|:-----|-----------------:|----------------:|\n",
       "| mean |           3688.3 |         962.404 |\n",
       "| std  |           3482.8 |         458.434 |\n",
       "| min  |              0   |           0     |\n",
       "| 25%  |           1791   |         665     |\n",
       "| 50%  |           2835   |         878     |\n",
       "| 75%  |           4608   |        1184     |\n",
       "| max  |         754814   |       17874     |\n",
       "\n",
       "**Метаданные:** есть тройки, по которым можно измерять качество\n",
       "\n",
       "## Lenta\n",
       "**Скачать** `api.load_dataset('lenta')`\n",
       "\n",
       "|       |    D |   @topmine W |   @topmine len D |   @topmine len D uniq |   @time_n W |   @time_n len D |   @time_n len D uniq |   @lemmatized_title W |   @lemmatized_title len D |   @lemmatized_title len D uniq |   @lemmatized W |   @lemmatized len D |   @lemmatized len D uniq |   @theme W |   @theme len D |   @theme len D uniq |\n",
       "|:------|------------:|--------------------:|------------------------:|-----------------------------:|-------------------:|-----------------------:|----------------------------:|-----------------------------:|---------------------------------:|--------------------------------------:|-----------------------:|---------------------------:|--------------------------------:|------------------:|----------------------:|---------------------------:|\n",
       "| value |      263557 |         2.32892e+07 |                  88.365 |                      83.8258 |             263557 |                      1 |                           1 |                  2.05546e+06 |                          7.79894 |                               7.72848 |            2.90254e+07 |                     110.13 |                         84.5878 |            383816 |               1.45629 |                    1.45629 |\n",
       "\n",
       "\n",
       "|      |   len_total@topmine |   len_total@time_n |   len_total@lemmatized_title |   len_total@lemmatized |   len_total@theme |   len_uniq@topmine |   len_uniq@time_n |   len_uniq@lemmatized_title |   len_uniq@lemmatized |   len_uniq@theme |\n",
       "|:-----|--------------------:|-------------------:|-----------------------------:|-----------------------:|------------------:|-------------------:|------------------:|----------------------------:|----------------------:|-----------------:|\n",
       "| mean |             88.365  |                  1 |                      7.79894 |               110.13   |          1.45629  |            83.8258 |                 1 |                     7.72848 |               84.5878 |         1.45629  |\n",
       "| std  |             50.2072 |                  0 |                      1.86916 |                39.7804 |          0.722741 |            47.5763 |                 0 |                     1.81461 |               26.7959 |         0.722741 |\n",
       "| min  |              1      |                  1 |                      1       |                 7      |          1        |             1      |                 1 |                     1       |                7      |         1        |\n",
       "| 25%  |             54      |                  1 |                      6       |                83      |          1        |            51      |                 1 |                     6       |               66      |         1        |\n",
       "| 50%  |             77      |                  1 |                      8       |               104      |          1        |            73      |                 1 |                     8       |               81      |         1        |\n",
       "| 75%  |            110      |                  1 |                      9       |               131      |          2        |           104      |                 1 |                     9       |               99      |         2        |\n",
       "| max  |            791      |                  1 |                     17       |              1000      |          3        |           647      |                 1 |                    16       |              542      |         3        |\n",
       "\n",
       "\n",
       "## NIPS\n",
       "**Скачать** `api.load_dataset('NIPS')`\n",
       "\n",
       "|       |    D |   @word W |   @word len D |   @word len D uniq |\n",
       "|:------|------------:|-----------------:|---------------------:|--------------------------:|\n",
       "| value |        7241 |      1.18333e+07 |              1634.21 |                    644.49 |\n",
       "\n",
       "\n",
       "|      |   len_total@word |   len_uniq@word |\n",
       "|:-----|-----------------:|----------------:|\n",
       "| mean |         1634.21  |          644.49 |\n",
       "| std  |          481.923 |          162.31 |\n",
       "| min  |            0     |            0    |\n",
       "| 25%  |         1249     |          524    |\n",
       "| 50%  |         1663     |          641    |\n",
       "| 75%  |         1978     |          755    |\n",
       "| max  |         6000     |         1513    |\n",
       "\n",
       "есть несколько версий, использованные в сторонних статьях\n",
       "\n",
       "## PScience\n",
       "**Скачать** `api.load_dataset('postnauka')`\n",
       "\n",
       "Модальность `@snippet` -- сырой необработанный текст, предназначенный для показа топ-документов. Существует \"эталонная\" тематическая модель на 19 + 1 тему.\n",
       "\n",
       "|       |    D |   @title W |   @title len D |   @title len D uniq |   @2gramm W |   @2gramm len D |   @2gramm len D uniq |   @3gramm W |   @3gramm len D |   @3gramm len D uniq |   @snippet W |   @snippet len D |   @snippet len D uniq |   @word W |   @word len D |   @word len D uniq |   @post_tag W |   @post_tag len D |   @post_tag len D uniq |   @author W |   @author len D |   @author len D uniq |\n",
       "|:------|------------:|------------------:|----------------------:|---------------------------:|-------------------:|-----------------------:|----------------------------:|-------------------:|-----------------------:|----------------------------:|--------------------:|------------------------:|-----------------------------:|-----------------:|---------------------:|--------------------------:|---------------------:|-------------------------:|------------------------------:|-------------------:|-----------------------:|----------------------------:|\n",
       "| value |        3404 |             13546 |               3.97944 |                    3.98002 |             163289 |                47.9697 |                     37.0029 |               5848 |                1.71798 |                     1.30288 |               46033 |                 13.5232 |                      13.5256 |      1.43373e+06 |              421.189 |                    254.58 |                18031 |                    5.297 |                         5.297 |               3685 |                1.08255 |                     1.08255 |\n",
       "\n",
       "\n",
       "|      |   len_total@title |   len_total@2gramm |   len_total@3gramm |   len_total@snippet |   len_total@word |   len_total@post_tag |   len_total@author |   len_uniq@title |   len_uniq@2gramm |   len_uniq@3gramm |   len_uniq@snippet |   len_uniq@word |   len_uniq@post_tag |   len_uniq@author |\n",
       "|:-----|------------------:|-------------------:|-------------------:|--------------------:|-----------------:|---------------------:|-------------------:|-----------------:|------------------:|------------------:|-------------------:|----------------:|--------------------:|------------------:|\n",
       "| mean |           3.98002 |            47.9697 |            1.71798 |             13.5244 |          421.189 |              5.297   |           1.08255  |          3.98002 |           37.0029 |           1.30288 |           13.5256  |         254.58  |             5.297   |          1.08255  |\n",
       "| std  |           2.21452 |            49.9748 |            3.07432 |              3.0559 |          482.422 |              2.01979 |           0.413483 |          2.21452 |           35.4885 |           1.81957 |            3.05377 |         249.288 |             2.01979 |          0.413483 |\n",
       "| min  |           1       |             0      |            0       |              5      |            3     |              1       |           0        |          1       |            0      |           0       |            5       |           3     |             1       |          0        |\n",
       "| 25%  |           2       |            15      |            0       |             12      |          106     |              4       |           1        |          2       |           13      |           0       |           12       |          80     |             4       |          1        |\n",
       "| 50%  |           3.5     |            33      |            1       |             13      |          294     |              5       |           1        |          3.5     |           27      |           1       |           13       |         197     |             5       |          1        |\n",
       "| 75%  |           5       |            67      |            2       |             15      |          563     |              6       |           1        |          5       |           52      |           2       |           15       |         343     |             6       |          1        |\n",
       "| max  |          22       |           778      |           69       |             28      |         4892     |             18       |           4        |         22       |          444      |          27       |           28       |        2277     |            18       |          4        |\n",
       "\n",
       "## RTL-Wiki\n",
       "**Скачать** `api.load_dataset('RTL_WiKi')`\n",
       "\n",
       "|       |    D |   @lemmatized W |   @lemmatized len D |   @lemmatized len D uniq |   @bigram W |   @bigram len D |   @bigram len D uniq |\n",
       "|:------|------------:|-----------------------:|---------------------------:|--------------------------------:|-------------------:|-----------------------:|----------------------------:|\n",
       "| value |        7838 |            1.28065e+07 |                     1633.9 |                         691.157 |             503619 |                64.2535 |                     30.8372 |\n",
       "\n",
       "|      |   len_total@lemmatized |   len_total@bigram |   len_uniq@lemmatized |   len_uniq@bigram |\n",
       "|:-----|-----------------------:|-------------------:|----------------------:|------------------:|\n",
       "| mean |                1633.9  |            64.2535 |               691.157 |           30.8372 |\n",
       "| std  |                1565.19 |            73.1737 |               521.463 |           28.071  |\n",
       "| min  |                   2    |             0      |                 2     |            0      |\n",
       "| 25%  |                 500    |            18      |               283     |           11      |\n",
       "| 50%  |                1115.5  |            41      |               554     |           22      |\n",
       "| 75%  |                2233.5  |            85      |               961     |           42      |\n",
       "| max  |               15851    |          1098      |              4184     |          283      |\n",
       "\n",
       "есть версия, отфильтрованная по персоналиям, содержащая только 1201 документ\n",
       "\n",
       "**Скачать** `api.load_dataset('RTL_Wiki_person')`\n",
       "\n",
       "|       |    D |   @lemmatized W |   @lemmatized len D |   @lemmatized len D uniq |   @bigram W |   @bigram len D |   @bigram len D uniq |\n",
       "|:------|------------:|-----------------------:|---------------------------:|--------------------------------:|-------------------:|-----------------------:|----------------------------:|\n",
       "| value |        1201 |            1.92167e+06 |                    1600.06 |                          729.93 |             371430 |                309.267 |                     196.595 |\n",
       "\n",
       "\n",
       "|      |   len_total@lemmatized |   len_total@bigram |   len_uniq@lemmatized |   len_uniq@bigram |\n",
       "|:-----|-----------------------:|-------------------:|----------------------:|------------------:|\n",
       "| mean |                1600.06 |            309.267 |               729.93  |           196.595 |\n",
       "| std  |                1569.31 |            323.991 |               541.153 |           170.06  |\n",
       "| min  |                  73    |              4     |                60     |             4     |\n",
       "| 25%  |                 484    |             90     |               305     |            70     |\n",
       "| 50%  |                1036    |            206     |               575     |           147     |\n",
       "| 75%  |                2117    |            403     |              1007     |           268     |\n",
       "| max  |               11661    |           3212     |              3108     |          1216     |\n",
       "\n",
       "\n",
       "## RuWiki-Good\n",
       "\n",
       "Отдельные статьи из дампа русской Википедии за `20200301`. Подвыборка состоит из статей, относящихся к числу \"добротных\", \"хороших\" или \"избранных\" (см. https://ru.wikipedia.org/wiki/Википедия:Добротные_статьи)\n",
       "\n",
       "**Скачать** `api.load_dataset('ruwiki_good')`\n",
       "\n",
       "\n",
       "|       |    D |   @categories W |   @categories len D |   @categories len D uniq |   @lemmatized W |   @lemmatized len D |   @lemmatized len D uniq |   @ngramms W |   @ngramms len D |   @ngramms len D uniq |\n",
       "|:------|------------:|-----------------------:|---------------------------:|--------------------------------:|-----------------------:|---------------------------:|--------------------------------:|--------------------:|------------------------:|-----------------------------:|\n",
       "| value |        8603 |                  44493 |                     5.1718 |                         5.17296 |            1.66434e+07 |                     1934.6 |                         721.389 |         3.11727e+06 |                 362.347 |                      287.695 |\n",
       "\n",
       "\n",
       "\n",
       "|      |   len_total@categories |   len_total@lemmatized |   len_total@ngramms |   len_uniq@categories |   len_uniq@lemmatized |   len_uniq@ngramms |\n",
       "|:-----|-----------------------:|-----------------------:|--------------------:|----------------------:|----------------------:|-------------------:|\n",
       "| mean |                5.17296 |                1934.6  |             362.347 |               5.17296 |               721.389 |            287.695 |\n",
       "| std  |                4.0102  |                2065.78 |             384.164 |               4.0102  |               596.947 |            292.717 |\n",
       "| min  |                0       |                  55    |               8     |               0       |                35     |              8     |\n",
       "| 25%  |                2       |                 593    |             112.5   |               2       |               296     |             96     |\n",
       "| 50%  |                4       |                1223    |             229     |               4       |               540     |            189     |\n",
       "| 75%  |                7       |                2447    |             458     |               7       |               943     |            365     |\n",
       "| max  |               51       |               17472    |            3335     |              51       |              4423     |           2459     |\n",
       "\n",
       "## Related links\n",
       "* https://datasetsearch.research.google.com\n",
       "* https://github.com/sebastianruder/NLP-progress\n",
       "* https://docs.google.com/document/d/1nyQeuP1nY96UYkvbJKBIxxE-BjZ0IGxgifaF4gkvlDc/edit\n",
       "\n",
       "## TODO\n",
       "Twitter & NIPS from https://github.com/ilirhin/python_artm/tree/master/pyartm_datasets/pyartm_datasets\n",
       "\n",
       "Victorian Era\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(api.get_info()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<topicnet.cooking_machine.dataset.Dataset at 0x7f88e41d3908>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "api.load_dataset('NIPS')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<topicnet.cooking_machine.dataset.Dataset at 0x7f88e41d35c0>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "api.load_dataset('lenta')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:artm10] *",
   "language": "python",
   "name": "conda-env-artm10-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}