00_mount.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Upload CrossRef to a local database\n",
    "\n",
    "CrossRef have kindly made all of their 'works' database of publication data available for free via bit-torrent. Once we have downloaded the data, it's a good idea to upload it to a database so that we can 'index' the data. \n",
    "\n",
    "_Indexing_ gives the database a framework to 'look up' records instantly instead of having to scan over the whole database to look for things. They also allow us to create custom queries, like 'get all the records for journal articles published by a particular publisher in 2020'. That kind of thing.\n",
    "\n",
    "# Acquire data\n",
    "The first step here is to get the CrossRef data dump. You will need a bit-torrent client and then follow the instructions in the [CrossRef blog](\"https://www.crossref.org/blog/free-public-data-file-of-112-million-crossref-records/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import all the packages we need\n",
    "import pymongo\n",
    "import json\n",
    "import gzip\n",
    "import glob\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First, take a look at the data. What does it look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list all the files in our data-dump\n",
    "loc = r\"D:\\crossref\\datadump\\crossref\\*\"\n",
    "filepaths = glob.glob(loc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'D:\\\\crossref\\\\datadump\\\\crossref\\\\0.json.gz'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# example filepath\n",
    "filepaths[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example open a file and look inside\n",
    "with gzip.open(filepaths[0], 'r') as f:\n",
    "    data = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3000"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data['items'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We find that the data is one big JSON object with 'items' as a key and then a list of 3000 CrossRef 'works' records.  Works records generally pertain to published research papers, but occasionally other documents, too. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'indexed': {'date-parts': [[2019, 11, 19]],\n",
       "  'date-time': '2019-11-19T17:15:40Z',\n",
       "  'timestamp': 1574183740979},\n",
       " 'reference-count': 0,\n",
       " 'publisher': 'American Medical Association (AMA)',\n",
       " 'issue': '4',\n",
       " 'content-domain': {'domain': [], 'crossmark-restriction': False},\n",
       " 'short-container-title': ['Archives of Internal Medicine'],\n",
       " 'published-print': {'date-parts': [[2006, 2, 27]]},\n",
       " 'DOI': '10.1001/.389',\n",
       " 'type': 'journal-article',\n",
       " 'created': {'date-parts': [[2006, 2, 27]],\n",
       "  'date-time': '2006-02-27T21:28:23Z',\n",
       "  'timestamp': 1141075703000},\n",
       " 'page': '389-390',\n",
       " 'source': 'Crossref',\n",
       " 'is-referenced-by-count': 0,\n",
       " 'title': ['Decision Making at the Fringe of Evidence: Take What You Can Get'],\n",
       " 'prefix': '10.1001',\n",
       " 'volume': '166',\n",
       " 'author': [{'given': 'N. F.', 'family': 'Col', 'affiliation': []}],\n",
       " 'member': '10',\n",
       " 'container-title': ['Archives of Internal Medicine'],\n",
       " 'original-title': [],\n",
       " 'deposited': {'date-parts': [[2007, 2, 13]],\n",
       "  'date-time': '2007-02-13T20:56:13Z',\n",
       "  'timestamp': 1171400173000},\n",
       " 'score': None,\n",
       " 'subtitle': [],\n",
       " 'short-title': [],\n",
       " 'issued': {'date-parts': [[2006, 2, 27]]},\n",
       " 'references-count': 0,\n",
       " 'URL': 'http://dx.doi.org/10.1001/.389',\n",
       " 'relation': {},\n",
       " 'ISSN': ['0003-9926'],\n",
       " 'issn-type': [{'value': '0003-9926', 'type': 'print'}]}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# example works record\n",
    "data['items'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we assume that the structure of all of the JSON files in our data dump is the same, then we can build a function to load all of that data into our database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a class to iterate through the data\n",
    "\n",
    "class CRDump():\n",
    "    loc = r\"D:\\crossref\\datadump\\crossref\\*\"\n",
    "    filepaths = glob.glob(loc)\n",
    "    def __iter__(self):\n",
    "        \"\"\"\n",
    "        The class is an object a bit like a list - in that we can iterate over it.\n",
    "        This function defines how that iteration happens. \n",
    "        \"\"\"\n",
    "        for filepath in filepaths:\n",
    "            with gzip.open(filepath, 'r') as f:\n",
    "                data = json.load(f)\n",
    "                items = data['items']\n",
    "                for item in items:\n",
    "                    try:\n",
    "                        doi = item['DOI']\n",
    "                        yield doi,item\n",
    "                    except:\n",
    "                        print('Fail')\n",
    "                        print(item)\n",
    "                        pass\n",
    "                \n",
    "cr_iterator = CRDump()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dump into MongoDB\n",
    "\n",
    "MongoDB is a popular NoSQL database which is useful for storing and indexing large quantities of JSON data. \n",
    "\n",
    "Other databases may also be worth trying:\n",
    "- Other NoSQL databases include neo4j, or elasticsearch. These are recommended if you need to do complex queries on the data.\n",
    "- PostgresSQL also allows indexing of JSON as well as being a sophisticated SQL db\n",
    "\n",
    "## Setting up MongoDB\n",
    "First, <a href='https://www.mongodb.com/community'>download and install MongoDB community edition.</a>\n",
    "\n",
    "You will need to create a folder 'data\\db' in the top-level of a hard drive. E.g. create `D:\\data\\db`, or `C:\\data\\db`\n",
    "\n",
    "Finally, you might need to start a MongoDB server, by opening a cmd window and entering `mongod`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a client (this is your connection to the mongod service you started)\n",
    "client = pymongo.MongoClient(\"mongodb://localhost:27017/\")\n",
    "# create or connect to a mongodb database via the client\n",
    "db = client[\"crossref\"]\n",
    "# create or connect to a table (or 'collection') in that mongodb database\n",
    "collection = db[\"works\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\AdamD\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:1: DeprecationWarning: database_names is deprecated. Use list_database_names instead.\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['admin', 'config', 'crossref', 'local']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check which databases already exist - note that some dbs are created by default when you install mongodb\n",
    "client.database_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Collection(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True), 'crossref'), 'works')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Upload data to MongoDB\n",
    "- pick a batch size to upload records in (I find 1000 seems optimal)\n",
    "- then run this cell to upload the data to MongoDB. <span style=\"color:red\">This took around 11+1/2 hrs on my home computer, so be prepared to wait!</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-04-14 14:30:41.158039 100000  records written to Mongo\n",
      "2020-04-14 14:31:09.000146 200000  records written to Mongo\n",
      "2020-04-14 14:31:35.101790 300000  records written to Mongo\n",
      "2020-04-14 14:32:00.490812 400000  records written to Mongo\n",
      "2020-04-14 14:32:27.754762 500000  records written to Mongo\n",
      "2020-04-14 14:32:51.848780 600000  records written to Mongo\n",
      "2020-04-14 14:33:18.059584 700000  records written to Mongo\n",
      "2020-04-14 14:33:41.828271 800000  records written to Mongo\n",
      "2020-04-14 14:34:07.855025 900000  records written to Mongo\n",
      "2020-04-14 14:34:33.314709 1000000  records written to Mongo\n",
      "2020-04-14 14:34:58.051775 1100000  records written to Mongo\n",
      "2020-04-14 14:35:22.821386 1200000  records written to Mongo\n",
      "2020-04-14 14:35:47.859687 1300000  records written to Mongo\n",
      "2020-04-14 14:36:12.015898 1400000  records written to Mongo\n",
      "2020-04-14 14:36:38.383670 1500000  records written to Mongo\n",
      "2020-04-14 14:37:04.656983 1600000  records written to Mongo\n",
      "2020-04-14 14:37:29.772917 1700000  records written to Mongo\n",
      "2020-04-14 14:37:57.004942 1800000  records written to Mongo\n",
      "2020-04-14 14:38:23.733763 1900000  records written to Mongo\n",
      "2020-04-14 14:38:50.301094 2000000  records written to Mongo\n",
      "2020-04-14 14:39:16.708347 2100000  records written to Mongo\n",
      "2020-04-14 14:39:42.171869 2200000  records written to Mongo\n",
      "2020-04-14 14:40:07.896707 2300000  records written to Mongo\n",
      "2020-04-14 14:40:33.313063 2400000  records written to Mongo\n",
      "2020-04-14 14:41:00.214594 2500000  records written to Mongo\n",
      "2020-04-14 14:41:26.768353 2600000  records written to Mongo\n",
      "2020-04-14 14:41:53.704204 2700000  records written to Mongo\n",
      "2020-04-14 14:42:19.569158 2800000  records written to Mongo\n",
      "2020-04-14 14:42:47.362346 2900000  records written to Mongo\n",
      "2020-04-14 14:43:13.375060 3000000  records written to Mongo\n",
      "2020-04-14 14:43:38.631177 3100000  records written to Mongo\n",
      "2020-04-14 14:44:04.173486 3200000  records written to Mongo\n",
      "2020-04-14 14:44:29.740962 3300000  records written to Mongo\n",
      "2020-04-14 14:44:55.261513 3400000  records written to Mongo\n",
      "2020-04-14 14:45:20.039191 3500000  records written to Mongo\n",
      "2020-04-14 14:45:46.941641 3600000  records written to Mongo\n",
      "2020-04-14 14:46:12.175822 3700000  records written to Mongo\n",
      "2020-04-14 14:46:38.123695 3800000  records written to Mongo\n",
      "2020-04-14 14:47:04.623020 3900000  records written to Mongo\n",
      "2020-04-14 14:47:29.310544 4000000  records written to Mongo\n",
      "2020-04-14 14:47:54.101348 4100000  records written to Mongo\n",
      "2020-04-14 14:48:20.108300 4200000  records written to Mongo\n",
      "2020-04-14 14:48:46.291463 4300000  records written to Mongo\n",
      "2020-04-14 14:49:13.043606 4400000  records written to Mongo\n",
      "2020-04-14 14:49:40.048369 4500000  records written to Mongo\n",
      "2020-04-14 14:50:19.728894 4600000  records written to Mongo\n",
      "2020-04-14 14:51:47.397120 4700000  records written to Mongo\n",
      "2020-04-14 14:52:36.503728 4800000  records written to Mongo\n",
      "2020-04-14 14:53:05.411745 4900000  records written to Mongo\n",
      "2020-04-14 14:53:34.776221 5000000  records written to Mongo\n",
      "2020-04-14 14:53:57.380016 5100000  records written to Mongo\n",
      "2020-04-14 14:54:20.147723 5200000  records written to Mongo\n",
      "2020-04-14 14:55:51.550600 5300000  records written to Mongo\n",
      "2020-04-14 14:56:27.416598 5400000  records written to Mongo\n",
      "2020-04-14 14:57:06.811396 5500000  records written to Mongo\n",
      "2020-04-14 14:57:38.663387 5600000  records written to Mongo\n",
      "2020-04-14 14:58:12.540758 5700000  records written to Mongo\n",
      "2020-04-14 14:58:48.032543 5800000  records written to Mongo\n",
      "2020-04-14 14:59:27.349193 5900000  records written to Mongo\n",
      "2020-04-14 14:59:59.471238 6000000  records written to Mongo\n",
      "2020-04-14 15:00:39.365079 6100000  records written to Mongo\n",
      "2020-04-14 15:01:18.056380 6200000  records written to Mongo\n",
      "2020-04-14 15:01:57.638680 6300000  records written to Mongo\n",
      "2020-04-14 15:02:39.513332 6400000  records written to Mongo\n",
      "2020-04-14 15:03:15.194071 6500000  records written to Mongo\n",
      "2020-04-14 15:03:55.471315 6600000  records written to Mongo\n",
      "2020-04-14 15:04:34.087632 6700000  records written to Mongo\n",
      "2020-04-14 15:05:07.199203 6800000  records written to Mongo\n",
      "2020-04-14 15:05:41.649410 6900000  records written to Mongo\n",
      "2020-04-14 15:06:17.240423 7000000  records written to Mongo\n",
      "2020-04-14 15:06:52.249040 7100000  records written to Mongo\n",
      "2020-04-14 15:07:24.640202 7200000  records written to Mongo\n",
      "2020-04-14 15:07:51.883347 7300000  records written to Mongo\n",
      "2020-04-14 15:08:25.061092 7400000  records written to Mongo\n",
      "2020-04-14 15:08:51.906177 7500000  records written to Mongo\n",
      "2020-04-14 15:09:22.098155 7600000  records written to Mongo\n",
      "2020-04-14 15:09:48.943155 7700000  records written to Mongo\n",
      "2020-04-14 15:10:17.547307 7800000  records written to Mongo\n",
      "2020-04-14 15:10:44.439339 7900000  records written to Mongo\n",
      "2020-04-14 15:11:13.229555 8000000  records written to Mongo\n",
      "2020-04-14 15:11:41.858828 8100000  records written to Mongo\n",
      "2020-04-14 15:12:09.221824 8200000  records written to Mongo\n",
      "2020-04-14 15:12:36.671806 8300000  records written to Mongo\n",
      "2020-04-14 15:13:06.739868 8400000  records written to Mongo\n",
      "2020-04-14 15:13:37.401956 8500000  records written to Mongo\n",
      "2020-04-14 15:14:02.619842 8600000  records written to Mongo\n",
      "2020-04-14 15:14:27.434356 8700000  records written to Mongo\n",
      "2020-04-14 15:14:53.263838 8800000  records written to Mongo\n",
      "2020-04-14 15:15:27.118074 8900000  records written to Mongo\n",
      "2020-04-14 15:16:48.630369 9000000  records written to Mongo\n",
      "2020-04-14 15:17:44.444340 9100000  records written to Mongo\n",
      "2020-04-14 15:18:24.652369 9200000  records written to Mongo\n",
      "2020-04-14 15:18:51.289788 9300000  records written to Mongo\n",
      "2020-04-14 15:19:19.476542 9400000  records written to Mongo\n",
      "2020-04-14 15:19:42.833046 9500000  records written to Mongo\n",
      "2020-04-14 15:20:06.264003 9600000  records written to Mongo\n",
      "2020-04-14 15:20:30.691427 9700000  records written to Mongo\n",
      "2020-04-14 15:20:54.091446 9800000  records written to Mongo\n",
      "2020-04-14 15:21:20.145277 9900000  records written to Mongo\n",
      "2020-04-14 15:21:47.104433 10000000  records written to Mongo\n",
      "2020-04-14 15:22:19.116099 10100000  records written to Mongo\n",
      "2020-04-14 15:22:50.937067 10200000  records written to Mongo\n",
      "2020-04-14 15:23:28.375065 10300000  records written to Mongo\n",
      "2020-04-14 15:24:04.635597 10400000  records written to Mongo\n",
      "2020-04-14 15:24:54.357478 10500000  records written to Mongo\n",
      "2020-04-14 15:25:56.821727 10600000  records written to Mongo\n",
      "2020-04-14 15:26:56.011726 10700000  records written to Mongo\n",
      "2020-04-14 15:27:56.296738 10800000  records written to Mongo\n",
      "2020-04-14 15:29:32.900561 10900000  records written to Mongo\n",
      "2020-04-14 15:31:13.208274 11000000  records written to Mongo\n",
      "2020-04-14 15:31:41.295562 11100000  records written to Mongo\n",
      "2020-04-14 15:32:13.921986 11200000  records written to Mongo\n",
      "2020-04-14 15:33:01.156505 11300000  records written to Mongo\n",
      "2020-04-14 15:34:22.477238 11400000  records written to Mongo\n",
      "2020-04-14 15:34:46.671256 11500000  records written to Mongo\n",
      "2020-04-14 15:35:48.714577 11600000  records written to Mongo\n",
      "2020-04-14 15:36:38.464428 11700000  records written to Mongo\n",
      "2020-04-14 15:37:56.465157 11800000  records written to Mongo\n",
      "2020-04-14 15:39:18.749537 11900000  records written to Mongo\n",
      "2020-04-14 15:40:48.119155 12000000  records written to Mongo\n",
      "2020-04-14 15:41:19.793230 12100000  records written to Mongo\n",
      "2020-04-14 15:41:56.997143 12200000  records written to Mongo\n",
      "2020-04-14 15:42:39.382726 12300000  records written to Mongo\n",
      "2020-04-14 15:43:26.022649 12400000  records written to Mongo\n",
      "2020-04-14 15:44:18.983994 12500000  records written to Mongo\n",
      "2020-04-14 15:45:00.205185 12600000  records written to Mongo\n",
      "2020-04-14 15:45:26.930398 12700000  records written to Mongo\n",
      "2020-04-14 15:46:07.697166 12800000  records written to Mongo\n",
      "2020-04-14 15:46:51.563389 12900000  records written to Mongo\n",
      "2020-04-14 15:47:30.517060 13000000  records written to Mongo\n",
      "2020-04-14 15:48:01.969632 13100000  records written to Mongo\n",
      "2020-04-14 15:48:47.350969 13200000  records written to Mongo\n",
      "2020-04-14 15:49:13.694228 13300000  records written to Mongo\n",
      "2020-04-14 15:49:37.517902 13400000  records written to Mongo\n",
      "2020-04-14 15:50:00.538355 13500000  records written to Mongo\n",
      "2020-04-14 15:50:26.217321 13600000  records written to Mongo\n",
      "2020-04-14 15:50:53.626303 13700000  records written to Mongo\n",
      "2020-04-14 15:51:19.669028 13800000  records written to Mongo\n",
      "2020-04-14 15:51:48.665120 13900000  records written to Mongo\n",
      "2020-04-14 15:52:14.194607 14000000  records written to Mongo\n",
      "2020-04-14 15:52:40.344105 14100000  records written to Mongo\n",
      "2020-04-14 15:53:05.499255 14200000  records written to Mongo\n",
      "2020-04-14 15:53:28.779984 14300000  records written to Mongo\n",
      "2020-04-14 15:53:53.583209 14400000  records written to Mongo\n",
      "2020-04-14 15:54:32.464681 14500000  records written to Mongo\n",
      "2020-04-14 15:55:19.104486 14600000  records written to Mongo\n",
      "2020-04-14 15:55:56.142425 14700000  records written to Mongo\n",
      "2020-04-14 15:56:42.189623 14800000  records written to Mongo\n",
      "2020-04-14 15:57:25.417129 14900000  records written to Mongo\n",
      "2020-04-14 15:57:55.567732 15000000  records written to Mongo\n",
      "2020-04-14 15:58:30.392762 15100000  records written to Mongo\n",
      "2020-04-14 15:59:01.528636 15200000  records written to Mongo\n",
      "2020-04-14 15:59:35.665738 15300000  records written to Mongo\n",
      "2020-04-14 16:00:12.994419 15400000  records written to Mongo\n",
      "2020-04-14 16:00:52.046104 15500000  records written to Mongo\n",
      "2020-04-14 16:01:23.316435 15600000  records written to Mongo\n",
      "2020-04-14 16:02:00.603930 15700000  records written to Mongo\n",
      "2020-04-14 16:02:43.590807 15800000  records written to Mongo\n",
      "2020-04-14 16:03:12.434287 15900000  records written to Mongo\n",
      "2020-04-14 16:03:45.839953 16000000  records written to Mongo\n",
      "2020-04-14 16:04:28.031895 16100000  records written to Mongo\n",
      "2020-04-14 16:05:36.947895 16200000  records written to Mongo\n",
      "2020-04-14 16:06:19.721445 16300000  records written to Mongo\n",
      "2020-04-14 16:07:17.287926 16400000  records written to Mongo\n",
      "2020-04-14 16:07:53.305779 16500000  records written to Mongo\n",
      "2020-04-14 16:08:28.798437 16600000  records written to Mongo\n",
      "2020-04-14 16:09:02.831770 16700000  records written to Mongo\n",
      "2020-04-14 16:09:37.510212 16800000  records written to Mongo\n",
      "2020-04-14 16:10:10.297868 16900000  records written to Mongo\n",
      "2020-04-14 16:10:52.148481 17000000  records written to Mongo\n",
      "2020-04-14 16:11:26.465143 17100000  records written to Mongo\n",
      "2020-04-14 16:12:04.744367 17200000  records written to Mongo\n",
      "2020-04-14 16:12:42.261604 17300000  records written to Mongo\n",
      "2020-04-14 16:13:19.719681 17400000  records written to Mongo\n",
      "2020-04-14 16:14:06.277312 17500000  records written to Mongo\n",
      "2020-04-14 16:14:43.636557 17600000  records written to Mongo\n",
      "2020-04-14 16:15:29.988554 17700000  records written to Mongo\n",
      "2020-04-14 16:16:11.291651 17800000  records written to Mongo\n",
      "2020-04-14 16:16:55.500335 17900000  records written to Mongo\n",
      "2020-04-14 16:17:41.509538 18000000  records written to Mongo\n",
      "2020-04-14 16:18:22.509535 18100000  records written to Mongo\n",
      "2020-04-14 16:18:59.439547 18200000  records written to Mongo\n",
      "2020-04-14 16:19:34.575593 18300000  records written to Mongo\n",
      "2020-04-14 16:20:14.513570 18400000  records written to Mongo\n",
      "2020-04-14 16:20:59.888502 18500000  records written to Mongo\n",
      "2020-04-14 16:21:38.869809 18600000  records written to Mongo\n",
      "2020-04-14 16:22:26.553345 18700000  records written to Mongo\n",
      "2020-04-14 16:23:13.228456 18800000  records written to Mongo\n",
      "2020-04-14 16:23:53.856180 18900000  records written to Mongo\n",
      "2020-04-14 16:24:39.338637 19000000  records written to Mongo\n",
      "2020-04-14 16:25:27.679525 19100000  records written to Mongo\n",
      "2020-04-14 16:26:09.079522 19200000  records written to Mongo\n",
      "2020-04-14 16:26:56.170508 19300000  records written to Mongo\n",
      "2020-04-14 16:27:38.821475 19400000  records written to Mongo\n",
      "2020-04-14 16:28:18.442803 19500000  records written to Mongo\n",
      "2020-04-14 16:29:06.453230 19600000  records written to Mongo\n",
      "2020-04-14 16:29:54.419090 19700000  records written to Mongo\n",
      "2020-04-14 16:30:31.828823 19800000  records written to Mongo\n",
      "2020-04-14 16:31:18.031311 19900000  records written to Mongo\n",
      "2020-04-14 16:32:00.441028 20000000  records written to Mongo\n",
      "2020-04-14 16:32:41.221710 20100000  records written to Mongo\n",
      "2020-04-14 16:33:30.614777 20200000  records written to Mongo\n",
      "2020-04-14 16:34:15.667494 20300000  records written to Mongo\n",
      "2020-04-14 16:34:57.948264 20400000  records written to Mongo\n",
      "2020-04-14 16:35:38.922472 20500000  records written to Mongo\n",
      "2020-04-14 16:35:59.750820 20600000  records written to Mongo\n",
      "2020-04-14 16:36:27.170751 20700000  records written to Mongo\n",
      "2020-04-14 16:36:56.911729 20800000  records written to Mongo\n",
      "2020-04-14 16:37:28.895457 20900000  records written to Mongo\n",
      "2020-04-14 16:37:55.633850 21000000  records written to Mongo\n",
      "2020-04-14 16:38:19.579087 21100000  records written to Mongo\n",
      "2020-04-14 16:38:59.836176 21200000  records written to Mongo\n",
      "2020-04-14 16:39:36.811050 21300000  records written to Mongo\n",
      "2020-04-14 16:40:21.783914 21400000  records written to Mongo\n",
      "2020-04-14 16:41:01.349310 21500000  records written to Mongo\n",
      "2020-04-14 16:41:45.868668 21600000  records written to Mongo\n",
      "2020-04-14 16:42:38.350222 21700000  records written to Mongo\n",
      "2020-04-14 16:43:20.981188 21800000  records written to Mongo\n",
      "2020-04-14 16:44:06.414010 21900000  records written to Mongo\n",
      "2020-04-14 16:44:49.607530 22000000  records written to Mongo\n",
      "2020-04-14 16:45:19.715211 22100000  records written to Mongo\n",
      "2020-04-14 16:45:51.276638 22200000  records written to Mongo\n",
      "2020-04-14 16:46:14.896942 22300000  records written to Mongo\n",
      "2020-04-14 16:46:43.973768 22400000  records written to Mongo\n",
      "2020-04-14 16:47:06.257921 22500000  records written to Mongo\n",
      "2020-04-14 16:47:32.179320 22600000  records written to Mongo\n",
      "2020-04-14 16:48:04.830497 22700000  records written to Mongo\n",
      "2020-04-14 16:48:32.079453 22800000  records written to Mongo\n",
      "2020-04-14 16:49:04.927316 22900000  records written to Mongo\n",
      "2020-04-14 16:49:27.182297 23000000  records written to Mongo\n",
      "2020-04-14 16:50:03.233505 23100000  records written to Mongo\n",
      "2020-04-14 16:50:58.687417 23200000  records written to Mongo\n",
      "2020-04-14 16:51:35.997366 23300000  records written to Mongo\n",
      "2020-04-14 16:52:11.661451 23400000  records written to Mongo\n",
      "2020-04-14 16:52:49.243270 23500000  records written to Mongo\n",
      "2020-04-14 16:53:20.458182 23600000  records written to Mongo\n",
      "2020-04-14 16:53:49.577346 23700000  records written to Mongo\n",
      "2020-04-14 16:54:32.847705 23800000  records written to Mongo\n",
      "2020-04-14 16:55:12.880058 23900000  records written to Mongo\n",
      "2020-04-14 16:55:49.334109 24000000  records written to Mongo\n",
      "2020-04-14 16:56:28.135321 24100000  records written to Mongo\n",
      "2020-04-14 16:57:08.185750 24200000  records written to Mongo\n",
      "2020-04-14 16:57:37.383060 24300000  records written to Mongo\n",
      "2020-04-14 16:58:05.178136 24400000  records written to Mongo\n",
      "2020-04-14 16:58:59.059231 24500000  records written to Mongo\n",
      "2020-04-14 16:59:33.240729 24600000  records written to Mongo\n",
      "2020-04-14 17:00:03.782395 24700000  records written to Mongo\n",
      "2020-04-14 17:00:31.109356 24800000  records written to Mongo\n",
      "2020-04-14 17:00:56.962386 24900000  records written to Mongo\n",
      "2020-04-14 17:01:22.517682 25000000  records written to Mongo\n",
      "2020-04-14 17:01:47.101245 25100000  records written to Mongo\n",
      "2020-04-14 17:02:10.799755 25200000  records written to Mongo\n",
      "2020-04-14 17:02:36.726103 25300000  records written to Mongo\n",
      "2020-04-14 17:03:03.128647 25400000  records written to Mongo\n",
      "2020-04-14 17:03:40.906004 25500000  records written to Mongo\n",
      "2020-04-14 17:04:06.830018 25600000  records written to Mongo\n",
      "2020-04-14 17:04:41.763904 25700000  records written to Mongo\n",
      "2020-04-14 17:05:13.159099 25800000  records written to Mongo\n",
      "2020-04-14 17:05:35.999672 25900000  records written to Mongo\n",
      "2020-04-14 17:06:02.818251 26000000  records written to Mongo\n",
      "2020-04-14 17:06:43.396518 26100000  records written to Mongo\n",
      "2020-04-14 17:07:09.233095 26200000  records written to Mongo\n",
      "2020-04-14 17:07:35.687666 26300000  records written to Mongo\n",
      "2020-04-14 17:08:01.519750 26400000  records written to Mongo\n",
      "2020-04-14 17:08:27.164247 26500000  records written to Mongo\n",
      "2020-04-14 17:08:54.488025 26600000  records written to Mongo\n",
      "2020-04-14 17:09:19.297532 26700000  records written to Mongo\n",
      "2020-04-14 17:09:49.517241 26800000  records written to Mongo\n",
      "2020-04-14 17:10:13.799344 26900000  records written to Mongo\n",
      "2020-04-14 17:10:38.123599 27000000  records written to Mongo\n",
      "2020-04-14 17:11:04.296156 27100000  records written to Mongo\n",
      "2020-04-14 17:11:29.996333 27200000  records written to Mongo\n",
      "2020-04-14 17:11:58.405305 27300000  records written to Mongo\n",
      "2020-04-14 17:12:23.522892 27400000  records written to Mongo\n",
      "2020-04-14 17:12:51.071920 27500000  records written to Mongo\n",
      "2020-04-14 17:13:18.610472 27600000  records written to Mongo\n",
      "2020-04-14 17:13:46.841476 27700000  records written to Mongo\n",
      "2020-04-14 17:14:16.880358 27800000  records written to Mongo\n",
      "2020-04-14 17:14:43.896948 27900000  records written to Mongo\n",
      "2020-04-14 17:15:13.981261 28000000  records written to Mongo\n",
      "2020-04-14 17:15:54.258715 28100000  records written to Mongo\n",
      "2020-04-14 17:16:36.581603 28200000  records written to Mongo\n",
      "2020-04-14 17:17:41.593044 28300000  records written to Mongo\n",
      "2020-04-14 17:18:13.802332 28400000  records written to Mongo\n",
      "2020-04-14 17:19:06.968704 28500000  records written to Mongo\n",
      "2020-04-14 17:19:53.883397 28600000  records written to Mongo\n",
      "2020-04-14 17:20:40.880389 28700000  records written to Mongo\n",
      "2020-04-14 17:21:29.387432 28800000  records written to Mongo\n",
      "2020-04-14 17:22:25.657947 28900000  records written to Mongo\n",
      "2020-04-14 17:23:19.956793 29000000  records written to Mongo\n",
      "2020-04-14 17:24:06.922663 29100000  records written to Mongo\n",
      "2020-04-14 17:24:36.321637 29200000  records written to Mongo\n",
      "2020-04-14 17:25:05.021258 29300000  records written to Mongo\n",
      "2020-04-14 17:25:31.783078 29400000  records written to Mongo\n",
      "2020-04-14 17:26:08.385338 29500000  records written to Mongo\n",
      "2020-04-14 17:26:44.654990 29600000  records written to Mongo\n",
      "2020-04-14 17:27:22.355518 29700000  records written to Mongo\n",
      "2020-04-14 17:28:04.816672 29800000  records written to Mongo\n",
      "2020-04-14 17:28:45.773527 29900000  records written to Mongo\n",
      "2020-04-14 17:29:16.410084 30000000  records written to Mongo\n",
      "2020-04-14 17:29:44.997852 30100000  records written to Mongo\n",
      "2020-04-14 17:30:10.246675 30200000  records written to Mongo\n",
      "2020-04-14 17:30:35.254652 30300000  records written to Mongo\n",
      "2020-04-14 17:30:59.779652 30400000  records written to Mongo\n",
      "2020-04-14 17:31:27.450934 30500000  records written to Mongo\n",
      "2020-04-14 17:31:53.816577 30600000  records written to Mongo\n",
      "2020-04-14 17:32:23.512664 30700000  records written to Mongo\n",
      "2020-04-14 17:32:51.484709 30800000  records written to Mongo\n",
      "2020-04-14 17:33:16.785396 30900000  records written to Mongo\n",
      "2020-04-14 17:33:44.168352 31000000  records written to Mongo\n",
      "2020-04-14 17:34:10.633428 31100000  records written to Mongo\n",
      "2020-04-14 17:34:35.099000 31200000  records written to Mongo\n",
      "2020-04-14 17:35:00.152810 31300000  records written to Mongo\n",
      "2020-04-14 17:35:24.946359 31400000  records written to Mongo\n",
      "2020-04-14 17:35:51.247362 31500000  records written to Mongo\n",
      "2020-04-14 17:36:16.100756 31600000  records written to Mongo\n",
      "2020-04-14 17:36:40.663783 31700000  records written to Mongo\n",
      "2020-04-14 17:37:05.549419 31800000  records written to Mongo\n",
      "2020-04-14 17:37:29.915995 31900000  records written to Mongo\n",
      "2020-04-14 17:37:54.992110 32000000  records written to Mongo\n",
      "2020-04-14 17:38:19.389017 32100000  records written to Mongo\n",
      "2020-04-14 17:38:43.937017 32200000  records written to Mongo\n",
      "2020-04-14 17:39:08.468052 32300000  records written to Mongo\n",
      "2020-04-14 17:39:32.942030 32400000  records written to Mongo\n",
      "2020-04-14 17:39:57.200225 32500000  records written to Mongo\n",
      "2020-04-14 17:40:22.620678 32600000  records written to Mongo\n",
      "2020-04-14 17:40:46.775752 32700000  records written to Mongo\n",
      "2020-04-14 17:41:11.836753 32800000  records written to Mongo\n",
      "2020-04-14 17:41:36.776052 32900000  records written to Mongo\n",
      "2020-04-14 17:42:02.016705 33000000  records written to Mongo\n",
      "2020-04-14 17:42:28.326273 33100000  records written to Mongo\n",
      "2020-04-14 17:42:52.438320 33200000  records written to Mongo\n",
      "2020-04-14 17:43:19.602861 33300000  records written to Mongo\n",
      "2020-04-14 17:43:43.435880 33400000  records written to Mongo\n",
      "2020-04-14 17:44:08.409003 33500000  records written to Mongo\n",
      "2020-04-14 17:44:32.805067 33600000  records written to Mongo\n",
      "2020-04-14 17:44:57.547040 33700000  records written to Mongo\n",
      "2020-04-14 17:45:22.631240 33800000  records written to Mongo\n",
      "2020-04-14 17:45:48.614353 33900000  records written to Mongo\n",
      "2020-04-14 17:46:17.754959 34000000  records written to Mongo\n",
      "2020-04-14 17:46:43.955986 34100000  records written to Mongo\n",
      "2020-04-14 17:47:10.843096 34200000  records written to Mongo\n",
      "2020-04-14 17:47:41.590421 34300000  records written to Mongo\n",
      "2020-04-14 17:48:09.955529 34400000  records written to Mongo\n",
      "2020-04-14 17:48:37.637256 34500000  records written to Mongo\n",
      "2020-04-14 17:49:06.161524 34600000  records written to Mongo\n",
      "2020-04-14 17:49:33.856300 34700000  records written to Mongo\n",
      "2020-04-14 17:50:00.280019 34800000  records written to Mongo\n",
      "2020-04-14 17:50:26.997216 34900000  records written to Mongo\n",
      "2020-04-14 17:51:10.598016 35000000  records written to Mongo\n",
      "2020-04-14 17:52:02.356010 35100000  records written to Mongo\n",
      "2020-04-14 17:52:54.935401 35200000  records written to Mongo\n",
      "2020-04-14 17:53:48.878915 35300000  records written to Mongo\n",
      "2020-04-14 17:54:47.152528 35400000  records written to Mongo\n",
      "2020-04-14 17:55:45.634186 35500000  records written to Mongo\n",
      "2020-04-14 17:56:46.502012 35600000  records written to Mongo\n",
      "2020-04-14 17:57:43.150945 35700000  records written to Mongo\n",
      "2020-04-14 17:58:32.952565 35800000  records written to Mongo\n",
      "2020-04-14 17:59:25.085191 35900000  records written to Mongo\n",
      "2020-04-14 18:00:21.465690 36000000  records written to Mongo\n",
      "2020-04-14 18:01:27.369000 36100000  records written to Mongo\n",
      "2020-04-14 18:02:28.758996 36200000  records written to Mongo\n",
      "2020-04-14 18:03:33.084391 36300000  records written to Mongo\n",
      "2020-04-14 18:04:20.514582 36400000  records written to Mongo\n",
      "2020-04-14 18:05:18.185637 36500000  records written to Mongo\n",
      "2020-04-14 18:06:05.639901 36600000  records written to Mongo\n",
      "2020-04-14 18:06:45.189663 36700000  records written to Mongo\n",
      "2020-04-14 18:07:34.564036 36800000  records written to Mongo\n",
      "2020-04-14 18:08:23.979366 36900000  records written to Mongo\n",
      "2020-04-14 18:09:20.820751 37000000  records written to Mongo\n",
      "2020-04-14 18:10:04.268814 37100000  records written to Mongo\n",
      "2020-04-14 18:10:56.368840 37200000  records written to Mongo\n",
      "2020-04-14 18:11:44.296354 37300000  records written to Mongo\n",
      "2020-04-14 18:12:31.209080 37400000  records written to Mongo\n",
      "2020-04-14 18:13:29.471716 37500000  records written to Mongo\n",
      "2020-04-14 18:14:26.174738 37600000  records written to Mongo\n",
      "2020-04-14 18:15:13.170442 37700000  records written to Mongo\n",
      "2020-04-14 18:15:57.585880 37800000  records written to Mongo\n",
      "2020-04-14 18:16:48.970884 37900000  records written to Mongo\n",
      "2020-04-14 18:17:27.325256 38000000  records written to Mongo\n",
      "2020-04-14 18:18:06.718327 38100000  records written to Mongo\n",
      "2020-04-14 18:19:08.263087 38200000  records written to Mongo\n",
      "2020-04-14 18:20:13.194595 38300000  records written to Mongo\n",
      "2020-04-14 18:21:17.857049 38400000  records written to Mongo\n",
      "2020-04-14 18:22:18.383893 38500000  records written to Mongo\n",
      "2020-04-14 18:23:08.443430 38600000  records written to Mongo\n",
      "2020-04-14 18:23:47.457271 38700000  records written to Mongo\n",
      "2020-04-14 18:24:32.267030 38800000  records written to Mongo\n",
      "2020-04-14 18:25:17.828161 38900000  records written to Mongo\n",
      "2020-04-14 18:25:51.607482 39000000  records written to Mongo\n",
      "2020-04-14 18:26:30.547513 39100000  records written to Mongo\n",
      "2020-04-14 18:27:00.951827 39200000  records written to Mongo\n",
      "2020-04-14 18:27:42.579855 39300000  records written to Mongo\n",
      "2020-04-14 18:28:12.148605 39400000  records written to Mongo\n",
      "2020-04-14 18:28:41.344871 39500000  records written to Mongo\n",
      "2020-04-14 18:29:08.520436 39600000  records written to Mongo\n",
      "2020-04-14 18:29:38.095060 39700000  records written to Mongo\n",
      "2020-04-14 18:30:10.135790 39800000  records written to Mongo\n",
      "2020-04-14 18:30:34.987943 39900000  records written to Mongo\n",
      "2020-04-14 18:31:05.503580 40000000  records written to Mongo\n",
      "2020-04-14 18:31:42.153075 40100000  records written to Mongo\n",
      "2020-04-14 18:32:08.288227 40200000  records written to Mongo\n",
      "2020-04-14 18:32:35.049916 40300000  records written to Mongo\n",
      "2020-04-14 18:33:19.556401 40400000  records written to Mongo\n",
      "2020-04-14 18:34:02.208243 40500000  records written to Mongo\n",
      "2020-04-14 18:34:36.303058 40600000  records written to Mongo\n",
      "2020-04-14 18:35:21.756343 40700000  records written to Mongo\n",
      "2020-04-14 18:35:58.785153 40800000  records written to Mongo\n",
      "2020-04-14 18:37:02.489445 40900000  records written to Mongo\n",
      "2020-04-14 18:38:03.131106 41000000  records written to Mongo\n",
      "2020-04-14 18:38:59.006001 41100000  records written to Mongo\n",
      "2020-04-14 18:39:31.804622 41200000  records written to Mongo\n",
      "2020-04-14 18:40:04.144057 41300000  records written to Mongo\n",
      "2020-04-14 18:40:29.322093 41400000  records written to Mongo\n",
      "2020-04-14 18:40:53.017092 41500000  records written to Mongo\n",
      "2020-04-14 18:41:27.034480 41600000  records written to Mongo\n",
      "2020-04-14 18:41:55.450521 41700000  records written to Mongo\n",
      "2020-04-14 18:42:29.279902 41800000  records written to Mongo\n",
      "2020-04-14 18:43:07.115635 41900000  records written to Mongo\n",
      "2020-04-14 18:43:55.826784 42000000  records written to Mongo\n",
      "2020-04-14 18:44:30.956778 42100000  records written to Mongo\n",
      "2020-04-14 18:45:04.632682 42200000  records written to Mongo\n",
      "2020-04-14 18:45:48.302487 42300000  records written to Mongo\n",
      "2020-04-14 18:46:43.973334 42400000  records written to Mongo\n",
      "2020-04-14 18:47:15.273073 42500000  records written to Mongo\n",
      "2020-04-14 18:47:40.384667 42600000  records written to Mongo\n",
      "2020-04-14 18:48:18.907819 42700000  records written to Mongo\n",
      "2020-04-14 18:48:59.734135 42800000  records written to Mongo\n",
      "2020-04-14 18:49:31.370662 42900000  records written to Mongo\n",
      "2020-04-14 18:50:06.712296 43000000  records written to Mongo\n",
      "2020-04-14 18:50:39.399044 43100000  records written to Mongo\n",
      "2020-04-14 18:51:07.963611 43200000  records written to Mongo\n",
      "2020-04-14 18:51:44.286369 43300000  records written to Mongo\n",
      "2020-04-14 18:52:33.398091 43400000  records written to Mongo\n",
      "2020-04-14 18:53:20.996127 43500000  records written to Mongo\n",
      "2020-04-14 18:53:47.331369 43600000  records written to Mongo\n",
      "2020-04-14 18:54:23.166301 43700000  records written to Mongo\n",
      "2020-04-14 18:54:54.442550 43800000  records written to Mongo\n",
      "2020-04-14 18:55:38.781820 43900000  records written to Mongo\n",
      "2020-04-14 18:56:27.741657 44000000  records written to Mongo\n",
      "2020-04-14 18:57:15.795140 44100000  records written to Mongo\n",
      "2020-04-14 18:58:12.512032 44200000  records written to Mongo\n",
      "2020-04-14 18:59:02.565871 44300000  records written to Mongo\n",
      "2020-04-14 18:59:33.842177 44400000  records written to Mongo\n",
      "2020-04-14 19:00:04.458120 44500000  records written to Mongo\n",
      "2020-04-14 19:00:38.820644 44600000  records written to Mongo\n",
      "2020-04-14 19:01:10.434780 44700000  records written to Mongo\n",
      "2020-04-14 19:01:39.371040 44800000  records written to Mongo\n",
      "2020-04-14 19:02:11.650368 44900000  records written to Mongo\n",
      "2020-04-14 19:02:44.422051 45000000  records written to Mongo\n",
      "2020-04-14 19:03:13.835154 45100000  records written to Mongo\n",
      "2020-04-14 19:03:50.561904 45200000  records written to Mongo\n",
      "2020-04-14 19:04:38.969187 45300000  records written to Mongo\n",
      "2020-04-14 19:05:12.952067 45400000  records written to Mongo\n",
      "2020-04-14 19:05:53.200371 45500000  records written to Mongo\n",
      "2020-04-14 19:06:18.821068 45600000  records written to Mongo\n",
      "2020-04-14 19:06:41.150474 45700000  records written to Mongo\n",
      "2020-04-14 19:07:04.372925 45800000  records written to Mongo\n",
      "2020-04-14 19:07:26.657594 45900000  records written to Mongo\n",
      "2020-04-14 19:07:56.841914 46000000  records written to Mongo\n",
      "2020-04-14 19:08:21.401470 46100000  records written to Mongo\n",
      "2020-04-14 19:08:44.454639 46200000  records written to Mongo\n",
      "2020-04-14 19:09:09.307477 46300000  records written to Mongo\n",
      "2020-04-14 19:09:36.469241 46400000  records written to Mongo\n",
      "2020-04-14 19:10:03.213033 46500000  records written to Mongo\n",
      "2020-04-14 19:10:27.471330 46600000  records written to Mongo\n",
      "2020-04-14 19:10:57.182822 46700000  records written to Mongo\n",
      "2020-04-14 19:11:27.870688 46800000  records written to Mongo\n",
      "2020-04-14 19:11:57.463331 46900000  records written to Mongo\n",
      "2020-04-14 19:12:31.975409 47000000  records written to Mongo\n",
      "2020-04-14 19:13:00.913791 47100000  records written to Mongo\n",
      "2020-04-14 19:13:32.255923 47200000  records written to Mongo\n",
      "2020-04-14 19:14:13.210572 47300000  records written to Mongo\n",
      "2020-04-14 19:14:53.137991 47400000  records written to Mongo\n",
      "2020-04-14 19:15:28.304989 47500000  records written to Mongo\n",
      "2020-04-14 19:16:11.288736 47600000  records written to Mongo\n",
      "2020-04-14 19:16:50.280703 47700000  records written to Mongo\n",
      "2020-04-14 19:17:24.461259 47800000  records written to Mongo\n",
      "2020-04-14 19:18:04.878310 47900000  records written to Mongo\n",
      "2020-04-14 19:18:46.430937 48000000  records written to Mongo\n",
      "2020-04-14 19:19:34.238967 48100000  records written to Mongo\n",
      "2020-04-14 19:20:20.261614 48200000  records written to Mongo\n",
      "2020-04-14 19:20:57.861457 48300000  records written to Mongo\n",
      "2020-04-14 19:21:44.028420 48400000  records written to Mongo\n",
      "2020-04-14 19:22:32.333786 48500000  records written to Mongo\n",
      "2020-04-14 19:23:16.226671 48600000  records written to Mongo\n",
      "2020-04-14 19:24:04.790662 48700000  records written to Mongo\n",
      "2020-04-14 19:24:51.548914 48800000  records written to Mongo\n",
      "2020-04-14 19:25:31.239554 48900000  records written to Mongo\n",
      "2020-04-14 19:26:25.026176 49000000  records written to Mongo\n",
      "2020-04-14 19:27:20.113904 49100000  records written to Mongo\n",
      "2020-04-14 19:28:05.616600 49200000  records written to Mongo\n",
      "2020-04-14 19:28:58.083716 49300000  records written to Mongo\n",
      "2020-04-14 19:29:45.039738 49400000  records written to Mongo\n",
      "2020-04-14 19:30:22.691396 49500000  records written to Mongo\n",
      "2020-04-14 19:31:26.794631 49600000  records written to Mongo\n",
      "2020-04-14 19:32:23.065582 49700000  records written to Mongo\n",
      "2020-04-14 19:33:47.473257 49800000  records written to Mongo\n",
      "2020-04-14 19:34:38.902464 49900000  records written to Mongo\n",
      "2020-04-14 19:35:34.054707 50000000  records written to Mongo\n",
      "2020-04-14 19:37:25.475285 50100000  records written to Mongo\n",
      "2020-04-14 19:39:26.152617 50200000  records written to Mongo\n",
      "2020-04-14 19:40:34.274230 50300000  records written to Mongo\n",
      "2020-04-14 19:41:25.155024 50400000  records written to Mongo\n",
      "2020-04-14 19:42:09.104963 50500000  records written to Mongo\n",
      "2020-04-14 19:42:44.560017 50600000  records written to Mongo\n",
      "2020-04-14 19:43:14.066076 50700000  records written to Mongo\n",
      "2020-04-14 19:43:49.851821 50800000  records written to Mongo\n",
      "2020-04-14 19:44:27.063789 50900000  records written to Mongo\n",
      "2020-04-14 19:45:10.804276 51000000  records written to Mongo\n",
      "2020-04-14 19:45:46.955160 51100000  records written to Mongo\n",
      "2020-04-14 19:46:28.749211 51200000  records written to Mongo\n",
      "2020-04-14 19:47:14.014219 51300000  records written to Mongo\n",
      "2020-04-14 19:47:45.508244 51400000  records written to Mongo\n",
      "2020-04-14 19:48:13.998207 51500000  records written to Mongo\n",
      "2020-04-14 19:49:25.211490 51600000  records written to Mongo\n",
      "2020-04-14 19:50:08.474337 51700000  records written to Mongo\n",
      "2020-04-14 19:50:50.796285 51800000  records written to Mongo\n",
      "2020-04-14 19:51:20.221341 51900000  records written to Mongo\n",
      "2020-04-14 19:51:53.123145 52000000  records written to Mongo\n",
      "2020-04-14 19:52:28.449719 52100000  records written to Mongo\n",
      "2020-04-14 19:53:08.889832 52200000  records written to Mongo\n",
      "2020-04-14 19:53:39.602793 52300000  records written to Mongo\n",
      "2020-04-14 19:54:23.376319 52400000  records written to Mongo\n",
      "2020-04-14 19:54:58.968469 52500000  records written to Mongo\n",
      "2020-04-14 19:55:27.615944 52600000  records written to Mongo\n",
      "2020-04-14 19:56:01.641882 52700000  records written to Mongo\n",
      "2020-04-14 19:56:36.629318 52800000  records written to Mongo\n",
      "2020-04-14 19:57:10.160566 52900000  records written to Mongo\n",
      "2020-04-14 19:57:40.335356 53000000  records written to Mongo\n",
      "2020-04-14 19:58:05.420748 53100000  records written to Mongo\n",
      "2020-04-14 19:58:33.083705 53200000  records written to Mongo\n",
      "2020-04-14 19:59:01.937696 53300000  records written to Mongo\n",
      "2020-04-14 19:59:35.389619 53400000  records written to Mongo\n",
      "2020-04-14 20:00:11.326481 53500000  records written to Mongo\n",
      "2020-04-14 20:00:34.508326 53600000  records written to Mongo\n",
      "2020-04-14 20:01:06.161816 53700000  records written to Mongo\n",
      "2020-04-14 20:01:33.413048 53800000  records written to Mongo\n",
      "2020-04-14 20:02:12.454896 53900000  records written to Mongo\n",
      "2020-04-14 20:02:38.767679 54000000  records written to Mongo\n",
      "2020-04-14 20:03:06.362254 54100000  records written to Mongo\n",
      "2020-04-14 20:03:32.080728 54200000  records written to Mongo\n",
      "2020-04-14 20:03:56.941713 54300000  records written to Mongo\n",
      "2020-04-14 20:04:33.798849 54400000  records written to Mongo\n",
      "2020-04-14 20:05:10.593056 54500000  records written to Mongo\n",
      "2020-04-14 20:05:46.757230 54600000  records written to Mongo\n",
      "2020-04-14 20:06:31.024943 54700000  records written to Mongo\n",
      "2020-04-14 20:07:14.302560 54800000  records written to Mongo\n",
      "2020-04-14 20:07:41.671481 54900000  records written to Mongo\n",
      "2020-04-14 20:08:10.039512 55000000  records written to Mongo\n",
      "2020-04-14 20:08:38.490664 55100000  records written to Mongo\n",
      "2020-04-14 20:09:06.741458 55200000  records written to Mongo\n",
      "2020-04-14 20:09:34.250008 55300000  records written to Mongo\n",
      "2020-04-14 20:10:01.706955 55400000  records written to Mongo\n",
      "2020-04-14 20:10:31.184141 55500000  records written to Mongo\n",
      "2020-04-14 20:10:58.743874 55600000  records written to Mongo\n",
      "2020-04-14 20:11:39.297315 55700000  records written to Mongo\n",
      "2020-04-14 20:12:06.493110 55800000  records written to Mongo\n",
      "2020-04-14 20:12:34.700463 55900000  records written to Mongo\n",
      "2020-04-14 20:13:07.770168 56000000  records written to Mongo\n",
      "2020-04-14 20:13:37.500177 56100000  records written to Mongo\n",
      "2020-04-14 20:14:12.690127 56200000  records written to Mongo\n",
      "2020-04-14 20:14:42.200738 56300000  records written to Mongo\n",
      "2020-04-14 20:15:20.101511 56400000  records written to Mongo\n",
      "2020-04-14 20:15:49.356402 56500000  records written to Mongo\n",
      "2020-04-14 20:16:18.262922 56600000  records written to Mongo\n",
      "2020-04-14 20:16:49.451621 56700000  records written to Mongo\n",
      "2020-04-14 20:17:17.463000 56800000  records written to Mongo\n",
      "2020-04-14 20:17:47.784668 56900000  records written to Mongo\n",
      "2020-04-14 20:18:16.078470 57000000  records written to Mongo\n",
      "2020-04-14 20:18:45.378020 57100000  records written to Mongo\n",
      "2020-04-14 20:19:15.188065 57200000  records written to Mongo\n",
      "2020-04-14 20:19:42.674447 57300000  records written to Mongo\n",
      "2020-04-14 20:20:13.183393 57400000  records written to Mongo\n",
      "2020-04-14 20:20:41.609364 57500000  records written to Mongo\n",
      "2020-04-14 20:21:12.065023 57600000  records written to Mongo\n",
      "2020-04-14 20:21:41.304142 57700000  records written to Mongo\n",
      "2020-04-14 20:22:10.913491 57800000  records written to Mongo\n",
      "2020-04-14 20:22:40.527704 57900000  records written to Mongo\n",
      "2020-04-14 20:23:08.583511 58000000  records written to Mongo\n",
      "2020-04-14 20:23:48.052614 58100000  records written to Mongo\n",
      "2020-04-14 20:24:17.824640 58200000  records written to Mongo\n",
      "2020-04-14 20:24:46.583984 58300000  records written to Mongo\n",
      "2020-04-14 20:25:16.296543 58400000  records written to Mongo\n",
      "2020-04-14 20:25:45.469397 58500000  records written to Mongo\n",
      "2020-04-14 20:26:16.688592 58600000  records written to Mongo\n",
      "2020-04-14 20:26:42.696841 58700000  records written to Mongo\n",
      "2020-04-14 20:27:18.297543 58800000  records written to Mongo\n",
      "2020-04-14 20:27:46.395774 58900000  records written to Mongo\n",
      "2020-04-14 20:28:19.775430 59000000  records written to Mongo\n",
      "2020-04-14 20:28:47.370074 59100000  records written to Mongo\n",
      "2020-04-14 20:29:15.714567 59200000  records written to Mongo\n",
      "2020-04-14 20:29:57.195718 59300000  records written to Mongo\n",
      "2020-04-14 20:30:24.548639 59400000  records written to Mongo\n",
      "2020-04-14 20:30:52.831313 59500000  records written to Mongo\n",
      "2020-04-14 20:31:19.489390 59600000  records written to Mongo\n",
      "2020-04-14 20:31:44.873544 59700000  records written to Mongo\n",
      "2020-04-14 20:32:13.021154 59800000  records written to Mongo\n",
      "2020-04-14 20:32:43.228694 59900000  records written to Mongo\n",
      "2020-04-14 20:33:19.557684 60000000  records written to Mongo\n",
      "2020-04-14 20:33:51.244022 60100000  records written to Mongo\n",
      "2020-04-14 20:34:24.244707 60200000  records written to Mongo\n",
      "2020-04-14 20:34:57.366604 60300000  records written to Mongo\n",
      "2020-04-14 20:35:38.686708 60400000  records written to Mongo\n",
      "2020-04-14 20:36:10.413281 60500000  records written to Mongo\n",
      "2020-04-14 20:36:41.769024 60600000  records written to Mongo\n",
      "2020-04-14 20:37:10.221305 60700000  records written to Mongo\n",
      "2020-04-14 20:37:47.783696 60800000  records written to Mongo\n",
      "2020-04-14 20:38:17.328239 60900000  records written to Mongo\n",
      "2020-04-14 20:39:01.949381 61000000  records written to Mongo\n",
      "2020-04-14 20:39:24.117531 61100000  records written to Mongo\n",
      "2020-04-14 20:39:53.039368 61200000  records written to Mongo\n",
      "2020-04-14 20:40:18.470679 61300000  records written to Mongo\n",
      "2020-04-14 20:40:41.463732 61400000  records written to Mongo\n",
      "2020-04-14 20:41:15.227990 61500000  records written to Mongo\n",
      "2020-04-14 20:41:42.377991 61600000  records written to Mongo\n",
      "2020-04-14 20:42:11.843253 61700000  records written to Mongo\n",
      "2020-04-14 20:42:44.995793 61800000  records written to Mongo\n",
      "2020-04-14 20:43:20.047136 61900000  records written to Mongo\n",
      "2020-04-14 20:43:45.442857 62000000  records written to Mongo\n",
      "2020-04-14 20:44:15.153707 62100000  records written to Mongo\n",
      "2020-04-14 20:44:44.727311 62200000  records written to Mongo\n",
      "2020-04-14 20:45:15.881269 62300000  records written to Mongo\n",
      "2020-04-14 20:45:57.670066 62400000  records written to Mongo\n",
      "2020-04-14 20:46:26.113854 62500000  records written to Mongo\n",
      "2020-04-14 20:46:56.894055 62600000  records written to Mongo\n",
      "2020-04-14 20:47:24.140834 62700000  records written to Mongo\n",
      "2020-04-14 20:48:24.416358 62800000  records written to Mongo\n",
      "2020-04-14 20:49:40.999556 62900000  records written to Mongo\n",
      "2020-04-14 20:50:29.570960 63000000  records written to Mongo\n",
      "2020-04-14 20:51:20.741058 63100000  records written to Mongo\n",
      "2020-04-14 20:52:26.725478 63200000  records written to Mongo\n",
      "2020-04-14 20:52:56.751884 63300000  records written to Mongo\n",
      "2020-04-14 20:53:36.417174 63400000  records written to Mongo\n",
      "2020-04-14 20:54:10.113172 63500000  records written to Mongo\n",
      "2020-04-14 20:54:42.196172 63600000  records written to Mongo\n",
      "2020-04-14 20:55:10.619174 63700000  records written to Mongo\n",
      "2020-04-14 20:55:34.542206 63800000  records written to Mongo\n",
      "2020-04-14 20:56:02.953201 63900000  records written to Mongo\n",
      "2020-04-14 20:56:29.378824 64000000  records written to Mongo\n",
      "2020-04-14 20:57:00.688535 64100000  records written to Mongo\n",
      "2020-04-14 20:57:27.822535 64200000  records written to Mongo\n",
      "2020-04-14 20:57:52.959534 64300000  records written to Mongo\n",
      "2020-04-14 20:58:20.830566 64400000  records written to Mongo\n",
      "2020-04-14 20:58:46.724536 64500000  records written to Mongo\n",
      "2020-04-14 20:59:07.082386 64600000  records written to Mongo\n",
      "2020-04-14 20:59:27.479415 64700000  records written to Mongo\n",
      "2020-04-14 20:59:48.898385 64800000  records written to Mongo\n",
      "2020-04-14 21:00:13.361962 64900000  records written to Mongo\n",
      "2020-04-14 21:00:36.768961 65000000  records written to Mongo\n",
      "2020-04-14 21:01:02.430313 65100000  records written to Mongo\n",
      "2020-04-14 21:01:29.991822 65200000  records written to Mongo\n",
      "2020-04-14 21:01:57.163852 65300000  records written to Mongo\n",
      "2020-04-14 21:02:18.781854 65400000  records written to Mongo\n",
      "2020-04-14 21:02:42.889822 65500000  records written to Mongo\n",
      "2020-04-14 21:03:09.336853 65600000  records written to Mongo\n",
      "2020-04-14 21:03:33.884823 65700000  records written to Mongo\n",
      "2020-04-14 21:04:00.364823 65800000  records written to Mongo\n",
      "2020-04-14 21:04:29.634823 65900000  records written to Mongo\n",
      "2020-04-14 21:04:59.793824 66000000  records written to Mongo\n",
      "2020-04-14 21:05:27.560344 66100000  records written to Mongo\n",
      "2020-04-14 21:05:59.476344 66200000  records written to Mongo\n",
      "2020-04-14 21:06:28.203345 66300000  records written to Mongo\n",
      "2020-04-14 21:06:52.750885 66400000  records written to Mongo\n",
      "2020-04-14 21:07:17.634941 66500000  records written to Mongo\n",
      "2020-04-14 21:07:46.837946 66600000  records written to Mongo\n",
      "2020-04-14 21:08:13.220611 66700000  records written to Mongo\n",
      "2020-04-14 21:08:40.384641 66800000  records written to Mongo\n",
      "2020-04-14 21:09:05.813150 66900000  records written to Mongo\n",
      "2020-04-14 21:09:31.672910 67000000  records written to Mongo\n",
      "2020-04-14 21:10:06.516910 67100000  records written to Mongo\n",
      "2020-04-14 21:10:38.837910 67200000  records written to Mongo\n",
      "2020-04-14 21:11:01.724529 67300000  records written to Mongo\n",
      "2020-04-14 21:11:28.006527 67400000  records written to Mongo\n",
      "2020-04-14 21:11:49.028527 67500000  records written to Mongo\n",
      "2020-04-14 21:12:13.396528 67600000  records written to Mongo\n",
      "2020-04-14 21:12:42.342042 67700000  records written to Mongo\n",
      "2020-04-14 21:13:12.803912 67800000  records written to Mongo\n",
      "2020-04-14 21:13:44.041911 67900000  records written to Mongo\n",
      "2020-04-14 21:14:14.147718 68000000  records written to Mongo\n",
      "2020-04-14 21:14:38.265518 68100000  records written to Mongo\n",
      "2020-04-14 21:15:16.739029 68200000  records written to Mongo\n",
      "2020-04-14 21:15:41.731061 68300000  records written to Mongo\n",
      "2020-04-14 21:16:06.593028 68400000  records written to Mongo\n",
      "2020-04-14 21:16:29.600536 68500000  records written to Mongo\n",
      "2020-04-14 21:17:00.250247 68600000  records written to Mongo\n",
      "2020-04-14 21:17:35.028898 68700000  records written to Mongo\n",
      "2020-04-14 21:18:01.438687 68800000  records written to Mongo\n",
      "2020-04-14 21:18:32.922937 68900000  records written to Mongo\n",
      "2020-04-14 21:19:08.104321 69000000  records written to Mongo\n",
      "2020-04-14 21:19:36.128375 69100000  records written to Mongo\n",
      "2020-04-14 21:20:00.013564 69200000  records written to Mongo\n",
      "2020-04-14 21:20:25.090942 69300000  records written to Mongo\n",
      "2020-04-14 21:21:04.815568 69400000  records written to Mongo\n",
      "2020-04-14 21:21:32.533868 69500000  records written to Mongo\n",
      "2020-04-14 21:22:01.278586 69600000  records written to Mongo\n",
      "2020-04-14 21:22:31.006700 69700000  records written to Mongo\n",
      "2020-04-14 21:23:03.826188 69800000  records written to Mongo\n",
      "2020-04-14 21:23:27.969510 69900000  records written to Mongo\n",
      "2020-04-14 21:23:50.774272 70000000  records written to Mongo\n",
      "2020-04-14 21:24:17.509164 70100000  records written to Mongo\n",
      "2020-04-14 21:24:42.856276 70200000  records written to Mongo\n",
      "2020-04-14 21:25:09.484975 70300000  records written to Mongo\n",
      "2020-04-14 21:25:36.768092 70400000  records written to Mongo\n",
      "2020-04-14 21:25:59.764011 70500000  records written to Mongo\n",
      "2020-04-14 21:26:26.748397 70600000  records written to Mongo\n",
      "2020-04-14 21:26:49.258360 70700000  records written to Mongo\n",
      "2020-04-14 21:27:10.766878 70800000  records written to Mongo\n",
      "2020-04-14 21:27:33.203799 70900000  records written to Mongo\n",
      "2020-04-14 21:27:55.299667 71000000  records written to Mongo\n",
      "2020-04-14 21:28:18.002232 71100000  records written to Mongo\n",
      "2020-04-14 21:28:41.469268 71200000  records written to Mongo\n",
      "2020-04-14 21:29:06.815005 71300000  records written to Mongo\n",
      "2020-04-14 21:29:32.637007 71400000  records written to Mongo\n",
      "2020-04-14 21:30:02.505869 71500000  records written to Mongo\n",
      "2020-04-14 21:30:28.188043 71600000  records written to Mongo\n",
      "2020-04-14 21:30:54.238810 71700000  records written to Mongo\n",
      "2020-04-14 21:31:19.898694 71800000  records written to Mongo\n",
      "2020-04-14 21:31:45.022532 71900000  records written to Mongo\n",
      "2020-04-14 21:32:10.885575 72000000  records written to Mongo\n",
      "2020-04-14 21:32:35.127254 72100000  records written to Mongo\n",
      "2020-04-14 21:32:59.391683 72200000  records written to Mongo\n",
      "2020-04-14 21:33:22.393020 72300000  records written to Mongo\n",
      "2020-04-14 21:33:46.027472 72400000  records written to Mongo\n",
      "2020-04-14 21:34:08.509706 72500000  records written to Mongo\n",
      "2020-04-14 21:34:28.177417 72600000  records written to Mongo\n",
      "2020-04-14 21:34:48.008253 72700000  records written to Mongo\n",
      "2020-04-14 21:35:08.742179 72800000  records written to Mongo\n",
      "2020-04-14 21:35:28.511733 72900000  records written to Mongo\n",
      "2020-04-14 21:35:51.608519 73000000  records written to Mongo\n",
      "2020-04-14 21:36:16.627954 73100000  records written to Mongo\n",
      "2020-04-14 21:36:39.387783 73200000  records written to Mongo\n",
      "2020-04-14 21:37:06.754568 73300000  records written to Mongo\n",
      "2020-04-14 21:37:37.811829 73400000  records written to Mongo\n",
      "2020-04-14 21:38:04.459226 73500000  records written to Mongo\n",
      "2020-04-14 21:38:32.433046 73600000  records written to Mongo\n",
      "2020-04-14 21:39:01.202603 73700000  records written to Mongo\n",
      "2020-04-14 21:39:28.867878 73800000  records written to Mongo\n",
      "2020-04-14 21:40:04.014081 73900000  records written to Mongo\n",
      "2020-04-14 21:40:41.012122 74000000  records written to Mongo\n",
      "2020-04-14 21:41:25.920911 74100000  records written to Mongo\n",
      "2020-04-14 21:41:56.585959 74200000  records written to Mongo\n",
      "2020-04-14 21:42:25.569537 74300000  records written to Mongo\n",
      "2020-04-14 21:43:04.023264 74400000  records written to Mongo\n",
      "2020-04-14 21:43:45.557516 74500000  records written to Mongo\n",
      "2020-04-14 21:44:24.312171 74600000  records written to Mongo\n",
      "2020-04-14 21:44:59.217067 74700000  records written to Mongo\n",
      "2020-04-14 21:45:29.366238 74800000  records written to Mongo\n",
      "2020-04-14 21:45:59.479576 74900000  records written to Mongo\n",
      "2020-04-14 21:46:27.595989 75000000  records written to Mongo\n",
      "2020-04-14 21:47:01.976025 75100000  records written to Mongo\n",
      "2020-04-14 21:47:36.692554 75200000  records written to Mongo\n",
      "2020-04-14 21:48:12.972280 75300000  records written to Mongo\n",
      "2020-04-14 21:48:47.384185 75400000  records written to Mongo\n",
      "2020-04-14 21:49:14.827710 75500000  records written to Mongo\n",
      "2020-04-14 21:49:41.766660 75600000  records written to Mongo\n",
      "2020-04-14 21:50:12.625377 75700000  records written to Mongo\n",
      "2020-04-14 21:50:42.287843 75800000  records written to Mongo\n",
      "2020-04-14 21:51:14.838323 75900000  records written to Mongo\n",
      "2020-04-14 21:51:49.363746 76000000  records written to Mongo\n",
      "2020-04-14 21:52:32.509812 76100000  records written to Mongo\n",
      "2020-04-14 21:53:23.711634 76200000  records written to Mongo\n",
      "2020-04-14 21:54:13.797700 76300000  records written to Mongo\n",
      "2020-04-14 21:54:48.479700 76400000  records written to Mongo\n",
      "2020-04-14 21:55:22.342706 76500000  records written to Mongo\n",
      "2020-04-14 21:55:49.945708 76600000  records written to Mongo\n",
      "2020-04-14 21:56:15.115372 76700000  records written to Mongo\n",
      "2020-04-14 21:56:47.783370 76800000  records written to Mongo\n",
      "2020-04-14 21:57:13.445029 76900000  records written to Mongo\n",
      "2020-04-14 21:57:42.639540 77000000  records written to Mongo\n",
      "2020-04-14 21:58:15.624542 77100000  records written to Mongo\n",
      "2020-04-14 21:58:44.403513 77200000  records written to Mongo\n",
      "2020-04-14 21:59:27.754049 77300000  records written to Mongo\n",
      "2020-04-14 21:59:54.608518 77400000  records written to Mongo\n",
      "2020-04-14 22:00:23.918021 77500000  records written to Mongo\n",
      "2020-04-14 22:00:58.574035 77600000  records written to Mongo\n",
      "2020-04-14 22:01:46.009144 77700000  records written to Mongo\n",
      "2020-04-14 22:02:11.832114 77800000  records written to Mongo\n",
      "2020-04-14 22:02:39.372112 77900000  records written to Mongo\n",
      "2020-04-14 22:03:12.240928 78000000  records written to Mongo\n",
      "2020-04-14 22:03:41.400532 78100000  records written to Mongo\n",
      "2020-04-14 22:04:25.117045 78200000  records written to Mongo\n",
      "2020-04-14 22:04:53.489812 78300000  records written to Mongo\n",
      "2020-04-14 22:05:29.708353 78400000  records written to Mongo\n",
      "2020-04-14 22:06:10.288354 78500000  records written to Mongo\n",
      "2020-04-14 22:07:26.054890 78600000  records written to Mongo\n",
      "2020-04-14 22:08:44.152966 78700000  records written to Mongo\n",
      "2020-04-14 22:09:47.743185 78800000  records written to Mongo\n",
      "2020-04-14 22:10:54.172723 78900000  records written to Mongo\n",
      "2020-04-14 22:11:53.083694 79000000  records written to Mongo\n",
      "2020-04-14 22:13:03.097450 79100000  records written to Mongo\n",
      "2020-04-14 22:13:37.185525 79200000  records written to Mongo\n",
      "2020-04-14 22:13:58.326016 79300000  records written to Mongo\n",
      "2020-04-14 22:14:22.171688 79400000  records written to Mongo\n",
      "2020-04-14 22:14:47.334199 79500000  records written to Mongo\n",
      "2020-04-14 22:15:10.885206 79600000  records written to Mongo\n",
      "2020-04-14 22:15:35.526204 79700000  records written to Mongo\n",
      "2020-04-14 22:15:58.089715 79800000  records written to Mongo\n",
      "2020-04-14 22:16:20.428744 79900000  records written to Mongo\n",
      "2020-04-14 22:16:48.950252 80000000  records written to Mongo\n",
      "2020-04-14 22:17:18.305254 80100000  records written to Mongo\n",
      "2020-04-14 22:17:49.141766 80200000  records written to Mongo\n",
      "2020-04-14 22:18:16.961809 80300000  records written to Mongo\n",
      "2020-04-14 22:18:38.433413 80400000  records written to Mongo\n",
      "2020-04-14 22:19:12.975467 80500000  records written to Mongo\n",
      "2020-04-14 22:20:00.048982 80600000  records written to Mongo\n",
      "2020-04-14 22:20:31.957981 80700000  records written to Mongo\n",
      "2020-04-14 22:20:59.217010 80800000  records written to Mongo\n",
      "2020-04-14 22:21:44.418998 80900000  records written to Mongo\n",
      "2020-04-14 22:22:11.901000 81000000  records written to Mongo\n",
      "2020-04-14 22:22:42.117999 81100000  records written to Mongo\n",
      "2020-04-14 22:23:09.744000 81200000  records written to Mongo\n",
      "2020-04-14 22:23:34.392999 81300000  records written to Mongo\n",
      "2020-04-14 22:24:06.574546 81400000  records written to Mongo\n",
      "2020-04-14 22:24:34.237217 81500000  records written to Mongo\n",
      "2020-04-14 22:25:02.989224 81600000  records written to Mongo\n",
      "2020-04-14 22:25:35.020252 81700000  records written to Mongo\n",
      "2020-04-14 22:26:05.444235 81800000  records written to Mongo\n",
      "2020-04-14 22:26:29.073731 81900000  records written to Mongo\n",
      "2020-04-14 22:26:50.421831 82000000  records written to Mongo\n",
      "2020-04-14 22:27:17.021831 82100000  records written to Mongo\n",
      "2020-04-14 22:27:45.651370 82200000  records written to Mongo\n",
      "2020-04-14 22:28:36.383513 82300000  records written to Mongo\n",
      "2020-04-14 22:29:18.158024 82400000  records written to Mongo\n",
      "2020-04-14 22:29:55.735566 82500000  records written to Mongo\n",
      "2020-04-14 22:30:34.170567 82600000  records written to Mongo\n",
      "2020-04-14 22:31:22.817603 82700000  records written to Mongo\n",
      "2020-04-14 22:31:59.977638 82800000  records written to Mongo\n",
      "2020-04-14 22:32:39.634640 82900000  records written to Mongo\n",
      "2020-04-14 22:33:14.451714 83000000  records written to Mongo\n",
      "2020-04-14 22:33:49.080744 83100000  records written to Mongo\n",
      "2020-04-14 22:34:16.326743 83200000  records written to Mongo\n",
      "2020-04-14 22:34:46.443747 83300000  records written to Mongo\n",
      "2020-04-14 22:35:16.514714 83400000  records written to Mongo\n",
      "2020-04-14 22:35:43.838806 83500000  records written to Mongo\n",
      "2020-04-14 22:36:11.984805 83600000  records written to Mongo\n",
      "2020-04-14 22:36:37.572806 83700000  records written to Mongo\n",
      "2020-04-14 22:36:59.760810 83800000  records written to Mongo\n",
      "2020-04-14 22:37:30.390282 83900000  records written to Mongo\n",
      "2020-04-14 22:37:53.870791 84000000  records written to Mongo\n",
      "2020-04-14 22:38:28.014480 84100000  records written to Mongo\n",
      "2020-04-14 22:38:50.973470 84200000  records written to Mongo\n",
      "2020-04-14 22:39:21.993451 84300000  records written to Mongo\n",
      "2020-04-14 22:39:52.861988 84400000  records written to Mongo\n",
      "2020-04-14 22:40:19.383958 84500000  records written to Mongo\n",
      "2020-04-14 22:40:47.333991 84600000  records written to Mongo\n",
      "2020-04-14 22:41:14.144988 84700000  records written to Mongo\n",
      "2020-04-14 22:41:41.738466 84800000  records written to Mongo\n",
      "2020-04-14 22:42:09.164595 84900000  records written to Mongo\n",
      "2020-04-14 22:42:30.613594 85000000  records written to Mongo\n",
      "2020-04-14 22:43:02.744164 85100000  records written to Mongo\n",
      "2020-04-14 22:43:27.141745 85200000  records written to Mongo\n",
      "2020-04-14 22:43:50.518309 85300000  records written to Mongo\n",
      "2020-04-14 22:44:18.434820 85400000  records written to Mongo\n",
      "2020-04-14 22:44:42.372790 85500000  records written to Mongo\n",
      "2020-04-14 22:45:06.346790 85600000  records written to Mongo\n",
      "2020-04-14 22:45:33.885330 85700000  records written to Mongo\n",
      "2020-04-14 22:45:56.916758 85800000  records written to Mongo\n",
      "2020-04-14 22:46:26.658796 85900000  records written to Mongo\n",
      "2020-04-14 22:46:49.365722 86000000  records written to Mongo\n",
      "2020-04-14 22:47:14.402768 86100000  records written to Mongo\n",
      "2020-04-14 22:47:37.178288 86200000  records written to Mongo\n",
      "2020-04-14 22:47:57.068956 86300000  records written to Mongo\n",
      "2020-04-14 22:48:29.188033 86400000  records written to Mongo\n",
      "2020-04-14 22:48:54.703680 86500000  records written to Mongo\n",
      "2020-04-14 22:49:22.671722 86600000  records written to Mongo\n",
      "2020-04-14 22:50:14.630737 86700000  records written to Mongo\n",
      "2020-04-14 22:50:56.797919 86800000  records written to Mongo\n",
      "2020-04-14 22:51:34.975083 86900000  records written to Mongo\n",
      "2020-04-14 22:52:04.588586 87000000  records written to Mongo\n",
      "2020-04-14 22:52:39.763741 87100000  records written to Mongo\n",
      "2020-04-14 22:53:05.902121 87200000  records written to Mongo\n",
      "2020-04-14 22:53:35.889310 87300000  records written to Mongo\n",
      "2020-04-14 22:54:07.150775 87400000  records written to Mongo\n",
      "2020-04-14 22:54:34.531862 87500000  records written to Mongo\n",
      "2020-04-14 22:55:06.357971 87600000  records written to Mongo\n",
      "2020-04-14 22:55:31.414598 87700000  records written to Mongo\n",
      "2020-04-14 22:55:56.125569 87800000  records written to Mongo\n",
      "2020-04-14 22:56:26.362217 87900000  records written to Mongo\n",
      "2020-04-14 22:56:59.042841 88000000  records written to Mongo\n",
      "2020-04-14 22:57:34.889808 88100000  records written to Mongo\n",
      "2020-04-14 22:58:03.147766 88200000  records written to Mongo\n",
      "2020-04-14 22:58:29.532262 88300000  records written to Mongo\n",
      "2020-04-14 22:58:57.000057 88400000  records written to Mongo\n",
      "2020-04-14 22:59:36.974074 88500000  records written to Mongo\n",
      "2020-04-14 23:00:11.033672 88600000  records written to Mongo\n",
      "2020-04-14 23:00:38.632160 88700000  records written to Mongo\n",
      "2020-04-14 23:01:10.178656 88800000  records written to Mongo\n",
      "2020-04-14 23:01:35.200000 88900000  records written to Mongo\n",
      "2020-04-14 23:02:00.723000 89000000  records written to Mongo\n",
      "2020-04-14 23:02:33.399707 89100000  records written to Mongo\n",
      "2020-04-14 23:03:01.598749 89200000  records written to Mongo\n",
      "2020-04-14 23:03:28.763702 89300000  records written to Mongo\n",
      "2020-04-14 23:03:52.051605 89400000  records written to Mongo\n",
      "2020-04-14 23:04:18.726114 89500000  records written to Mongo\n",
      "2020-04-14 23:04:46.931647 89600000  records written to Mongo\n",
      "2020-04-14 23:05:13.306507 89700000  records written to Mongo\n",
      "2020-04-14 23:05:43.769627 89800000  records written to Mongo\n",
      "2020-04-14 23:06:16.867223 89900000  records written to Mongo\n",
      "2020-04-14 23:06:42.182125 90000000  records written to Mongo\n",
      "2020-04-14 23:07:10.442074 90100000  records written to Mongo\n",
      "2020-04-14 23:07:36.260677 90200000  records written to Mongo\n",
      "2020-04-14 23:08:01.796155 90300000  records written to Mongo\n",
      "2020-04-14 23:08:28.833371 90400000  records written to Mongo\n",
      "2020-04-14 23:08:55.504813 90500000  records written to Mongo\n",
      "2020-04-14 23:09:28.776413 90600000  records written to Mongo\n",
      "2020-04-14 23:09:54.708494 90700000  records written to Mongo\n",
      "2020-04-14 23:10:23.195924 90800000  records written to Mongo\n",
      "2020-04-14 23:10:53.013705 90900000  records written to Mongo\n",
      "2020-04-14 23:11:17.202634 91000000  records written to Mongo\n",
      "2020-04-14 23:11:43.221770 91100000  records written to Mongo\n",
      "2020-04-14 23:12:09.601667 91200000  records written to Mongo\n",
      "2020-04-14 23:12:35.476207 91300000  records written to Mongo\n",
      "2020-04-14 23:13:06.307841 91400000  records written to Mongo\n",
      "2020-04-14 23:13:32.586845 91500000  records written to Mongo\n",
      "2020-04-14 23:14:00.942425 91600000  records written to Mongo\n",
      "2020-04-14 23:14:34.158421 91700000  records written to Mongo\n",
      "2020-04-14 23:15:42.264965 91800000  records written to Mongo\n",
      "2020-04-14 23:16:58.934136 91900000  records written to Mongo\n",
      "2020-04-14 23:18:22.189865 92000000  records written to Mongo\n",
      "2020-04-14 23:18:59.738878 92100000  records written to Mongo\n",
      "2020-04-14 23:19:58.274177 92200000  records written to Mongo\n",
      "2020-04-14 23:20:59.510239 92300000  records written to Mongo\n",
      "2020-04-14 23:21:42.488057 92400000  records written to Mongo\n",
      "2020-04-14 23:22:32.853197 92500000  records written to Mongo\n",
      "2020-04-14 23:23:15.490192 92600000  records written to Mongo\n",
      "2020-04-14 23:24:03.055662 92700000  records written to Mongo\n",
      "2020-04-14 23:24:54.498224 92800000  records written to Mongo\n",
      "2020-04-14 23:25:39.288710 92900000  records written to Mongo\n",
      "2020-04-14 23:26:33.732726 93000000  records written to Mongo\n",
      "2020-04-14 23:27:25.711576 93100000  records written to Mongo\n",
      "2020-04-14 23:28:06.412159 93200000  records written to Mongo\n",
      "2020-04-14 23:28:59.698367 93300000  records written to Mongo\n",
      "2020-04-14 23:29:41.381402 93400000  records written to Mongo\n",
      "2020-04-14 23:30:33.347914 93500000  records written to Mongo\n",
      "2020-04-14 23:31:22.815797 93600000  records written to Mongo\n",
      "2020-04-14 23:32:10.172027 93700000  records written to Mongo\n",
      "2020-04-14 23:33:03.169064 93800000  records written to Mongo\n",
      "2020-04-14 23:33:54.219964 93900000  records written to Mongo\n",
      "2020-04-14 23:34:40.968573 94000000  records written to Mongo\n",
      "2020-04-14 23:35:38.328381 94100000  records written to Mongo\n",
      "2020-04-14 23:36:26.041716 94200000  records written to Mongo\n",
      "2020-04-14 23:37:21.593530 94300000  records written to Mongo\n",
      "2020-04-14 23:38:11.969877 94400000  records written to Mongo\n",
      "2020-04-14 23:38:51.469196 94500000  records written to Mongo\n",
      "2020-04-14 23:39:44.534853 94600000  records written to Mongo\n",
      "2020-04-14 23:41:13.368289 94700000  records written to Mongo\n",
      "2020-04-14 23:42:27.712711 94800000  records written to Mongo\n",
      "2020-04-14 23:43:50.894894 94900000  records written to Mongo\n",
      "2020-04-14 23:45:19.798967 95000000  records written to Mongo\n",
      "2020-04-14 23:47:03.885019 95100000  records written to Mongo\n",
      "2020-04-14 23:48:29.766019 95200000  records written to Mongo\n",
      "2020-04-14 23:49:58.929150 95300000  records written to Mongo\n",
      "2020-04-14 23:51:28.443118 95400000  records written to Mongo\n",
      "2020-04-14 23:53:33.784697 95500000  records written to Mongo\n",
      "2020-04-14 23:55:04.101305 95600000  records written to Mongo\n",
      "2020-04-14 23:56:32.045813 95700000  records written to Mongo\n",
      "2020-04-14 23:57:54.647895 95800000  records written to Mongo\n",
      "2020-04-14 23:59:24.380580 95900000  records written to Mongo\n",
      "2020-04-15 00:01:02.077106 96000000  records written to Mongo\n",
      "2020-04-15 00:02:43.017210 96100000  records written to Mongo\n",
      "2020-04-15 00:04:18.738749 96200000  records written to Mongo\n",
      "2020-04-15 00:06:03.540749 96300000  records written to Mongo\n",
      "2020-04-15 00:07:43.352718 96400000  records written to Mongo\n",
      "2020-04-15 00:09:34.797294 96500000  records written to Mongo\n",
      "2020-04-15 00:11:01.902358 96600000  records written to Mongo\n",
      "2020-04-15 00:12:39.508150 96700000  records written to Mongo\n",
      "2020-04-15 00:14:13.832742 96800000  records written to Mongo\n",
      "2020-04-15 00:15:52.998253 96900000  records written to Mongo\n",
      "2020-04-15 00:17:53.315252 97000000  records written to Mongo\n",
      "2020-04-15 00:19:28.385440 97100000  records written to Mongo\n",
      "2020-04-15 00:20:55.701629 97200000  records written to Mongo\n",
      "2020-04-15 00:22:30.469923 97300000  records written to Mongo\n",
      "2020-04-15 00:24:20.901433 97400000  records written to Mongo\n",
      "2020-04-15 00:25:58.614014 97500000  records written to Mongo\n",
      "2020-04-15 00:27:35.209096 97600000  records written to Mongo\n",
      "2020-04-15 00:29:21.469599 97700000  records written to Mongo\n",
      "2020-04-15 00:31:04.324867 97800000  records written to Mongo\n",
      "2020-04-15 00:31:54.636136 97900000  records written to Mongo\n",
      "2020-04-15 00:33:03.708708 98000000  records written to Mongo\n",
      "2020-04-15 00:34:30.498710 98100000  records written to Mongo\n",
      "2020-04-15 00:35:30.324782 98200000  records written to Mongo\n",
      "2020-04-15 00:35:57.302782 98300000  records written to Mongo\n",
      "2020-04-15 00:36:34.538290 98400000  records written to Mongo\n",
      "2020-04-15 00:37:07.201800 98500000  records written to Mongo\n",
      "2020-04-15 00:37:39.033829 98600000  records written to Mongo\n",
      "2020-04-15 00:38:12.427308 98700000  records written to Mongo\n",
      "2020-04-15 00:38:51.255917 98800000  records written to Mongo\n",
      "2020-04-15 00:39:22.179553 98900000  records written to Mongo\n",
      "2020-04-15 00:40:00.130556 99000000  records written to Mongo\n",
      "2020-04-15 00:40:35.134524 99100000  records written to Mongo\n",
      "2020-04-15 00:41:10.832067 99200000  records written to Mongo\n",
      "2020-04-15 00:41:35.670066 99300000  records written to Mongo\n",
      "2020-04-15 00:42:03.703574 99400000  records written to Mongo\n",
      "2020-04-15 00:42:33.681573 99500000  records written to Mongo\n",
      "2020-04-15 00:43:03.230590 99600000  records written to Mongo\n",
      "2020-04-15 00:43:38.320836 99700000  records written to Mongo\n",
      "2020-04-15 00:44:09.418865 99800000  records written to Mongo\n",
      "2020-04-15 00:44:45.849837 99900000  records written to Mongo\n",
      "2020-04-15 00:45:13.150836 100000000  records written to Mongo\n",
      "2020-04-15 00:45:49.234886 100100000  records written to Mongo\n",
      "2020-04-15 00:46:16.074805 100200000  records written to Mongo\n",
      "2020-04-15 00:46:46.836302 100300000  records written to Mongo\n",
      "2020-04-15 00:47:12.891752 100400000  records written to Mongo\n",
      "2020-04-15 00:47:42.924001 100500000  records written to Mongo\n",
      "2020-04-15 00:48:12.656373 100600000  records written to Mongo\n",
      "2020-04-15 00:48:40.501583 100700000  records written to Mongo\n",
      "2020-04-15 00:49:12.323554 100800000  records written to Mongo\n",
      "2020-04-15 00:49:38.344686 100900000  records written to Mongo\n",
      "2020-04-15 00:50:08.259126 101000000  records written to Mongo\n",
      "2020-04-15 00:50:33.233251 101100000  records written to Mongo\n",
      "2020-04-15 00:51:03.140668 101200000  records written to Mongo\n",
      "2020-04-15 00:51:30.584493 101300000  records written to Mongo\n",
      "2020-04-15 00:52:06.301545 101400000  records written to Mongo\n",
      "2020-04-15 00:52:36.966633 101500000  records written to Mongo\n",
      "2020-04-15 00:53:11.425039 101600000  records written to Mongo\n",
      "2020-04-15 00:53:42.798549 101700000  records written to Mongo\n",
      "2020-04-15 00:54:16.262529 101800000  records written to Mongo\n",
      "2020-04-15 00:54:40.969622 101900000  records written to Mongo\n",
      "2020-04-15 00:55:08.000198 102000000  records written to Mongo\n",
      "2020-04-15 00:55:35.910853 102100000  records written to Mongo\n",
      "2020-04-15 00:56:02.387871 102200000  records written to Mongo\n",
      "2020-04-15 00:56:32.924046 102300000  records written to Mongo\n",
      "2020-04-15 00:57:00.434016 102400000  records written to Mongo\n",
      "2020-04-15 00:57:33.595696 102500000  records written to Mongo\n",
      "2020-04-15 00:57:59.846972 102600000  records written to Mongo\n",
      "2020-04-15 00:58:24.550652 102700000  records written to Mongo\n",
      "2020-04-15 00:58:54.990476 102800000  records written to Mongo\n",
      "2020-04-15 00:59:17.501954 102900000  records written to Mongo\n",
      "2020-04-15 00:59:43.431657 103000000  records written to Mongo\n",
      "2020-04-15 01:00:11.490973 103100000  records written to Mongo\n",
      "2020-04-15 01:00:38.372973 103200000  records written to Mongo\n",
      "2020-04-15 01:01:18.569178 103300000  records written to Mongo\n",
      "2020-04-15 01:01:52.911444 103400000  records written to Mongo\n",
      "2020-04-15 01:02:32.886107 103500000  records written to Mongo\n",
      "2020-04-15 01:03:04.698176 103600000  records written to Mongo\n",
      "2020-04-15 01:03:43.616621 103700000  records written to Mongo\n",
      "2020-04-15 01:04:16.681731 103800000  records written to Mongo\n",
      "2020-04-15 01:04:56.319555 103900000  records written to Mongo\n",
      "2020-04-15 01:05:31.316248 104000000  records written to Mongo\n",
      "2020-04-15 01:06:08.352240 104100000  records written to Mongo\n",
      "2020-04-15 01:06:49.440847 104200000  records written to Mongo\n",
      "2020-04-15 01:07:27.037483 104300000  records written to Mongo\n",
      "2020-04-15 01:08:07.283508 104400000  records written to Mongo\n",
      "2020-04-15 01:08:40.116508 104500000  records written to Mongo\n",
      "2020-04-15 01:09:17.151825 104600000  records written to Mongo\n",
      "2020-04-15 01:09:48.117069 104700000  records written to Mongo\n",
      "2020-04-15 01:10:26.423470 104800000  records written to Mongo\n",
      "2020-04-15 01:10:58.768435 104900000  records written to Mongo\n",
      "2020-04-15 01:11:36.755634 105000000  records written to Mongo\n",
      "2020-04-15 01:12:08.325116 105100000  records written to Mongo\n",
      "2020-04-15 01:12:47.152336 105200000  records written to Mongo\n",
      "2020-04-15 01:13:26.700621 105300000  records written to Mongo\n",
      "2020-04-15 01:13:57.500594 105400000  records written to Mongo\n",
      "2020-04-15 01:14:37.992593 105500000  records written to Mongo\n",
      "2020-04-15 01:15:16.200333 105600000  records written to Mongo\n",
      "2020-04-15 01:15:59.760514 105700000  records written to Mongo\n",
      "2020-04-15 01:16:34.535549 105800000  records written to Mongo\n",
      "2020-04-15 01:17:17.947369 105900000  records written to Mongo\n",
      "2020-04-15 01:17:53.967677 106000000  records written to Mongo\n",
      "2020-04-15 01:18:40.313133 106100000  records written to Mongo\n",
      "2020-04-15 01:19:22.717231 106200000  records written to Mongo\n",
      "2020-04-15 01:19:56.258833 106300000  records written to Mongo\n",
      "2020-04-15 01:20:34.741997 106400000  records written to Mongo\n",
      "2020-04-15 01:21:06.855125 106500000  records written to Mongo\n",
      "2020-04-15 01:21:47.857561 106600000  records written to Mongo\n",
      "2020-04-15 01:22:23.557589 106700000  records written to Mongo\n",
      "2020-04-15 01:23:05.853764 106800000  records written to Mongo\n",
      "2020-04-15 01:23:40.287257 106900000  records written to Mongo\n",
      "2020-04-15 01:24:19.188560 107000000  records written to Mongo\n",
      "2020-04-15 01:24:59.986636 107100000  records written to Mongo\n",
      "2020-04-15 01:25:34.937210 107200000  records written to Mongo\n",
      "2020-04-15 01:26:10.250469 107300000  records written to Mongo\n",
      "2020-04-15 01:26:48.294398 107400000  records written to Mongo\n",
      "2020-04-15 01:27:28.283538 107500000  records written to Mongo\n",
      "2020-04-15 01:28:00.067712 107600000  records written to Mongo\n",
      "2020-04-15 01:28:41.179685 107700000  records written to Mongo\n",
      "2020-04-15 01:29:16.082167 107800000  records written to Mongo\n",
      "2020-04-15 01:29:53.690700 107900000  records written to Mongo\n",
      "2020-04-15 01:30:27.156209 108000000  records written to Mongo\n",
      "2020-04-15 01:31:05.636270 108100000  records written to Mongo\n",
      "2020-04-15 01:31:38.490655 108200000  records written to Mongo\n",
      "2020-04-15 01:32:19.078472 108300000  records written to Mongo\n",
      "2020-04-15 01:33:02.434197 108400000  records written to Mongo\n",
      "2020-04-15 01:33:39.005271 108500000  records written to Mongo\n",
      "2020-04-15 01:34:20.922266 108600000  records written to Mongo\n",
      "2020-04-15 01:35:00.336504 108700000  records written to Mongo\n",
      "2020-04-15 01:35:45.761961 108800000  records written to Mongo\n",
      "2020-04-15 01:36:21.339218 108900000  records written to Mongo\n",
      "2020-04-15 01:36:59.943696 109000000  records written to Mongo\n",
      "2020-04-15 01:37:39.015206 109100000  records written to Mongo\n",
      "2020-04-15 01:38:10.922509 109200000  records written to Mongo\n",
      "2020-04-15 01:38:47.765541 109300000  records written to Mongo\n",
      "2020-04-15 01:39:21.814495 109400000  records written to Mongo\n",
      "2020-04-15 01:39:59.435906 109500000  records written to Mongo\n",
      "2020-04-15 01:40:32.453032 109600000  records written to Mongo\n",
      "2020-04-15 01:41:13.265554 109700000  records written to Mongo\n",
      "2020-04-15 01:41:46.413099 109800000  records written to Mongo\n",
      "2020-04-15 01:42:24.226007 109900000  records written to Mongo\n",
      "2020-04-15 01:43:04.618711 110000000  records written to Mongo\n",
      "2020-04-15 01:43:45.294783 110100000  records written to Mongo\n",
      "2020-04-15 01:44:18.678295 110200000  records written to Mongo\n",
      "2020-04-15 01:44:55.829295 110300000  records written to Mongo\n",
      "2020-04-15 01:45:35.977296 110400000  records written to Mongo\n",
      "2020-04-15 01:46:08.093325 110500000  records written to Mongo\n",
      "2020-04-15 01:46:45.591836 110600000  records written to Mongo\n",
      "2020-04-15 01:47:18.104313 110700000  records written to Mongo\n",
      "2020-04-15 01:47:55.944394 110800000  records written to Mongo\n",
      "2020-04-15 01:48:27.829901 110900000  records written to Mongo\n",
      "2020-04-15 01:49:05.700900 111000000  records written to Mongo\n",
      "2020-04-15 01:49:37.840901 111100000  records written to Mongo\n",
      "2020-04-15 01:50:19.197921 111200000  records written to Mongo\n",
      "2020-04-15 01:50:50.467953 111300000  records written to Mongo\n",
      "2020-04-15 01:51:34.377167 111400000  records written to Mongo\n",
      "2020-04-15 01:52:06.169677 111500000  records written to Mongo\n",
      "2020-04-15 01:52:42.831707 111600000  records written to Mongo\n",
      "2020-04-15 01:53:11.235678 111700000  records written to Mongo\n",
      "2020-04-15 01:53:43.720677 111800000  records written to Mongo\n",
      "2020-04-15 01:54:11.191707 111900000  records written to Mongo\n",
      "2020-04-15 01:54:43.446707 112000000  records written to Mongo\n",
      "2020-04-15 01:55:15.152678 112100000  records written to Mongo\n",
      "2020-04-15 01:55:50.026677 112200000  records written to Mongo\n",
      "2020-04-15 01:56:16.976679 112300000  records written to Mongo\n",
      "2020-04-15 01:56:47.045715 112400000  records written to Mongo\n",
      "2020-04-15 01:57:12.986767 112500000  records written to Mongo\n",
      "Wall time: 11h 26min 51s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "batch_size = 1000\n",
    "batch = []\n",
    "i=0\n",
    "for doi, works_record in cr_iterator:\n",
    "    i+=1\n",
    "    works_record['_uploaded'] = datetime.strftime(datetime.now(), \"%Y-%m-%d\")\n",
    "    batch.append(pymongo.UpdateOne({'DOI':doi}, {'$setOnInsert':works_record}, upsert = True))\n",
    "    if i%batch_size==0:\n",
    "        result = collection.bulk_write(batch)\n",
    "        batch = []\n",
    "        if i%100000==0:\n",
    "            print(datetime.now(),i, ' records written to Mongo')\n",
    "result = collection.bulk_write(batch)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Add indexes\n",
    "Now we need to add indexes. see: https://docs.mongodb.com/manual/indexes/\n",
    "\n",
    "#### Index on\n",
    "- DOI\n",
    "- publication date (note that CrossRef stores several dates for each entry - you might need multiple indexes)\n",
    "- Publisher\n",
    "- ISSN / eISSN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "db.works.create_index([( 'DOI', pymongo.ASCENDING )])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'issn-type.value_1'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "db.works.create_index([( 'issn-type.value', pymongo.ASCENDING )])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'_id': ObjectId('5e9600dd96e4f11cab42b892'),\n",
       " 'DOI': '10.1177/2158244017712774',\n",
       " 'ISSN': ['2158-2440', '2158-2440'],\n",
       " 'URL': 'http://dx.doi.org/10.1177/2158244017712774',\n",
       " '_uploaded': '2020-04-14',\n",
       " 'alternative-id': ['10.1177/2158244017712774'],\n",
       " 'author': [{'given': 'Logan S.',\n",
       "   'family': 'Casey',\n",
       "   'affiliation': [{'name': 'Harvard T.H. Chan School of Public Health, Boston, MA, USA'}]},\n",
       "  {'given': 'Jesse',\n",
       "   'family': 'Chandler',\n",
       "   'affiliation': [{'name': 'University of Michigan, Ann Arbor, USA'},\n",
       "    {'name': 'Mathematica Policy Research, Ann Arbor, MI, USA'}]},\n",
       "  {'given': 'Adam Seth',\n",
       "   'family': 'Levine',\n",
       "   'affiliation': [{'name': 'Cornell University, Ithaca, NY, USA'}]},\n",
       "  {'given': 'Andrew',\n",
       "   'family': 'Proctor',\n",
       "   'affiliation': [{'name': 'Princeton University, NJ, USA'}]},\n",
       "  {'given': 'Dara Z.',\n",
       "   'family': 'Strolovitch',\n",
       "   'affiliation': [{'name': 'Princeton University, NJ, USA'}]}],\n",
       " 'container-title': ['SAGE Open'],\n",
       " 'content-domain': {'domain': ['journals.sagepub.com'],\n",
       "  'crossmark-restriction': True},\n",
       " 'created': {'date-parts': [[2017, 6, 14]],\n",
       "  'date-time': '2017-06-14T12:57:07Z',\n",
       "  'timestamp': 1497445027000},\n",
       " 'deposited': {'date-parts': [[2017, 12, 12]],\n",
       "  'date-time': '2017-12-12T11:01:36Z',\n",
       "  'timestamp': 1513076496000},\n",
       " 'indexed': {'date-parts': [[2020, 3, 2]],\n",
       "  'date-time': '2020-03-02T11:03:26Z',\n",
       "  'timestamp': 1583147006643},\n",
       " 'is-referenced-by-count': 17,\n",
       " 'issn-type': [{'value': '2158-2440', 'type': 'print'},\n",
       "  {'value': '2158-2440', 'type': 'electronic'}],\n",
       " 'issue': '2',\n",
       " 'issued': {'date-parts': [[2017, 4]]},\n",
       " 'license': [{'URL': 'http://journals.sagepub.com/page/policies/text-and-data-mining-license',\n",
       "   'start': {'date-parts': [[2017, 4, 1]],\n",
       "    'date-time': '2017-04-01T00:00:00Z',\n",
       "    'timestamp': 1491004800000},\n",
       "   'delay-in-days': 0,\n",
       "   'content-version': 'tdm'}],\n",
       " 'link': [{'URL': 'http://journals.sagepub.com/doi/pdf/10.1177/2158244017712774',\n",
       "   'content-type': 'application/pdf',\n",
       "   'content-version': 'vor',\n",
       "   'intended-application': 'text-mining'},\n",
       "  {'URL': 'http://journals.sagepub.com/doi/full-xml/10.1177/2158244017712774',\n",
       "   'content-type': 'application/xml',\n",
       "   'content-version': 'vor',\n",
       "   'intended-application': 'text-mining'},\n",
       "  {'URL': 'http://journals.sagepub.com/doi/pdf/10.1177/2158244017712774',\n",
       "   'content-type': 'unspecified',\n",
       "   'content-version': 'vor',\n",
       "   'intended-application': 'similarity-checking'}],\n",
       " 'member': '179',\n",
       " 'original-title': [],\n",
       " 'page': '215824401771277',\n",
       " 'prefix': '10.1177',\n",
       " 'published-online': {'date-parts': [[2017, 6, 14]]},\n",
       " 'published-print': {'date-parts': [[2017, 4]]},\n",
       " 'publisher': 'SAGE Publications',\n",
       " 'reference': [{'key': 'bibr1-2158244017712774',\n",
       "   'DOI': '10.1016/j.hrmr.2012.06.003',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr4-2158244017712774',\n",
       "   'DOI': '10.1016/S0191-8869(01)00199-4',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr5-2158244017712774',\n",
       "   'DOI': '10.2307/2346101',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr6-2158244017712774',\n",
       "   'DOI': '10.1093/pan/mpr057',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr7-2158244017712774',\n",
       "   'DOI': '10.1016/j.chb.2015.08.031',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr9-2158244017712774',\n",
       "   'DOI': '10.1016/j.chb.2013.05.009',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr10-2158244017712774',\n",
       "   'DOI': '10.3758/s13428-013-0365-7',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr11-2158244017712774',\n",
       "   'DOI': '10.1177/0956797615585115',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr12-2158244017712774',\n",
       "   'DOI': '10.1146/annurev-clinpsy-021815-093623',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr13-2158244017712774',\n",
       "   'DOI': '10.1111/j.1467-6494.1991.tb00770.x',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr14-2158244017712774',\n",
       "   'DOI': '10.1016/j.psychres.2014.12.047',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr15-2158244017712774',\n",
       "   'DOI': '10.1086/318641',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr16-2158244017712774',\n",
       "   'DOI': '10.1093/poq/nfu046',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr17-2158244017712774',\n",
       "   'DOI': '10.1177/0894439314525918',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr18-2158244017712774',\n",
       "   'DOI': '10.1111/j.1467-9221.2012.00870.x',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr20-2158244017712774',\n",
       "   'DOI': '10.1086/268245',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr22-2158244017712774',\n",
       "   'DOI': '10.1371/journal.pone.0109019',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr23-2158244017712774',\n",
       "   'DOI': '10.1037/h0031907',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr26-2158244017712774',\n",
       "   'DOI': '10.1146/annurev-polisci-051010-111659',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr29-2158244017712774',\n",
       "   'DOI': '10.1016/S0092-6566(03)00046-1',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr31-2158244017712774',\n",
       "   'DOI': '10.1080/0092623X.2016.1178675',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr33-2158244017712774',\n",
       "   'DOI': '10.1093/pan/mpl013',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr34-2158244017712774',\n",
       "   'DOI': '10.1093/heapol/czq043',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr35-2158244017712774',\n",
       "   'author': 'John O. P.',\n",
       "   'volume': '2',\n",
       "   'first-page': '102',\n",
       "   'year': '1999',\n",
       "   'volume-title': 'Handbook of personality: Theory and research'},\n",
       "  {'key': 'bibr36-2158244017712774',\n",
       "   'DOI': '10.1145/2470654.2470684',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr37-2158244017712774',\n",
       "   'DOI': '10.1017/xps.2014.7',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr40-2158244017712774',\n",
       "   'DOI': '10.1177/0894439313485201',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr42-2158244017712774',\n",
       "   'DOI': '10.1017/XPS.2015.19',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr43-2158244017712774',\n",
       "   'DOI': '10.1177/0963721414531598',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr45-2158244017712774',\n",
       "   'DOI': '10.1016/j.jesp.2017.01.006',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr46-2158244017712774',\n",
       "   'DOI': '10.3758/s13428-013-0434-y',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr47-2158244017712774',\n",
       "   'DOI': '10.1016/j.jbusres.2013.08.010',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr51-2158244017712774',\n",
       "   'DOI': '10.1016/j.paid.2014.04.021',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr52-2158244017712774',\n",
       "   'DOI': '10.1177/2167702612469015',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr53-2158244017712774',\n",
       "   'DOI': '10.2478/jos-2014-0042',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr54-2158244017712774',\n",
       "   'DOI': '10.1177/0894439316633452',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr55-2158244017712774',\n",
       "   'author': 'Stewart N.',\n",
       "   'volume': '10',\n",
       "   'first-page': '479',\n",
       "   'year': '2015',\n",
       "   'journal-title': 'Judgment and Decision Making'},\n",
       "  {'key': 'bibr56-2158244017712774',\n",
       "   'DOI': '10.12758/mda.2015.014',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr60-2158244017712774',\n",
       "   'DOI': '10.1093/aje/kwf185',\n",
       "   'doi-asserted-by': 'publisher'},\n",
       "  {'key': 'bibr61-2158244017712774',\n",
       "   'DOI': '10.15195/v1.a19',\n",
       "   'doi-asserted-by': 'publisher'}],\n",
       " 'reference-count': 40,\n",
       " 'references-count': 40,\n",
       " 'relation': {'cites': []},\n",
       " 'score': None,\n",
       " 'short-container-title': ['SAGE Open'],\n",
       " 'short-title': [],\n",
       " 'source': 'Crossref',\n",
       " 'subtitle': [],\n",
       " 'title': ['Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection'],\n",
       " 'type': 'journal-article',\n",
       " 'update-policy': 'http://dx.doi.org/10.1177/sage-journals-update-policy',\n",
       " 'volume': '7'}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look up article by DOI\n",
    "collection.find_one({'DOI':'10.1177/2158244017712774'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "393"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look up articles by issn and year.  Easy.\n",
    "issn = '2158-2440'\n",
    "year = 2019\n",
    "# create a cursor - essentially an object which represents our 'lookup' of the data\n",
    "cur = collection.find({'issn-type.value':issn,\n",
    "                    'issued.date-parts.0.0':year})\n",
    "# read the data from that 'lookup' into a list object\n",
    "works = [x for x in cur]\n",
    "# how many did we get?\n",
    "len(works)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'_id': ObjectId('5e9600dd96e4f11cab42b9b4'),\n",
       "  'DOI': '10.1177/2158244018789229',\n",
       "  'ISSN': ['2158-2440', '2158-2440'],\n",
       "  'URL': 'http://dx.doi.org/10.1177/2158244018789229',\n",
       "  '_uploaded': '2020-04-14',\n",
       "  'alternative-id': ['10.1177/2158244018789229'],\n",
       "  'author': [{'given': 'Andrea',\n",
       "    'family': 'Ceron',\n",
       "    'affiliation': [{'name': 'Università degli Studi di Milano, Italy'}]},\n",
       "   {'given': 'Luigi',\n",
       "    'family': 'Curini',\n",
       "    'affiliation': [{'name': 'Università degli Studi di Milano, Italy'}]},\n",
       "   {'ORCID': 'http://orcid.org/0000-0002-4884-0047',\n",
       "    'authenticated-orcid': False,\n",
       "    'given': 'Stefano M.',\n",
       "    'family': 'Iacus',\n",
       "    'affiliation': [{'name': 'Università degli Studi di Milano, Italy'}]}],\n",
       "  'container-title': ['SAGE Open'],\n",
       "  'content-domain': {'domain': ['journals.sagepub.com'],\n",
       "   'crossmark-restriction': True},\n",
       "  'created': {'date-parts': [[2019, 3, 27]],\n",
       "   'date-time': '2019-03-27T17:26:18Z',\n",
       "   'timestamp': 1553707578000},\n",
       "  'deposited': {'date-parts': [[2019, 3, 27]],\n",
       "   'date-time': '2019-03-27T17:26:34Z',\n",
       "   'timestamp': 1553707594000},\n",
       "  'indexed': {'date-parts': [[2020, 3, 25]],\n",
       "   'date-time': '2020-03-25T02:58:50Z',\n",
       "   'timestamp': 1585105130751},\n",
       "  'is-referenced-by-count': 1,\n",
       "  'issn-type': [{'value': '2158-2440', 'type': 'print'},\n",
       "   {'value': '2158-2440', 'type': 'electronic'}],\n",
       "  'issue': '1',\n",
       "  'issued': {'date-parts': [[2019, 1]]},\n",
       "  'license': [{'URL': 'http://journals.sagepub.com/page/policies/text-and-data-mining-license',\n",
       "    'start': {'date-parts': [[2019, 1, 1]],\n",
       "     'date-time': '2019-01-01T00:00:00Z',\n",
       "     'timestamp': 1546300800000},\n",
       "    'delay-in-days': 0,\n",
       "    'content-version': 'tdm'}],\n",
       "  'link': [{'URL': 'http://journals.sagepub.com/doi/pdf/10.1177/2158244018789229',\n",
       "    'content-type': 'application/pdf',\n",
       "    'content-version': 'vor',\n",
       "    'intended-application': 'text-mining'},\n",
       "   {'URL': 'http://journals.sagepub.com/doi/full-xml/10.1177/2158244018789229',\n",
       "    'content-type': 'application/xml',\n",
       "    'content-version': 'vor',\n",
       "    'intended-application': 'text-mining'},\n",
       "   {'URL': 'http://journals.sagepub.com/doi/pdf/10.1177/2158244018789229',\n",
       "    'content-type': 'unspecified',\n",
       "    'content-version': 'vor',\n",
       "    'intended-application': 'similarity-checking'}],\n",
       "  'member': '179',\n",
       "  'original-title': [],\n",
       "  'page': '215824401878922',\n",
       "  'prefix': '10.1177',\n",
       "  'published-online': {'date-parts': [[2019, 3, 27]]},\n",
       "  'published-print': {'date-parts': [[2019, 1]]},\n",
       "  'publisher': 'SAGE Publications',\n",
       "  'reference': [{'key': 'bibr1-2158244018789229',\n",
       "    'author': 'Akhgar B.',\n",
       "    'year': '2016',\n",
       "    'volume-title': 'Open source intelligence investigation',\n",
       "    'DOI': '10.1007/978-3-319-47671-1',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr2-2158244018789229',\n",
       "    'author': 'Al-Moslmi T.',\n",
       "    'volume': '44',\n",
       "    'first-page': '363',\n",
       "    'year': '2018',\n",
       "    'journal-title': 'Journal of Information Science',\n",
       "    'DOI': '10.1177/0165551516683908',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr3-2158244018789229',\n",
       "    'DOI': '10.4135/9781473994829',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr4-2158244018789229',\n",
       "    'DOI': '10.1007/s42001-018-0015-z',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr6-2158244018789229',\n",
       "    'author': 'Benson D.',\n",
       "    'volume': '23',\n",
       "    'first-page': '293',\n",
       "    'year': '2014',\n",
       "    'journal-title': 'Security Studies',\n",
       "    'DOI': '10.1080/09636412.2014.905353',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr8-2158244018789229',\n",
       "    'author': 'Berinsky A. J.',\n",
       "    'volume': '43',\n",
       "    'first-page': '1209',\n",
       "    'year': '1999',\n",
       "    'journal-title': 'American Journal of Political Science',\n",
       "    'DOI': '10.2307/2991824',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr9-2158244018789229',\n",
       "    'author': 'Bishop G. F.',\n",
       "    'volume': '14',\n",
       "    'first-page': '5',\n",
       "    'year': '2003',\n",
       "    'journal-title': 'Public Perspective'},\n",
       "   {'key': 'bibr10-2158244018789229',\n",
       "    'author': 'Brands H.',\n",
       "    'volume': '96',\n",
       "    'first-page': '28',\n",
       "    'year': '2017',\n",
       "    'journal-title': 'Foreign Affairs'},\n",
       "   {'key': 'bibr11-2158244018789229',\n",
       "    'author': 'Briggs R.',\n",
       "    'volume': '17',\n",
       "    'first-page': '272',\n",
       "    'year': '2010',\n",
       "    'journal-title': 'Democratization',\n",
       "    'DOI': '10.1080/13510341003588690',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr12-2158244018789229',\n",
       "    'DOI': '10.1177/0022002705284829',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr13-2158244018789229',\n",
       "    'author': 'Burnap P.',\n",
       "    'volume': '7',\n",
       "    'first-page': '223',\n",
       "    'year': '2015',\n",
       "    'journal-title': 'Policy & Internet',\n",
       "    'DOI': '10.1002/poi3.85',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr14-2158244018789229',\n",
       "    'author': 'Byman D.',\n",
       "    'volume': '95',\n",
       "    'first-page': '76',\n",
       "    'year': '2016',\n",
       "    'journal-title': 'Foreign Affairs'},\n",
       "   {'key': 'bibr16-2158244018789229',\n",
       "    'author': 'Ceron A.',\n",
       "    'first-page': '367',\n",
       "    'year': '2016',\n",
       "    'journal-title': 'Information Sciences'},\n",
       "   {'key': 'bibr17-2158244018789229',\n",
       "    'author': 'Cohen J.',\n",
       "    'volume': '94',\n",
       "    'first-page': '52',\n",
       "    'year': '2015',\n",
       "    'journal-title': 'Foreign Affairs'},\n",
       "   {'key': 'bibr19-2158244018789229',\n",
       "    'author': 'Cronin A. K.',\n",
       "    'year': '2009',\n",
       "    'volume-title': 'How terrorism ends: Understanding the decline and demise of terrorist campaigns',\n",
       "    'DOI': '10.2307/j.ctt7rjbs',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr20-2158244018789229',\n",
       "    'author': 'Cunliffe E.',\n",
       "    'year': '2018',\n",
       "    'journal-title': 'Antiquity: A Review of World Archaeology'},\n",
       "   {'key': 'bibr21-2158244018789229',\n",
       "    'author': 'Dalacoura K.',\n",
       "    'volume': '13',\n",
       "    'first-page': '508',\n",
       "    'year': '2006',\n",
       "    'journal-title': 'Democratization',\n",
       "    'DOI': '10.1080/13510340600579516',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr22-2158244018789229',\n",
       "    'DOI': '10.1371/journal.pone.0079449',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr23-2158244018789229',\n",
       "    'author': 'el-Nawawy M.',\n",
       "    'year': '2009',\n",
       "    'volume-title': 'Islam dot com: Contemporary Islamic discourses in cyberspace',\n",
       "    'DOI': '10.1057/9780230622661',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr24-2158244018789229',\n",
       "    'author': 'Farwell J. P.',\n",
       "    'volume': '56',\n",
       "    'first-page': '49',\n",
       "    'year': '2014',\n",
       "    'journal-title': 'Survival',\n",
       "    'DOI': '10.1080/00396338.2014.985436',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr25-2158244018789229',\n",
       "    'author': 'Ferrara E.',\n",
       "    'volume': '418',\n",
       "    'first-page': '1',\n",
       "    'year': '2017',\n",
       "    'journal-title': 'Information Sciences',\n",
       "    'DOI': '10.1016/j.ins.2017.07.030',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr26-2158244018789229',\n",
       "    'author': 'Ferrara E.',\n",
       "    'volume': '10047',\n",
       "    'first-page': '22',\n",
       "    'year': '2016',\n",
       "    'volume-title': 'Lecture Notes in Computer Science: International conference on social informatics'},\n",
       "   {'key': 'bibr27-2158244018789229',\n",
       "    'author': 'Fisher R. J.',\n",
       "    'volume': '20',\n",
       "    'first-page': '303',\n",
       "    'year': '1993',\n",
       "    'journal-title': 'Journal of Consumer Research',\n",
       "    'DOI': '10.1086/209351',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr28-2158244018789229',\n",
       "    'author': 'Frey B. S.',\n",
       "    'volume': '14',\n",
       "    'first-page': '237',\n",
       "    'year': '2003',\n",
       "    'journal-title': 'Defense and Peace Economics',\n",
       "    'DOI': '10.1080/1024269032000052923',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr30-2158244018789229',\n",
       "    'author': 'González-Bailón S.',\n",
       "    'volume': '38',\n",
       "    'first-page': '16',\n",
       "    'year': '2014',\n",
       "    'journal-title': 'Social Networks',\n",
       "    'DOI': '10.1016/j.socnet.2014.01.004',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr31-2158244018789229',\n",
       "    'DOI': '10.1177/0002716216672635',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr32-2158244018789229',\n",
       "    'author': 'Grimmer J.',\n",
       "    'volume': '21',\n",
       "    'first-page': '267',\n",
       "    'year': '2013',\n",
       "    'journal-title': 'Political Analysis',\n",
       "    'DOI': '10.1093/pan/mps028',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr33-2158244018789229',\n",
       "    'author': 'Gurr T. R.',\n",
       "    'first-page': '85',\n",
       "    'year': '2006',\n",
       "    'volume-title': 'The roots of terrorism'},\n",
       "   {'key': 'bibr34-2158244018789229',\n",
       "    'author': 'Hashemi N.',\n",
       "    'volume': '15',\n",
       "    'first-page': '1',\n",
       "    'year': '2017',\n",
       "    'journal-title': 'The Review of Faith & International Affairs',\n",
       "    'DOI': '10.1080/15570274.2017.1354462',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr36-2158244018789229',\n",
       "    'author': 'Hirschman A. O.',\n",
       "    'year': '1970',\n",
       "    'volume-title': 'Exit, voice, and loyalty: Responses to decline in firms, organizations, and states'},\n",
       "   {'key': 'bibr37-2158244018789229',\n",
       "    'author': 'Hopkins D.',\n",
       "    'volume': '54',\n",
       "    'first-page': '229',\n",
       "    'year': '2010',\n",
       "    'journal-title': 'American Journal of Political Science',\n",
       "    'DOI': '10.1111/j.1540-5907.2009.00428.x',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr38-2158244018789229',\n",
       "    'author': 'Howard P. N.',\n",
       "    'volume': '22',\n",
       "    'first-page': '35',\n",
       "    'year': '2011',\n",
       "    'journal-title': 'Journal of Democracy',\n",
       "    'DOI': '10.1353/jod.2011.0041',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr39-2158244018789229',\n",
       "    'author': 'Jamal A. A.',\n",
       "    'volume': '13',\n",
       "    'first-page': '55',\n",
       "    'year': '2015',\n",
       "    'journal-title': 'Perspectives on Politics',\n",
       "    'DOI': '10.1017/S1537592714003132',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr40-2158244018789229',\n",
       "    'author': 'Jenkins B.',\n",
       "    'year': '1981',\n",
       "    'volume-title': 'The psychological implications of media-covered terrorism'},\n",
       "   {'key': 'bibr41-2158244018789229',\n",
       "    'author': 'Jurgens D.',\n",
       "    'volume': '13',\n",
       "    'first-page': '273',\n",
       "    'year': '2013',\n",
       "    'volume-title': 'International AAAI Conference on Web and Social Media'},\n",
       "   {'key': 'bibr42-2158244018789229',\n",
       "    'author': 'King G.',\n",
       "    'volume': '107',\n",
       "    'first-page': '326',\n",
       "    'year': '2013',\n",
       "    'journal-title': 'American Political Science Review',\n",
       "    'DOI': '10.1017/S0003055413000014',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr43-2158244018789229',\n",
       "    'author': 'King G.',\n",
       "    'volume': '345',\n",
       "    'first-page': '1',\n",
       "    'year': '2014',\n",
       "    'journal-title': 'Science',\n",
       "    'DOI': '10.1126/science.1251722',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr44-2158244018789229',\n",
       "    'author': 'Klausen J.',\n",
       "    'volume': '38',\n",
       "    'first-page': '1',\n",
       "    'year': '2015',\n",
       "    'journal-title': 'Studies in Conflict & Terrorism',\n",
       "    'DOI': '10.1080/1057610X.2014.974948',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr45-2158244018789229',\n",
       "    'author': 'Knopf J. W.',\n",
       "    'volume': '31',\n",
       "    'first-page': '1',\n",
       "    'year': '2010',\n",
       "    'journal-title': 'Contemporary Security Policy',\n",
       "    'DOI': '10.1080/13523261003640819',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr46-2158244018789229',\n",
       "    'DOI': '10.1177/0163443717725575',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr47-2158244018789229',\n",
       "    'DOI': '10.1177/0022002787031002003',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr50-2158244018789229',\n",
       "    'author': 'Monroe B. L.',\n",
       "    'volume': '48',\n",
       "    'first-page': '71',\n",
       "    'year': '2015',\n",
       "    'journal-title': 'PS: Political Science & Politics'},\n",
       "   {'key': 'bibr51-2158244018789229',\n",
       "    'author': 'Nagler J.',\n",
       "    'volume': '48',\n",
       "    'first-page': '84',\n",
       "    'year': '2015',\n",
       "    'journal-title': 'PS: Political Science & Politics'},\n",
       "   {'key': 'bibr52-2158244018789229',\n",
       "    'author': 'Neumann P.',\n",
       "    'volume': '36',\n",
       "    'first-page': '431',\n",
       "    'year': '2013',\n",
       "    'journal-title': 'Studies in Conflict and Terrorism',\n",
       "    'DOI': '10.1080/1057610X.2013.784568',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr54-2158244018789229',\n",
       "    'author': 'Novenario C. M. I.',\n",
       "    'volume': '39',\n",
       "    'first-page': '953',\n",
       "    'year': '2016',\n",
       "    'journal-title': 'Studies in Conflict & Terrorism',\n",
       "    'DOI': '10.1080/1057610X.2016.1151679',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr55-2158244018789229',\n",
       "    'author': 'O’Hara K.',\n",
       "    'volume': '7',\n",
       "    'first-page': '401',\n",
       "    'year': '2015',\n",
       "    'journal-title': 'Policy & Internet',\n",
       "    'DOI': '10.1002/poi3.88',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr56-2158244018789229',\n",
       "    'DOI': '10.1075/jlp.17005.sie',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr57-2158244018789229',\n",
       "    'author': 'Stephens-Davidowitz S.',\n",
       "    'volume': '118',\n",
       "    'first-page': '26',\n",
       "    'year': '2014',\n",
       "    'journal-title': 'Journal of Public Economics',\n",
       "    'DOI': '10.1016/j.jpubeco.2014.04.010',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr58-2158244018789229',\n",
       "    'author': 'Torres Soriano M',\n",
       "    'volume': '35',\n",
       "    'first-page': '263',\n",
       "    'year': '2012',\n",
       "    'journal-title': 'Studies in Conflict & Terrorism',\n",
       "    'DOI': '10.1080/1057610X.2012.656345',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr59-2158244018789229',\n",
       "    'author': 'Tucker J. A.',\n",
       "    'volume': '28',\n",
       "    'first-page': '46',\n",
       "    'year': '2017',\n",
       "    'journal-title': 'Journal of Democracy',\n",
       "    'DOI': '10.1353/jod.2017.0064',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr60-2158244018789229',\n",
       "    'author': 'Weimann G.',\n",
       "    'year': '1994',\n",
       "    'volume-title': 'The theatre of terror: Mass media and international terrorism'},\n",
       "   {'key': 'bibr61-2158244018789229',\n",
       "    'author': 'Whine M.',\n",
       "    'volume': '22',\n",
       "    'first-page': '231',\n",
       "    'year': '1999',\n",
       "    'journal-title': 'Studies in Conflict & Terrorism',\n",
       "    'DOI': '10.1080/105761099265748',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr62-2158244018789229',\n",
       "    'DOI': '10.1177/1461444809342775',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr63-2158244018789229',\n",
       "    'DOI': '10.1177/0022002711408014',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr64-2158244018789229',\n",
       "    'DOI': '10.1177/0022343314558700',\n",
       "    'doi-asserted-by': 'publisher'}],\n",
       "  'reference-count': 55,\n",
       "  'references-count': 55,\n",
       "  'relation': {'cites': []},\n",
       "  'score': None,\n",
       "  'short-container-title': ['SAGE Open'],\n",
       "  'short-title': [],\n",
       "  'source': 'Crossref',\n",
       "  'subject': ['General Arts and Humanities', 'General Social Sciences'],\n",
       "  'subtitle': [],\n",
       "  'title': ['ISIS at Its Apogee: The Arabic Discourse on Twitter and What We Can Learn From That About ISIS Support and Foreign Fighters'],\n",
       "  'type': 'journal-article',\n",
       "  'update-policy': 'http://dx.doi.org/10.1177/sage-journals-update-policy',\n",
       "  'volume': '9'},\n",
       " {'_id': ObjectId('5e9600dd96e4f11cab42b9bb'),\n",
       "  'DOI': '10.1177/2158244018791653',\n",
       "  'ISSN': ['2158-2440', '2158-2440'],\n",
       "  'URL': 'http://dx.doi.org/10.1177/2158244018791653',\n",
       "  '_uploaded': '2020-04-14',\n",
       "  'alternative-id': ['10.1177/2158244018791653'],\n",
       "  'author': [{'given': 'Anjie',\n",
       "    'family': 'Fang',\n",
       "    'affiliation': [{'name': 'University of Glasgow, Glasgow, UK'}]},\n",
       "   {'given': 'Philip',\n",
       "    'family': 'Habel',\n",
       "    'affiliation': [{'name': 'University of South Alabama, Mobile, USA'}]},\n",
       "   {'given': 'Iadh',\n",
       "    'family': 'Ounis',\n",
       "    'affiliation': [{'name': 'University of Glasgow, Glasgow, UK'}]},\n",
       "   {'given': 'Craig',\n",
       "    'family': 'MacDonald',\n",
       "    'affiliation': [{'name': 'University of Glasgow, Glasgow, UK'}]}],\n",
       "  'container-title': ['SAGE Open'],\n",
       "  'content-domain': {'domain': ['journals.sagepub.com'],\n",
       "   'crossmark-restriction': True},\n",
       "  'created': {'date-parts': [[2019, 3, 27]],\n",
       "   'date-time': '2019-03-27T21:27:09Z',\n",
       "   'timestamp': 1553722029000},\n",
       "  'deposited': {'date-parts': [[2019, 3, 27]],\n",
       "   'date-time': '2019-03-27T21:27:16Z',\n",
       "   'timestamp': 1553722036000},\n",
       "  'indexed': {'date-parts': [[2020, 3, 25]],\n",
       "   'date-time': '2020-03-25T02:58:50Z',\n",
       "   'timestamp': 1585105130874},\n",
       "  'is-referenced-by-count': 1,\n",
       "  'issn-type': [{'value': '2158-2440', 'type': 'print'},\n",
       "   {'value': '2158-2440', 'type': 'electronic'}],\n",
       "  'issue': '1',\n",
       "  'issued': {'date-parts': [[2019, 1]]},\n",
       "  'license': [{'URL': 'http://journals.sagepub.com/page/policies/text-and-data-mining-license',\n",
       "    'start': {'date-parts': [[2019, 1, 1]],\n",
       "     'date-time': '2019-01-01T00:00:00Z',\n",
       "     'timestamp': 1546300800000},\n",
       "    'delay-in-days': 0,\n",
       "    'content-version': 'tdm'}],\n",
       "  'link': [{'URL': 'http://journals.sagepub.com/doi/pdf/10.1177/2158244018791653',\n",
       "    'content-type': 'application/pdf',\n",
       "    'content-version': 'vor',\n",
       "    'intended-application': 'text-mining'},\n",
       "   {'URL': 'http://journals.sagepub.com/doi/full-xml/10.1177/2158244018791653',\n",
       "    'content-type': 'application/xml',\n",
       "    'content-version': 'vor',\n",
       "    'intended-application': 'text-mining'},\n",
       "   {'URL': 'http://journals.sagepub.com/doi/pdf/10.1177/2158244018791653',\n",
       "    'content-type': 'unspecified',\n",
       "    'content-version': 'vor',\n",
       "    'intended-application': 'similarity-checking'}],\n",
       "  'member': '179',\n",
       "  'original-title': [],\n",
       "  'page': '215824401879165',\n",
       "  'prefix': '10.1177',\n",
       "  'published-online': {'date-parts': [[2019, 3, 27]]},\n",
       "  'published-print': {'date-parts': [[2019, 1]]},\n",
       "  'publisher': 'SAGE Publications',\n",
       "  'reference': [{'key': 'bibr1-2158244018791653',\n",
       "    'author': 'Adamic L. A.',\n",
       "    'first-page': '36',\n",
       "    'year': '2005',\n",
       "    'volume-title': 'Proceedings of the 3rd international workshop on Link discovery',\n",
       "    'DOI': '10.1145/1134271.1134277',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr2-2158244018791653',\n",
       "    'author': 'Allcott H.',\n",
       "    'volume': '31',\n",
       "    'first-page': '211',\n",
       "    'year': '2017',\n",
       "    'journal-title': 'Journal of Economic Perspectives',\n",
       "    'DOI': '10.1257/jep.31.2.211',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr3-2158244018791653',\n",
       "    'DOI': '10.1177/0894439314558836',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr4-2158244018791653',\n",
       "    'author': 'Beauchamp N.',\n",
       "    'volume': '61',\n",
       "    'first-page': '490',\n",
       "    'year': '2016',\n",
       "    'journal-title': 'American Journal of Political Science',\n",
       "    'DOI': '10.1111/ajps.12274',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr5-2158244018791653',\n",
       "    'author': 'Bond R.',\n",
       "    'volume': '109',\n",
       "    'first-page': '62',\n",
       "    'year': '2015',\n",
       "    'journal-title': 'American Political Science Review',\n",
       "    'DOI': '10.1017/S0003055414000525',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr6-2158244018791653',\n",
       "    'author': 'Burnap P.',\n",
       "    'volume': '41',\n",
       "    'first-page': '230',\n",
       "    'year': '2016',\n",
       "    'journal-title': 'Electoral Studies',\n",
       "    'DOI': '10.1016/j.electstud.2015.11.017',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr7-2158244018791653',\n",
       "    'author': 'Dietterich T. G.',\n",
       "    'volume': '10',\n",
       "    'first-page': '1895',\n",
       "    'year': '1998',\n",
       "    'journal-title': 'Neural Computation',\n",
       "    'DOI': '10.1162/089976698300017197',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr8-2158244018791653',\n",
       "    'author': 'Fang A.',\n",
       "    'first-page': '825',\n",
       "    'year': '2016',\n",
       "    'volume-title': 'Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval'},\n",
       "   {'key': 'bibr9-2158244018791653',\n",
       "    'author': 'Fang A.',\n",
       "    'first-page': '1057',\n",
       "    'year': '2016',\n",
       "    'volume-title': 'Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval'},\n",
       "   {'key': 'bibr10-2158244018791653',\n",
       "    'author': 'Fang A.',\n",
       "    'first-page': '252',\n",
       "    'year': '2017',\n",
       "    'volume-title': 'Proceedings of the 39th European Conference on Information Retrieval'},\n",
       "   {'key': 'bibr11-2158244018791653',\n",
       "    'author': 'Fang A.',\n",
       "    'first-page': '791',\n",
       "    'year': '2015',\n",
       "    'volume-title': 'Proceedings of the 38th International ACM SIGIR conference on Research and Development in Information Retrieval'},\n",
       "   {'key': 'bibr12-2158244018791653',\n",
       "    'author': 'Grimmer J.',\n",
       "    'volume': '21',\n",
       "    'first-page': '267',\n",
       "    'year': '2013',\n",
       "    'journal-title': 'Political Analysis',\n",
       "    'DOI': '10.1093/pan/mps028',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr14-2158244018791653',\n",
       "    'author': 'Harrag F.',\n",
       "    'volume': '20',\n",
       "    'first-page': '441',\n",
       "    'year': '2010',\n",
       "    'journal-title': 'Neural Network World'},\n",
       "   {'key': 'bibr15-2158244018791653',\n",
       "    'author': 'Howard P. N.',\n",
       "    'volume': '15',\n",
       "    'first-page': '81',\n",
       "    'year': '2018',\n",
       "    'journal-title': 'Journal of Information Technology & Politics',\n",
       "    'DOI': '10.1080/19331681.2018.1448735',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr16-2158244018791653',\n",
       "    'author': 'Joachims T.',\n",
       "    'first-page': '137',\n",
       "    'year': '1998',\n",
       "    'volume-title': 'Proceedings of the 10th European Conference on Machine Learning'},\n",
       "   {'key': 'bibr17-2158244018791653',\n",
       "    'author': 'Jungherr A.',\n",
       "    'volume': '13',\n",
       "    'first-page': '72',\n",
       "    'year': '2016',\n",
       "    'journal-title': 'Journal of Information Technology & Politics',\n",
       "    'DOI': '10.1080/19331681.2015.1132401',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr18-2158244018791653',\n",
       "    'author': 'Khorsheed M. S.',\n",
       "    'volume': '47',\n",
       "    'first-page': '513',\n",
       "    'year': '2013',\n",
       "    'journal-title': 'Language Resources and Evaluation',\n",
       "    'DOI': '10.1007/s10579-013-9221-8',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr19-2158244018791653',\n",
       "    'author': 'Kim S.-B.',\n",
       "    'volume': '18',\n",
       "    'first-page': '1457',\n",
       "    'year': '2006',\n",
       "    'journal-title': 'IEEE Transactions on Knowledge and Data Engineering',\n",
       "    'DOI': '10.1109/TKDE.2006.180',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr20-2158244018791653',\n",
       "    'author': 'Klašnja M.',\n",
       "    'year': '2017',\n",
       "    'volume-title': 'The Oxford handbook of polling and survey methods'},\n",
       "   {'key': 'bibr21-2158244018791653',\n",
       "    'author': 'McCallum A.',\n",
       "    'first-page': '41',\n",
       "    'year': '1998',\n",
       "    'volume-title': 'Proceedings of the 15th AAAI conference workshop on learnning for text categorization'},\n",
       "   {'key': 'bibr22-2158244018791653',\n",
       "    'author': 'McKelvey K.',\n",
       "    'volume': '17',\n",
       "    'first-page': '436',\n",
       "    'year': '2014',\n",
       "    'journal-title': 'Journal of Information Technology & Politics'},\n",
       "   {'key': 'bibr23-2158244018791653',\n",
       "    'author': 'Morstatter F.',\n",
       "    'year': '2013',\n",
       "    'volume-title': 'Proceedings of the Seventh International AAAI Conference on Weblogs and Social Media'},\n",
       "   {'key': 'bibr24-2158244018791653',\n",
       "    'author': 'Porter M. A.',\n",
       "    'volume': '102',\n",
       "    'first-page': '7057',\n",
       "    'year': '2005',\n",
       "    'journal-title': 'Proceedings of the National Academy of Sciences of the United States of America',\n",
       "    'DOI': '10.1073/pnas.0500191102',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr25-2158244018791653',\n",
       "    'author': 'Porter M. F.',\n",
       "    'year': '1997',\n",
       "    'volume-title': 'Readings in information retrieval'},\n",
       "   {'key': 'bibr26-2158244018791653',\n",
       "    'author': 'Rijsbergen C. J. V.',\n",
       "    'year': '1979',\n",
       "    'edition': '2',\n",
       "    'volume-title': 'Information retrieval'},\n",
       "   {'key': 'bibr27-2158244018791653',\n",
       "    'author': 'Soroush V.',\n",
       "    'volume': '359',\n",
       "    'first-page': '1146',\n",
       "    'year': '2018',\n",
       "    'journal-title': 'Science',\n",
       "    'DOI': '10.1126/science.aap9559',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr29-2158244018791653',\n",
       "    'author': 'Tucker J. A.',\n",
       "    'volume': '28',\n",
       "    'first-page': '46',\n",
       "    'year': '2017',\n",
       "    'journal-title': 'Journal of Democracy',\n",
       "    'DOI': '10.1353/jod.2017.0064',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr30-2158244018791653',\n",
       "    'author': 'Zhang H.',\n",
       "    'first-page': '708',\n",
       "    'year': '2007',\n",
       "    'volume-title': 'IEEE International Conference on Granular computing'}],\n",
       "  'reference-count': 28,\n",
       "  'references-count': 28,\n",
       "  'relation': {'cites': []},\n",
       "  'score': None,\n",
       "  'short-container-title': ['SAGE Open'],\n",
       "  'short-title': [],\n",
       "  'source': 'Crossref',\n",
       "  'subject': ['General Arts and Humanities', 'General Social Sciences'],\n",
       "  'subtitle': [],\n",
       "  'title': ['Votes on Twitter: Assessing Candidate Preferences and Topics of Discussion During the 2016 U.S. Presidential Election'],\n",
       "  'type': 'journal-article',\n",
       "  'update-policy': 'http://dx.doi.org/10.1177/sage-journals-update-policy',\n",
       "  'volume': '9'},\n",
       " {'_id': ObjectId('5e9600dd96e4f11cab42ba15'),\n",
       "  'DOI': '10.1177/2158244018817141',\n",
       "  'ISSN': ['2158-2440', '2158-2440'],\n",
       "  'URL': 'http://dx.doi.org/10.1177/2158244018817141',\n",
       "  '_uploaded': '2020-04-14',\n",
       "  'alternative-id': ['10.1177/2158244018817141'],\n",
       "  'author': [{'ORCID': 'http://orcid.org/0000-0002-0974-9144',\n",
       "    'authenticated-orcid': False,\n",
       "    'given': 'Karel',\n",
       "    'family': 'Kouba',\n",
       "    'affiliation': [{'name': 'Department of Politics, Philosophical Faculty, University of Hradec Králové, Czech Republic'}]},\n",
       "   {'given': 'Stanislav',\n",
       "    'family': 'Mysicka',\n",
       "    'affiliation': [{'name': 'Department of Politics, Philosophical Faculty, University of Hradec Králové, Czech Republic'}]}],\n",
       "  'container-title': ['SAGE Open'],\n",
       "  'content-domain': {'domain': ['journals.sagepub.com'],\n",
       "   'crossmark-restriction': True},\n",
       "  'created': {'date-parts': [[2019, 3, 5]],\n",
       "   'date-time': '2019-03-05T10:33:22Z',\n",
       "   'timestamp': 1551782002000},\n",
       "  'deposited': {'date-parts': [[2019, 3, 5]],\n",
       "   'date-time': '2019-03-05T10:33:38Z',\n",
       "   'timestamp': 1551782018000},\n",
       "  'funder': [{'name': 'University of Hradec Kralove', 'award': []}],\n",
       "  'indexed': {'date-parts': [[2020, 3, 24]],\n",
       "   'date-time': '2020-03-24T23:31:54Z',\n",
       "   'timestamp': 1585092714630},\n",
       "  'is-referenced-by-count': 0,\n",
       "  'issn-type': [{'value': '2158-2440', 'type': 'print'},\n",
       "   {'value': '2158-2440', 'type': 'electronic'}],\n",
       "  'issue': '1',\n",
       "  'issued': {'date-parts': [[2019, 1]]},\n",
       "  'license': [{'URL': 'http://journals.sagepub.com/page/policies/text-and-data-mining-license',\n",
       "    'start': {'date-parts': [[2019, 1, 1]],\n",
       "     'date-time': '2019-01-01T00:00:00Z',\n",
       "     'timestamp': 1546300800000},\n",
       "    'delay-in-days': 0,\n",
       "    'content-version': 'tdm'}],\n",
       "  'link': [{'URL': 'http://journals.sagepub.com/doi/pdf/10.1177/2158244018817141',\n",
       "    'content-type': 'application/pdf',\n",
       "    'content-version': 'vor',\n",
       "    'intended-application': 'text-mining'},\n",
       "   {'URL': 'http://journals.sagepub.com/doi/full-xml/10.1177/2158244018817141',\n",
       "    'content-type': 'application/xml',\n",
       "    'content-version': 'vor',\n",
       "    'intended-application': 'text-mining'},\n",
       "   {'URL': 'http://journals.sagepub.com/doi/pdf/10.1177/2158244018817141',\n",
       "    'content-type': 'unspecified',\n",
       "    'content-version': 'vor',\n",
       "    'intended-application': 'similarity-checking'}],\n",
       "  'member': '179',\n",
       "  'original-title': [],\n",
       "  'page': '215824401881714',\n",
       "  'prefix': '10.1177',\n",
       "  'published-online': {'date-parts': [[2019, 3, 5]]},\n",
       "  'published-print': {'date-parts': [[2019, 1]]},\n",
       "  'publisher': 'SAGE Publications',\n",
       "  'reference': [{'key': 'bibr1-2158244018817141',\n",
       "    'author': 'Birch S.',\n",
       "    'volume': '16',\n",
       "    'first-page': '21',\n",
       "    'year': '2009',\n",
       "    'journal-title': 'Public Policy Research',\n",
       "    'DOI': '10.1111/j.1744-540X.2009.00550.x',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr2-2158244018817141',\n",
       "    'author': 'Birch S.',\n",
       "    'year': '2009',\n",
       "    'volume-title': 'Full participation: A comparative study of compulsory voting',\n",
       "    'DOI': '10.7228/manchester/9780719077623.001.0001',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr3-2158244018817141',\n",
       "    'author': 'Brennan J.',\n",
       "    'year': '2014',\n",
       "    'volume-title': 'Compulsory voting: For and against',\n",
       "    'DOI': '10.1017/CBO9781139649742',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr4-2158244018817141',\n",
       "    'author': 'Calero C.',\n",
       "    'year': '2008',\n",
       "    'volume-title': 'Mapa de pobreza y desigualdad en Ecuador'},\n",
       "   {'key': 'bibr5-2158244018817141',\n",
       "    'author': 'Carey J. M.',\n",
       "    'volume': '59',\n",
       "    'first-page': '122',\n",
       "    'year': '2017',\n",
       "    'journal-title': 'Latin American Politics and Society',\n",
       "    'DOI': '10.1111/laps.12021',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr6-2158244018817141',\n",
       "    'author': 'Carreras M.',\n",
       "    'volume': '43',\n",
       "    'first-page': '158',\n",
       "    'year': '2016',\n",
       "    'journal-title': 'Electoral Studies',\n",
       "    'DOI': '10.1016/j.electstud.2016.04.005',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr7-2158244018817141',\n",
       "    'author': 'Chong A.',\n",
       "    'volume': '20',\n",
       "    'first-page': '391',\n",
       "    'year': '2008',\n",
       "    'journal-title': 'Economics & Politics',\n",
       "    'DOI': '10.1111/j.1468-0343.2008.00336.x',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr8-2158244018817141',\n",
       "    'DOI': '10.1007/s11109-017-9405-9',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr9-2158244018817141',\n",
       "    'DOI': '10.1177/0010414016655537',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr10-2158244018817141',\n",
       "    'author': 'Dassonneville R.',\n",
       "    'volume': '40',\n",
       "    'first-page': '621',\n",
       "    'year': '2017',\n",
       "    'journal-title': 'West European Politics',\n",
       "    'DOI': '10.1080/01402382.2016.1266187',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr11-2158244018817141',\n",
       "    'DOI': '10.1177/1065912914524634',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr12-2158244018817141',\n",
       "    'author': 'Dworkin G.',\n",
       "    'volume': '56',\n",
       "    'first-page': '64',\n",
       "    'year': '1972',\n",
       "    'journal-title': 'The Monist',\n",
       "    'DOI': '10.5840/monist197256119',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr13-2158244018817141',\n",
       "    'DOI': '10.1111/j.1467-9256.2009.01358.x',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr14-2158244018817141',\n",
       "    'DOI': '10.1177/0010414004267981',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr15-2158244018817141',\n",
       "    'author': 'Freidenberg F.',\n",
       "    'year': '2011',\n",
       "    'volume-title': 'El Código de la Democracia: una evaluación del proceso de reforma electoral del Ecuador (2008-2009)'},\n",
       "   {'key': 'bibr16-2158244018817141',\n",
       "    'author': 'Gallego A.',\n",
       "    'volume': '29',\n",
       "    'first-page': '239',\n",
       "    'year': '2010',\n",
       "    'journal-title': 'Electoral Studies',\n",
       "    'DOI': '10.1016/j.electstud.2009.11.002',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr17-2158244018817141',\n",
       "    'author': 'Henn M.',\n",
       "    'volume': '19',\n",
       "    'first-page': '1259',\n",
       "    'year': '2016',\n",
       "    'journal-title': 'Journal of Youth Studies',\n",
       "    'DOI': '10.1080/13676261.2016.1154935',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr18-2158244018817141',\n",
       "    'author': 'Hill L.',\n",
       "    'volume': '42',\n",
       "    'first-page': '515',\n",
       "    'year': '2007',\n",
       "    'journal-title': 'Australian Journal of Political Science',\n",
       "    'DOI': '10.1080/10361140701513646',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr19-2158244018817141',\n",
       "    'author': 'Hurka T.',\n",
       "    'year': '1993',\n",
       "    'volume-title': 'Perfectionism'},\n",
       "   {'key': 'bibr20-2158244018817141',\n",
       "    'author': 'Jaitman L.',\n",
       "    'volume': '92',\n",
       "    'first-page': '79',\n",
       "    'year': '2013',\n",
       "    'journal-title': 'Journal of Economic Behavior & Organization',\n",
       "    'DOI': '10.1016/j.jebo.2013.05.008',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr21-2158244018817141',\n",
       "    'DOI': '10.1017/psrm.2016.49',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr22-2158244018817141',\n",
       "    'author': 'Kouba K.',\n",
       "    'volume': '41',\n",
       "    'first-page': '92',\n",
       "    'year': '2016',\n",
       "    'journal-title': 'Electoral Studies',\n",
       "    'DOI': '10.1016/j.electstud.2015.11.021',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr23-2158244018817141',\n",
       "    'DOI': '10.1111/j.1467-9256.2007.00299.x',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr24-2158244018817141',\n",
       "    'author': 'Lever A.',\n",
       "    'volume': '1',\n",
       "    'first-page': '57',\n",
       "    'year': '2009',\n",
       "    'journal-title': 'Public Reason'},\n",
       "   {'key': 'bibr25-2158244018817141',\n",
       "    'author': 'Lever A.',\n",
       "    'volume': '40',\n",
       "    'first-page': '897',\n",
       "    'year': '2010',\n",
       "    'journal-title': 'British Journal of Political Science',\n",
       "    'DOI': '10.1017/S0007123410000050',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr26-2158244018817141',\n",
       "    'author': 'Lijphart A.',\n",
       "    'volume': '91',\n",
       "    'first-page': '1',\n",
       "    'year': '1997',\n",
       "    'journal-title': 'American Political Science Review',\n",
       "    'DOI': '10.2307/2952255',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr27-2158244018817141',\n",
       "    'author': 'Lundell K.',\n",
       "    'volume': '48',\n",
       "    'first-page': '221',\n",
       "    'year': '2012',\n",
       "    'journal-title': 'Representation',\n",
       "    'DOI': '10.1080/00344893.2012.683488',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr28-2158244018817141',\n",
       "    'author': 'McAllister I.',\n",
       "    'volume': '12',\n",
       "    'first-page': '23',\n",
       "    'year': '1993',\n",
       "    'journal-title': 'Electoral Studies',\n",
       "    'DOI': '10.1016/0261-3794(93)90004-4',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr29-2158244018817141',\n",
       "    'author': 'O’Toole F.',\n",
       "    'volume': '7',\n",
       "    'first-page': '271',\n",
       "    'year': '1995',\n",
       "    'journal-title': 'Economics & Politics',\n",
       "    'DOI': '10.1111/j.1468-0343.1995.tb00115.x',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr30-2158244018817141',\n",
       "    'author': 'Payne J. M.',\n",
       "    'year': '2006',\n",
       "    'volume-title': 'La política importa: democracia y desarrollo en América Latina'},\n",
       "   {'key': 'bibr31-2158244018817141',\n",
       "    'author': 'Pogge T.',\n",
       "    'year': '2007',\n",
       "    'volume-title': 'John Rawls: His life and theory of justice',\n",
       "    'DOI': '10.1093/acprof:oso/9780195136364.001.0001',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr32-2158244018817141',\n",
       "    'author': 'Power T. J.',\n",
       "    'volume': '26',\n",
       "    'first-page': '432',\n",
       "    'year': '2007',\n",
       "    'journal-title': 'Electoral Studies',\n",
       "    'DOI': '10.1016/j.electstud.2006.11.001',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr33-2158244018817141',\n",
       "    'author': 'Rawls J.',\n",
       "    'year': '1971',\n",
       "    'volume-title': 'A theory of justice'},\n",
       "   {'key': 'bibr34-2158244018817141',\n",
       "    'author': 'Rawls J.',\n",
       "    'year': '1996',\n",
       "    'volume-title': 'Political liberalism'},\n",
       "   {'key': 'bibr35-2158244018817141',\n",
       "    'author': 'Rawls J.',\n",
       "    'year': '2001',\n",
       "    'volume-title': 'Justice as fairness: A restatement'},\n",
       "   {'key': 'bibr36-2158244018817141',\n",
       "    'author': 'Reynolds A.',\n",
       "    'volume': '25',\n",
       "    'first-page': '570',\n",
       "    'year': '2006',\n",
       "    'journal-title': 'Electoral Studies',\n",
       "    'DOI': '10.1016/j.electstud.2005.06.009',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr37-2158244018817141',\n",
       "    'author': 'Selb P.',\n",
       "    'volume': '48',\n",
       "    'first-page': '573',\n",
       "    'year': '2009',\n",
       "    'journal-title': 'European Journal of Political Research',\n",
       "    'DOI': '10.1111/j.1475-6765.2009.01834.x',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr38-2158244018817141',\n",
       "    'DOI': '10.1111/1467-9248.12117',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr39-2158244018817141',\n",
       "    'author': 'Singh S. P.',\n",
       "    'volume': '44',\n",
       "    'first-page': '56',\n",
       "    'year': '2016',\n",
       "    'journal-title': 'Electoral Studies',\n",
       "    'DOI': '10.1016/j.electstud.2016.08.005',\n",
       "    'doi-asserted-by': 'crossref'},\n",
       "   {'key': 'bibr40-2158244018817141',\n",
       "    'DOI': '10.1017/psrm.2017.11',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr41-2158244018817141',\n",
       "    'DOI': '10.1017/S0007123416000041',\n",
       "    'doi-asserted-by': 'publisher'},\n",
       "   {'key': 'bibr42-2158244018817141',\n",
       "    'DOI': '10.1177/0010414007301702',\n",
       "    'doi-asserted-by': 'publisher'}],\n",
       "  'reference-count': 42,\n",
       "  'references-count': 42,\n",
       "  'relation': {'cites': []},\n",
       "  'score': None,\n",
       "  'short-container-title': ['SAGE Open'],\n",
       "  'short-title': [],\n",
       "  'source': 'Crossref',\n",
       "  'subject': ['General Arts and Humanities', 'General Social Sciences'],\n",
       "  'subtitle': [],\n",
       "  'title': ['Should and Does Compulsory Voting Reduce Inequality?'],\n",
       "  'type': 'journal-article',\n",
       "  'update-policy': 'http://dx.doi.org/10.1177/sage-journals-update-policy',\n",
       "  'volume': '9'}]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# inspect the data manually\n",
    "works[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}