tweeps_scraping_tweepy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tweeps scraping\n",
    "\n",
    "## Introduction\n",
    "Scrapping twitter handles of popular users from twitter using tweepy. Geotags and hashtags are used to get Kenyan tweets from which user Ids are extracted. The popular users (determined by followers and friends) are added to the dataframe and the less popular ones dropped.\n",
    "\n",
    "The requests are split into batches in order to have an easy time to pinpoint errors when they arise. \n",
    "\n",
    "## Table of Contents\n",
    "1. [Libraries](#Libraries)\n",
    "2. [Kenya's co-ordinates](#geo)\n",
    "3. [Geo Scraper](#Geo-Scraper)\n",
    "4. [Keyword Scraper](#Keyword-Scraper)\n",
    "5. [Mega DataFrame](#Mega-Df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from requests import get\n",
    "from requests.exceptions import RequestException\n",
    "from contextlib import closing\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import os, sys\n",
    "\n",
    "import fire\n",
    "\n",
    "import sys\n",
    "import os\n",
    "import json\n",
    "import matplotlib.pyplot as plt\n",
    "import re\n",
    "import string\n",
    "\n",
    "import matplotlib.dates as mdates\n",
    "import seaborn as sns\n",
    "\n",
    "# to view all columns\n",
    "pd.set_option(\"display.max.columns\", None)\n",
    "\n",
    "import tweepy\n",
    "from tweepy.streaming import StreamListener\n",
    "from tweepy import OAuthHandler\n",
    "from tweepy import Stream\n",
    "from tweepy import API\n",
    "from tweepy import Cursor\n",
    "from datetime import datetime, date, time, timedelta\n",
    "from collections import Counter\n",
    "import sys\n",
    "\n",
    "\n",
    "import preprocessor as p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### geo\n",
    " The first step is to get Kenya's geo cordinates from twitter then use it with the search method to fetch Kenya tweets from Jan 1st this year. From the tweets the users are extracted. The limit is set to 2k users/tweets, from which a few 800 will be selected based on popularity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "consumer_key = os.environ.get('consumer_key')\n",
    "consumer_secret = os.environ.get('consumer_secret')\n",
    "access_token = os.environ.get('access_token')\n",
    "access_token_secret = os.environ.get('access_token_secret')\n",
    "\n",
    "auth = OAuthHandler(consumer_key, consumer_secret)\n",
    "auth.set_access_token(access_token, access_token_secret)\n",
    "auth_api = API(auth, wait_on_rate_limit=True)\n",
    "\n",
    "places = auth_api.geo_search(query=\"KENYA\", granularity=\"country\")\n",
    "place_id = places[0].id\n",
    "places"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Geo Scraper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>tyra_gracy</td>\n",
       "      <td>333</td>\n",
       "      <td>317</td>\n",
       "      <td>FAITH</td>\n",
       "      <td>183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>etadv</td>\n",
       "      <td>4415</td>\n",
       "      <td>4285</td>\n",
       "      <td>Advocate, Chairman LSK Nairobi Branch</td>\n",
       "      <td>9745</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>LaRecetteNwar</td>\n",
       "      <td>391</td>\n",
       "      <td>489</td>\n",
       "      <td>french producer • credits : ISK • Bino • Jeune...</td>\n",
       "      <td>5628</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>ParagraphKenya</td>\n",
       "      <td>184</td>\n",
       "      <td>151</td>\n",
       "      <td>Making News Interesting</td>\n",
       "      <td>39594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>BenjaminsOtenge</td>\n",
       "      <td>1127</td>\n",
       "      <td>5002</td>\n",
       "      <td>Male|Politik|Academician|\\n\\nA teacher by prof...</td>\n",
       "      <td>12415</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       screen_name  followers  friends  \\\n",
       "0       tyra_gracy        333      317   \n",
       "1            etadv       4415     4285   \n",
       "2    LaRecetteNwar        391      489   \n",
       "3   ParagraphKenya        184      151   \n",
       "4  BenjaminsOtenge       1127     5002   \n",
       "\n",
       "                                         description  tweets  \n",
       "0                                              FAITH     183  \n",
       "1              Advocate, Chairman LSK Nairobi Branch    9745  \n",
       "2  french producer • credits : ISK • Bino • Jeune...    5628  \n",
       "3                            Making News Interesting   39594  \n",
       "4  Male|Politik|Academician|\\n\\nA teacher by prof...   12415  "
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,\n",
    "                       geocode=\"0.0236,37.9062,530km\", since_id = '2020-05-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "    \n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "users_df = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "users_df.to_csv('starting users.csv')\n",
    "users_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>EmmaOltOlato</td>\n",
       "      <td>4169</td>\n",
       "      <td>3991</td>\n",
       "      <td>Fashion Model | King of the Iteso  | Civil Eng...</td>\n",
       "      <td>4259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>AmbuchiJR</td>\n",
       "      <td>45</td>\n",
       "      <td>71</td>\n",
       "      <td>Writer, Photographer and Screenplay Director. ...</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>mtieno_</td>\n",
       "      <td>233</td>\n",
       "      <td>425</td>\n",
       "      <td>Tech Enthusiast | Born Sinner | A child of God...</td>\n",
       "      <td>903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>JonathanKaribu</td>\n",
       "      <td>12253</td>\n",
       "      <td>6364</td>\n",
       "      <td>God_fearing, Optimist,Always  Faithful and humble</td>\n",
       "      <td>61646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2142bebe</td>\n",
       "      <td>9</td>\n",
       "      <td>61</td>\n",
       "      <td></td>\n",
       "      <td>3793</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      screen_name  followers  friends  \\\n",
       "0    EmmaOltOlato       4169     3991   \n",
       "1       AmbuchiJR         45       71   \n",
       "2         mtieno_        233      425   \n",
       "3  JonathanKaribu      12253     6364   \n",
       "4        2142bebe          9       61   \n",
       "\n",
       "                                         description  tweets  \n",
       "0  Fashion Model | King of the Iteso  | Civil Eng...    4259  \n",
       "1  Writer, Photographer and Screenplay Director. ...      37  \n",
       "2  Tech Enthusiast | Born Sinner | A child of God...     903  \n",
       "3  God_fearing, Optimist,Always  Faithful and humble   61646  \n",
       "4                                                       3793  "
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,\n",
    "                       geocode=\"0.0236,37.9062,530km\", since_id = '2020-06-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_2 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_2.to_csv('df_2 users.csv')\n",
    "df_2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>purity_ng</td>\n",
       "      <td>10225</td>\n",
       "      <td>137</td>\n",
       "      <td>Loves good Music,lover of life/Daughter/Sister...</td>\n",
       "      <td>46303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>ngubesi</td>\n",
       "      <td>825</td>\n",
       "      <td>5000</td>\n",
       "      <td></td>\n",
       "      <td>4167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Jabuto_Jabuto</td>\n",
       "      <td>2117</td>\n",
       "      <td>191</td>\n",
       "      <td>Agribusiness Strategist | Agripreneur | Projec...</td>\n",
       "      <td>59984</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>chief_sultan</td>\n",
       "      <td>6966</td>\n",
       "      <td>5168</td>\n",
       "      <td>🅖🅞🅓'🅢 🅟🅛🅐🅝.</td>\n",
       "      <td>42889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>jdrizzy254</td>\n",
       "      <td>27031</td>\n",
       "      <td>13192</td>\n",
       "      <td>God is above all \\nArsenal fan 🔴🔴🔴\\neconomist\\...</td>\n",
       "      <td>43855</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     screen_name  followers  friends  \\\n",
       "0      purity_ng      10225      137   \n",
       "1        ngubesi        825     5000   \n",
       "2  Jabuto_Jabuto       2117      191   \n",
       "3   chief_sultan       6966     5168   \n",
       "4     jdrizzy254      27031    13192   \n",
       "\n",
       "                                         description  tweets  \n",
       "0  Loves good Music,lover of life/Daughter/Sister...   46303  \n",
       "1                                                       4167  \n",
       "2  Agribusiness Strategist | Agripreneur | Projec...   59984  \n",
       "3                                        🅖🅞🅓'🅢 🅟🅛🅐🅝.   42889  \n",
       "4  God is above all \\nArsenal fan 🔴🔴🔴\\neconomist\\...   43855  "
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,\n",
    "                       geocode=\"0.0236,37.9062,530km\", since_id = '2020-07-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_3 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_3.to_csv('df_3 users.csv')\n",
    "df_3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>juscloedenise</td>\n",
       "      <td>7789</td>\n",
       "      <td>7524</td>\n",
       "      <td>Not today</td>\n",
       "      <td>15526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>ErickssonOdhia2</td>\n",
       "      <td>427</td>\n",
       "      <td>770</td>\n",
       "      <td>I'm a happy soul full of intrigue and colors\\n...</td>\n",
       "      <td>9591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>kidaha_alfred</td>\n",
       "      <td>411</td>\n",
       "      <td>316</td>\n",
       "      <td>A Communication Lecturer at Multimedia Univers...</td>\n",
       "      <td>159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>anselme_kwizera</td>\n",
       "      <td>7</td>\n",
       "      <td>125</td>\n",
       "      <td>Am proud to be Rwandan</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>DanKweno</td>\n",
       "      <td>9</td>\n",
       "      <td>70</td>\n",
       "      <td></td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       screen_name  followers  friends  \\\n",
       "0    juscloedenise       7789     7524   \n",
       "1  ErickssonOdhia2        427      770   \n",
       "2    kidaha_alfred        411      316   \n",
       "3  anselme_kwizera          7      125   \n",
       "4         DanKweno          9       70   \n",
       "\n",
       "                                         description  tweets  \n",
       "0                                          Not today   15526  \n",
       "1  I'm a happy soul full of intrigue and colors\\n...    9591  \n",
       "2  A Communication Lecturer at Multimedia Univers...     159  \n",
       "3                             Am proud to be Rwandan      21  \n",
       "4                                                         79  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,\n",
    "                       geocode=\"0.0236,37.9062,530km\", since_id = '2020-04-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_01 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_01.to_csv('df_01 users.csv')\n",
    "df_01.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>mr_Okabi</td>\n",
       "      <td>656</td>\n",
       "      <td>493</td>\n",
       "      <td>Happiness is a myth | Manchester United Fan | ...</td>\n",
       "      <td>11872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>SDEKenya</td>\n",
       "      <td>53648</td>\n",
       "      <td>280</td>\n",
       "      <td>We take Entertainment seriously! Ultimate Keny...</td>\n",
       "      <td>144816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Catalyst_Darsil</td>\n",
       "      <td>6006</td>\n",
       "      <td>1320</td>\n",
       "      <td>Lunatic psycho maniac💯💯😂\\nSocial influencer\\nT...</td>\n",
       "      <td>4535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>EdgarbrwnBrwn</td>\n",
       "      <td>2386</td>\n",
       "      <td>4999</td>\n",
       "      <td>Addicted to Reggae #VybezRadioke /RaceWalking ...</td>\n",
       "      <td>37867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>NzauPriscilla</td>\n",
       "      <td>879</td>\n",
       "      <td>935</td>\n",
       "      <td>Finance and Banking (Mauritius 2019). Commerce...</td>\n",
       "      <td>6540</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       screen_name  followers  friends  \\\n",
       "0         mr_Okabi        656      493   \n",
       "1         SDEKenya      53648      280   \n",
       "2  Catalyst_Darsil       6006     1320   \n",
       "3    EdgarbrwnBrwn       2386     4999   \n",
       "4    NzauPriscilla        879      935   \n",
       "\n",
       "                                         description  tweets  \n",
       "0  Happiness is a myth | Manchester United Fan | ...   11872  \n",
       "1  We take Entertainment seriously! Ultimate Keny...  144816  \n",
       "2  Lunatic psycho maniac💯💯😂\\nSocial influencer\\nT...    4535  \n",
       "3  Addicted to Reggae #VybezRadioke /RaceWalking ...   37867  \n",
       "4  Finance and Banking (Mauritius 2019). Commerce...    6540  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,\n",
    "                       geocode=\"0.0236,37.9062,530km\", since_id = '2020-03-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_02 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_02.to_csv('df_02 users.csv')\n",
    "df_02.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>VinnieBaite</td>\n",
       "      <td>17620</td>\n",
       "      <td>854</td>\n",
       "      <td>●Twitter Professor ●Actor ●Film technologist ●...</td>\n",
       "      <td>14549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>_sisiosontogago</td>\n",
       "      <td>789</td>\n",
       "      <td>269</td>\n",
       "      <td>si mika ang katumbas ng habang-buhay</td>\n",
       "      <td>38105</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Dayliff</td>\n",
       "      <td>42799</td>\n",
       "      <td>28808</td>\n",
       "      <td>We are the leaders in Water &amp; Energy solutions...</td>\n",
       "      <td>6008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>iam_jaymoe</td>\n",
       "      <td>11919</td>\n",
       "      <td>674</td>\n",
       "      <td></td>\n",
       "      <td>23368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>wanjerililiank</td>\n",
       "      <td>516</td>\n",
       "      <td>611</td>\n",
       "      <td>born n brought up ni kiambu county. working in...</td>\n",
       "      <td>1843</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       screen_name  followers  friends  \\\n",
       "0      VinnieBaite      17620      854   \n",
       "1  _sisiosontogago        789      269   \n",
       "2          Dayliff      42799    28808   \n",
       "3       iam_jaymoe      11919      674   \n",
       "4   wanjerililiank        516      611   \n",
       "\n",
       "                                         description  tweets  \n",
       "0  ●Twitter Professor ●Actor ●Film technologist ●...   14549  \n",
       "1               si mika ang katumbas ng habang-buhay   38105  \n",
       "2  We are the leaders in Water & Energy solutions...    6008  \n",
       "3                                                      23368  \n",
       "4  born n brought up ni kiambu county. working in...    1843  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,\n",
    "                       geocode=\"0.0236,37.9062,530km\", since_id = '2020-02-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_03 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_03.to_csv('df_03 users.csv')\n",
    "df_03.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Vikemu_Ke</td>\n",
       "      <td>4232</td>\n",
       "      <td>3243</td>\n",
       "      <td>🌟\\nCfc⚽\\nMy Love For Football Is Underrated 😅\\...</td>\n",
       "      <td>18796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>GKILINDAH</td>\n",
       "      <td>2</td>\n",
       "      <td>25</td>\n",
       "      <td></td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>SamuelOmerhi</td>\n",
       "      <td>1064</td>\n",
       "      <td>1242</td>\n",
       "      <td>THE NEXT BIG NAME‼️</td>\n",
       "      <td>830</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Isebania_Finest</td>\n",
       "      <td>3447</td>\n",
       "      <td>4993</td>\n",
       "      <td>#Dm for business promotions and Ads. |We are h...</td>\n",
       "      <td>4620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>pizzainnke</td>\n",
       "      <td>1458</td>\n",
       "      <td>92</td>\n",
       "      <td>Giving you nothing but the best, yummy Pizza!</td>\n",
       "      <td>1301</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       screen_name  followers  friends  \\\n",
       "0        Vikemu_Ke       4232     3243   \n",
       "1        GKILINDAH          2       25   \n",
       "2     SamuelOmerhi       1064     1242   \n",
       "3  Isebania_Finest       3447     4993   \n",
       "4       pizzainnke       1458       92   \n",
       "\n",
       "                                         description  tweets  \n",
       "0  🌟\\nCfc⚽\\nMy Love For Football Is Underrated 😅\\...   18796  \n",
       "1                                                         40  \n",
       "2                                THE NEXT BIG NAME‼️     830  \n",
       "3  #Dm for business promotions and Ads. |We are h...    4620  \n",
       "4      Giving you nothing but the best, yummy Pizza!    1301  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,\n",
    "                       geocode=\"0.0236,37.9062,530km\", since_id = '2020-01-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_04 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_04.to_csv('df_04 users.csv')\n",
    "df_04.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>BeckyKwagala</td>\n",
       "      <td>620</td>\n",
       "      <td>599</td>\n",
       "      <td>Loves to cook. 🤗 Potential public speaker. Gun...</td>\n",
       "      <td>1274</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>Kubaimugo1</td>\n",
       "      <td>159</td>\n",
       "      <td>1132</td>\n",
       "      <td>Pauline njeris husband father of Wambui n Loui...</td>\n",
       "      <td>447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Kush_Kahima</td>\n",
       "      <td>2153</td>\n",
       "      <td>1527</td>\n",
       "      <td>Student🌺 || Mark 10:27 || @Arsenal die hard ||...</td>\n",
       "      <td>2303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>KarokiNjogu</td>\n",
       "      <td>1351</td>\n",
       "      <td>1252</td>\n",
       "      <td>Contributing To Entropy Since 1998.\\n🇰🇪 ➡️ 🌍</td>\n",
       "      <td>7693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>Jr_LincolnKE</td>\n",
       "      <td>122464</td>\n",
       "      <td>1592</td>\n",
       "      <td>Your Favourite Voice of Reason! Digital Market...</td>\n",
       "      <td>54243</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    screen_name  followers  friends  \\\n",
       "0  BeckyKwagala        620      599   \n",
       "1    Kubaimugo1        159     1132   \n",
       "2   Kush_Kahima       2153     1527   \n",
       "3   KarokiNjogu       1351     1252   \n",
       "4  Jr_LincolnKE     122464     1592   \n",
       "\n",
       "                                         description  tweets  \n",
       "0  Loves to cook. 🤗 Potential public speaker. Gun...    1274  \n",
       "1  Pauline njeris husband father of Wambui n Loui...     447  \n",
       "2  Student🌺 || Mark 10:27 || @Arsenal die hard ||...    2303  \n",
       "3       Contributing To Entropy Since 1998.\\n🇰🇪 ➡️ 🌍    7693  \n",
       "4  Your Favourite Voice of Reason! Digital Market...   54243  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,\n",
    "                       geocode=\"0.0236,37.9062,530km\", since_id = '2019-12-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_05 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_05.to_csv('df_05 users.csv')\n",
    "df_05.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Geo df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6000, 5)"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.concat([users_df, df_2, df_3])\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3162, 6)"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['popularity'] = df['followers'] - df['friends']\n",
    "df = df.sort_values(by = 'popularity', ascending = False)\n",
    "df = df.drop_duplicates(subset =\"screen_name\") \n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(62, 6)"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop all users with popularity scores less than 50k\n",
    "new_df = df[df.popularity >= 50000]\n",
    "new_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "      <th>popularity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1639</td>\n",
       "      <td>citizentvkenya</td>\n",
       "      <td>2715770</td>\n",
       "      <td>219</td>\n",
       "      <td>Kenya's Premier TV Station. Breaking News / En...</td>\n",
       "      <td>535035</td>\n",
       "      <td>2715551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>887</td>\n",
       "      <td>ntvkenya</td>\n",
       "      <td>2551044</td>\n",
       "      <td>262</td>\n",
       "      <td>LIVE STREAM: https://t.co/BrEeOBOxu6 _________...</td>\n",
       "      <td>491465</td>\n",
       "      <td>2550782</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1684</td>\n",
       "      <td>dailynation</td>\n",
       "      <td>2142307</td>\n",
       "      <td>262</td>\n",
       "      <td>Welcome to Daily Nation on Twitter, for instan...</td>\n",
       "      <td>553869</td>\n",
       "      <td>2142045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1269</td>\n",
       "      <td>RobertAlai</td>\n",
       "      <td>1498640</td>\n",
       "      <td>3353</td>\n",
       "      <td>The Equalizer | There’s no comfort in the trut...</td>\n",
       "      <td>244149</td>\n",
       "      <td>1495287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1569</td>\n",
       "      <td>CKirubi</td>\n",
       "      <td>1432476</td>\n",
       "      <td>1091</td>\n",
       "      <td>Media Owner | Entrepreneur | Mentor. A trailbl...</td>\n",
       "      <td>27725</td>\n",
       "      <td>1431385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>247</td>\n",
       "      <td>DorahManya</td>\n",
       "      <td>32044</td>\n",
       "      <td>367</td>\n",
       "      <td>Award winning Radio Presenter |God fearing |Yo...</td>\n",
       "      <td>189944</td>\n",
       "      <td>31677</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>230</td>\n",
       "      <td>holydave</td>\n",
       "      <td>34236</td>\n",
       "      <td>2798</td>\n",
       "      <td>PhD Student | Financial Modelling Expert | Art...</td>\n",
       "      <td>38324</td>\n",
       "      <td>31438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1742</td>\n",
       "      <td>Lasha254</td>\n",
       "      <td>54219</td>\n",
       "      <td>23019</td>\n",
       "      <td>I'm not your ordinary B 🙄 Girls who know sport...</td>\n",
       "      <td>11965</td>\n",
       "      <td>31200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1029</td>\n",
       "      <td>SKachima</td>\n",
       "      <td>34059</td>\n",
       "      <td>3419</td>\n",
       "      <td>Simba SC ❤️ManUtd ❤️ Libra ⭐ Good 🎶🇹🇿 Psalm 23✌️</td>\n",
       "      <td>33308</td>\n",
       "      <td>30640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1329</td>\n",
       "      <td>discjockeyvin</td>\n",
       "      <td>30528</td>\n",
       "      <td>356</td>\n",
       "      <td>PROFESSIONAL DJ (CLUB, MOBILE, TV &amp; RADIO, CON...</td>\n",
       "      <td>3133</td>\n",
       "      <td>30172</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>87 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         screen_name  followers  friends  \\\n",
       "1639  citizentvkenya    2715770      219   \n",
       "887         ntvkenya    2551044      262   \n",
       "1684     dailynation    2142307      262   \n",
       "1269      RobertAlai    1498640     3353   \n",
       "1569         CKirubi    1432476     1091   \n",
       "...              ...        ...      ...   \n",
       "247       DorahManya      32044      367   \n",
       "230         holydave      34236     2798   \n",
       "1742        Lasha254      54219    23019   \n",
       "1029        SKachima      34059     3419   \n",
       "1329   discjockeyvin      30528      356   \n",
       "\n",
       "                                            description  tweets  popularity  \n",
       "1639  Kenya's Premier TV Station. Breaking News / En...  535035     2715551  \n",
       "887   LIVE STREAM: https://t.co/BrEeOBOxu6 _________...  491465     2550782  \n",
       "1684  Welcome to Daily Nation on Twitter, for instan...  553869     2142045  \n",
       "1269  The Equalizer | There’s no comfort in the trut...  244149     1495287  \n",
       "1569  Media Owner | Entrepreneur | Mentor. A trailbl...   27725     1431385  \n",
       "...                                                 ...     ...         ...  \n",
       "247   Award winning Radio Presenter |God fearing |Yo...  189944       31677  \n",
       "230   PhD Student | Financial Modelling Expert | Art...   38324       31438  \n",
       "1742  I'm not your ordinary B 🙄 Girls who know sport...   11965       31200  \n",
       "1029   Simba SC ❤️ManUtd ❤️ Libra ⭐ Good 🎶🇹🇿 Psalm 23✌️   33308       30640  \n",
       "1329  PROFESSIONAL DJ (CLUB, MOBILE, TV & RADIO, CON...    3133       30172  \n",
       "\n",
       "[87 rows x 6 columns]"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keyword Scraper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>valleysOfGod</td>\n",
       "      <td>1972</td>\n",
       "      <td>1980</td>\n",
       "      <td>political scientist,,upcoming entrepreneur,,so...</td>\n",
       "      <td>1189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>spinkingske</td>\n",
       "      <td>696</td>\n",
       "      <td>117</td>\n",
       "      <td>The Bicycle is a simple Solution to some of th...</td>\n",
       "      <td>1944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>DNyarere</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td>601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>luckydun</td>\n",
       "      <td>127</td>\n",
       "      <td>270</td>\n",
       "      <td></td>\n",
       "      <td>741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>_matheka_a</td>\n",
       "      <td>587</td>\n",
       "      <td>384</td>\n",
       "      <td></td>\n",
       "      <td>8915</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    screen_name  followers  friends  \\\n",
       "0  valleysOfGod       1972     1980   \n",
       "1   spinkingske        696      117   \n",
       "2      DNyarere          7        0   \n",
       "3      luckydun        127      270   \n",
       "4    _matheka_a        587      384   \n",
       "\n",
       "                                         description  tweets  \n",
       "0  political scientist,,upcoming entrepreneur,,so...    1189  \n",
       "1  The Bicycle is a simple Solution to some of th...    1944  \n",
       "2                                                        601  \n",
       "3                                                        741  \n",
       "4                                                       8915  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,q = 'Kenya', since_id = '2020-07-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_4 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_4.to_csv('df_4 users.csv')\n",
    "df_4.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>DStv_Kenya</td>\n",
       "      <td>341008</td>\n",
       "      <td>12213</td>\n",
       "      <td>DStv is Kenya's top entertainment destination....</td>\n",
       "      <td>317107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>iam_warloque</td>\n",
       "      <td>412</td>\n",
       "      <td>398</td>\n",
       "      <td>#LFC #30BG</td>\n",
       "      <td>366</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>its_vus</td>\n",
       "      <td>1692</td>\n",
       "      <td>1446</td>\n",
       "      <td></td>\n",
       "      <td>9253</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>80daysbot</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>80 days of botting around the world :)</td>\n",
       "      <td>30836</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>TheKimutai</td>\n",
       "      <td>10702</td>\n",
       "      <td>3692</td>\n",
       "      <td>Let go your earthly tether. Enter the void. Em...</td>\n",
       "      <td>206293</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    screen_name  followers  friends  \\\n",
       "0    DStv_Kenya     341008    12213   \n",
       "1  iam_warloque        412      398   \n",
       "2       its_vus       1692     1446   \n",
       "3     80daysbot          9        0   \n",
       "4    TheKimutai      10702     3692   \n",
       "\n",
       "                                         description  tweets  \n",
       "0  DStv is Kenya's top entertainment destination....  317107  \n",
       "1                                         #LFC #30BG     366  \n",
       "2                                                       9253  \n",
       "3             80 days of botting around the world :)   30836  \n",
       "4  Let go your earthly tether. Enter the void. Em...  206293  "
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,q = 'Kenya', since_id = '2020-06-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_5 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_5.to_csv('df_5 users.csv')\n",
    "df_5.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>KJeremiahN</td>\n",
       "      <td>2109</td>\n",
       "      <td>660</td>\n",
       "      <td>Outgoing and easy to associate with...Any view...</td>\n",
       "      <td>109414</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>annitii</td>\n",
       "      <td>2402</td>\n",
       "      <td>2510</td>\n",
       "      <td>FOLLOWING 🇰🇪 KENYANS ONLY</td>\n",
       "      <td>6483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>DaimaGreenSpace</td>\n",
       "      <td>39</td>\n",
       "      <td>47</td>\n",
       "      <td>Advocating for inclusive GreenSpaces for all</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>geonal</td>\n",
       "      <td>1335</td>\n",
       "      <td>872</td>\n",
       "      <td>Eccl. 9:11. Husband. Dreamer. Father.\\nStudent...</td>\n",
       "      <td>49244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>mohd_Hassan_Ali</td>\n",
       "      <td>388</td>\n",
       "      <td>180</td>\n",
       "      <td>So humble for human beings &amp; Slave of ALLAH</td>\n",
       "      <td>1344</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       screen_name  followers  friends  \\\n",
       "0       KJeremiahN       2109      660   \n",
       "1          annitii       2402     2510   \n",
       "2  DaimaGreenSpace         39       47   \n",
       "3           geonal       1335      872   \n",
       "4  mohd_Hassan_Ali        388      180   \n",
       "\n",
       "                                         description  tweets  \n",
       "0  Outgoing and easy to associate with...Any view...  109414  \n",
       "1                          FOLLOWING 🇰🇪 KENYANS ONLY    6483  \n",
       "2       Advocating for inclusive GreenSpaces for all      37  \n",
       "3  Eccl. 9:11. Husband. Dreamer. Father.\\nStudent...   49244  \n",
       "4        So humble for human beings & Slave of ALLAH    1344  "
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,q = 'Kenya', since_id = '2020-05-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_6 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_6.to_csv('df_6 users.csv')\n",
    "df_6.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>_dazils_</td>\n",
       "      <td>467</td>\n",
       "      <td>165</td>\n",
       "      <td>Fine young man with an old man mind.</td>\n",
       "      <td>28108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>iamjoseh_</td>\n",
       "      <td>46577</td>\n",
       "      <td>741</td>\n",
       "      <td>A Human. Being. my back up account 👉🏾 @iam_joseh</td>\n",
       "      <td>249788</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Waambui</td>\n",
       "      <td>35726</td>\n",
       "      <td>68</td>\n",
       "      <td>“The capacity to learn is a gift; The ability ...</td>\n",
       "      <td>5992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>drewgash</td>\n",
       "      <td>59</td>\n",
       "      <td>48</td>\n",
       "      <td>African socialist admirer.</td>\n",
       "      <td>4038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>lewys_chris</td>\n",
       "      <td>94</td>\n",
       "      <td>421</td>\n",
       "      <td>Director Ndege Tausi https://t.co/fzuLaCzFfj A...</td>\n",
       "      <td>777</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   screen_name  followers  friends  \\\n",
       "0     _dazils_        467      165   \n",
       "1    iamjoseh_      46577      741   \n",
       "2      Waambui      35726       68   \n",
       "3     drewgash         59       48   \n",
       "4  lewys_chris         94      421   \n",
       "\n",
       "                                         description  tweets  \n",
       "0               Fine young man with an old man mind.   28108  \n",
       "1   A Human. Being. my back up account 👉🏾 @iam_joseh  249788  \n",
       "2  “The capacity to learn is a gift; The ability ...    5992  \n",
       "3                         African socialist admirer.    4038  \n",
       "4  Director Ndege Tausi https://t.co/fzuLaCzFfj A...     777  "
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,q = 'Kenya', since_id = '2020-04-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_7 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_7.to_csv('df_7 users.csv')\n",
    "df_7.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>AherezaJanice</td>\n",
       "      <td>1037</td>\n",
       "      <td>1199</td>\n",
       "      <td>Environmentalist🍃🌿🍀\\nDancer,less have fun 😅\\nB...</td>\n",
       "      <td>303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>_nt__tn_</td>\n",
       "      <td>319</td>\n",
       "      <td>299</td>\n",
       "      <td>tu81 ee26, music addiction, namneung × noey × ...</td>\n",
       "      <td>137550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>HonOuko</td>\n",
       "      <td>303</td>\n",
       "      <td>331</td>\n",
       "      <td>change is inevitable</td>\n",
       "      <td>522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>split2tee</td>\n",
       "      <td>523</td>\n",
       "      <td>521</td>\n",
       "      <td>Humanitarian at heart, educator and an ardent ...</td>\n",
       "      <td>4217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>Wellington_ILO</td>\n",
       "      <td>819</td>\n",
       "      <td>294</td>\n",
       "      <td>This is the Official Twitter Account for the I...</td>\n",
       "      <td>2308</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      screen_name  followers  friends  \\\n",
       "0   AherezaJanice       1037     1199   \n",
       "1        _nt__tn_        319      299   \n",
       "2         HonOuko        303      331   \n",
       "3       split2tee        523      521   \n",
       "4  Wellington_ILO        819      294   \n",
       "\n",
       "                                         description  tweets  \n",
       "0  Environmentalist🍃🌿🍀\\nDancer,less have fun 😅\\nB...     303  \n",
       "1  tu81 ee26, music addiction, namneung × noey × ...  137550  \n",
       "2                               change is inevitable     522  \n",
       "3  Humanitarian at heart, educator and an ardent ...    4217  \n",
       "4  This is the Official Twitter Account for the I...    2308  "
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,q = 'Kenya', since_id = '2020-03-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_8 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_8.to_csv('df_8 users.csv')\n",
    "df_8.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Am0n_0ff</td>\n",
       "      <td>163</td>\n",
       "      <td>149</td>\n",
       "      <td>Africa4Africa//\\n\\nAfrica should be equal trad...</td>\n",
       "      <td>6104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>ItsArnoldOduor</td>\n",
       "      <td>3604</td>\n",
       "      <td>3834</td>\n",
       "      <td>Start small &amp; stay consistent..\\n\\n+254🇰🇪</td>\n",
       "      <td>5480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>voyegram</td>\n",
       "      <td>78</td>\n",
       "      <td>243</td>\n",
       "      <td>Travels and Tours Company\\n\\nVOVEGRAM TOURS &amp; ...</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Scanoma</td>\n",
       "      <td>595</td>\n",
       "      <td>392</td>\n",
       "      <td>日々是ぜ〜ぜ〜</td>\n",
       "      <td>41943</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>RoselyneNyabon4</td>\n",
       "      <td>35</td>\n",
       "      <td>58</td>\n",
       "      <td></td>\n",
       "      <td>876</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       screen_name  followers  friends  \\\n",
       "0         Am0n_0ff        163      149   \n",
       "1   ItsArnoldOduor       3604     3834   \n",
       "2         voyegram         78      243   \n",
       "3          Scanoma        595      392   \n",
       "4  RoselyneNyabon4         35       58   \n",
       "\n",
       "                                         description  tweets  \n",
       "0  Africa4Africa//\\n\\nAfrica should be equal trad...    6104  \n",
       "1          Start small & stay consistent..\\n\\n+254🇰🇪    5480  \n",
       "2  Travels and Tours Company\\n\\nVOVEGRAM TOURS & ...      27  \n",
       "3                                            日々是ぜ〜ぜ〜   41943  \n",
       "4                                                        876  "
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,q = 'Kenya', since_id = '2020-02-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_9 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_9.to_csv('df_9 users.csv')\n",
    "df_9.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>kenya_pu</td>\n",
       "      <td>371</td>\n",
       "      <td>232</td>\n",
       "      <td>現存する最古のｼﾝﾒの深海に溺れる青担ﾄﾆ担箱推し。妄想絵を描く酔払い。成人式は遥か昔 妄想...</td>\n",
       "      <td>58113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>nanga_tp</td>\n",
       "      <td>18993</td>\n",
       "      <td>2040</td>\n",
       "      <td>Luyotilo Holdings || Stay patient and trust yo...</td>\n",
       "      <td>37994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>KenInvest</td>\n",
       "      <td>20513</td>\n",
       "      <td>384</td>\n",
       "      <td>Kenya Investment Authority (KenInvest) is a Go...</td>\n",
       "      <td>7503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Njeriwaridi</td>\n",
       "      <td>334</td>\n",
       "      <td>143</td>\n",
       "      <td>Investment. Impact. Travel.Environment\\n\\nWorr...</td>\n",
       "      <td>14381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>ilegacy15</td>\n",
       "      <td>54</td>\n",
       "      <td>239</td>\n",
       "      <td>My truth might be a lie.</td>\n",
       "      <td>1199</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   screen_name  followers  friends  \\\n",
       "0     kenya_pu        371      232   \n",
       "1     nanga_tp      18993     2040   \n",
       "2    KenInvest      20513      384   \n",
       "3  Njeriwaridi        334      143   \n",
       "4    ilegacy15         54      239   \n",
       "\n",
       "                                         description  tweets  \n",
       "0  現存する最古のｼﾝﾒの深海に溺れる青担ﾄﾆ担箱推し。妄想絵を描く酔払い。成人式は遥か昔 妄想...   58113  \n",
       "1  Luyotilo Holdings || Stay patient and trust yo...   37994  \n",
       "2  Kenya Investment Authority (KenInvest) is a Go...    7503  \n",
       "3  Investment. Impact. Travel.Environment\\n\\nWorr...   14381  \n",
       "4                           My truth might be a lie.    1199  "
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = tweepy.Cursor(auth_api.search,q = 'Kenya', since_id = '2020-01-01').items(2000)\n",
    "users = []\n",
    "followers = []\n",
    "friends = []\n",
    "description = []\n",
    "tweet_count = []\n",
    "for status in tweets:\n",
    "    name = status.user.screen_name\n",
    "    desc = status.user.description\n",
    "    number_of_tweets = status.user.statuses_count\n",
    "    following = status.user.friends_count\n",
    "    follower = status.user.followers_count\n",
    "    users.append(name)\n",
    "    followers.append(follower)\n",
    "    friends.append(following)\n",
    "    description.append(desc)\n",
    "    tweet_count.append(number_of_tweets) \n",
    "\n",
    "data = {'screen_name':users, 'followers':followers, 'friends':friends, 'description':description, 'tweets':tweet_count}\n",
    "df_0 = pd.DataFrame(data)\n",
    "    \n",
    "#     df['screen_name'] = users\n",
    "#     df['location'] = location\n",
    "    \n",
    "df_0.to_csv('df_0 users.csv')\n",
    "df_0.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Remaining dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(14000, 5)"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_k = pd.concat([df_4, df_5, df_6, df_7, df_8, df_9, df_0])\n",
    "df_k.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(116, 6)"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_k['popularity'] = df_k['followers'] - df_k['friends']\n",
    "new_df_2 = df_k[df_k.popularity >= 50000]\n",
    "new_df_2 = new_df_2.drop_duplicates(subset =\"screen_name\") \n",
    "new_df_2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>screen_name</th>\n",
       "      <th>followers</th>\n",
       "      <th>friends</th>\n",
       "      <th>description</th>\n",
       "      <th>tweets</th>\n",
       "      <th>popularity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>43</td>\n",
       "      <td>RoadAlertsKE</td>\n",
       "      <td>162406</td>\n",
       "      <td>490</td>\n",
       "      <td>Official Twitter Account for Road Alerts KE. S...</td>\n",
       "      <td>123136</td>\n",
       "      <td>161916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>79</td>\n",
       "      <td>coldtusker</td>\n",
       "      <td>147140</td>\n",
       "      <td>3716</td>\n",
       "      <td>Simply me...</td>\n",
       "      <td>361943</td>\n",
       "      <td>143424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>91</td>\n",
       "      <td>pressecitron</td>\n",
       "      <td>649614</td>\n",
       "      <td>2160</td>\n",
       "      <td>Connectez-vous à l'innovation ! #Pressecitron1...</td>\n",
       "      <td>64786</td>\n",
       "      <td>647454</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>105</td>\n",
       "      <td>ntvkenya</td>\n",
       "      <td>2551233</td>\n",
       "      <td>262</td>\n",
       "      <td>LIVE STREAM: https://t.co/BrEeOBOxu6 _________...</td>\n",
       "      <td>491479</td>\n",
       "      <td>2550971</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>110</td>\n",
       "      <td>CofekRebranded</td>\n",
       "      <td>139997</td>\n",
       "      <td>19373</td>\n",
       "      <td>COFEK is Kenya and Africa's Premier Consumer P...</td>\n",
       "      <td>44135</td>\n",
       "      <td>120624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>957</td>\n",
       "      <td>TheKenyanPost</td>\n",
       "      <td>51940</td>\n",
       "      <td>418</td>\n",
       "      <td>News, Politics, Entertainment, Gossip, Feature...</td>\n",
       "      <td>326473</td>\n",
       "      <td>51522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1146</td>\n",
       "      <td>KBCChannel1</td>\n",
       "      <td>310081</td>\n",
       "      <td>1750</td>\n",
       "      <td>Kenya's National Broadcaster, bringing you sto...</td>\n",
       "      <td>203064</td>\n",
       "      <td>308331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1151</td>\n",
       "      <td>swahilitimes</td>\n",
       "      <td>581701</td>\n",
       "      <td>0</td>\n",
       "      <td>Habari | Elimu | Burudani | swahilitimes@gmail...</td>\n",
       "      <td>53538</td>\n",
       "      <td>581701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1189</td>\n",
       "      <td>WMutunga</td>\n",
       "      <td>1007081</td>\n",
       "      <td>847</td>\n",
       "      <td>INSPIRING THE YOUTH TO CREATE BETTER TOMORROWS.</td>\n",
       "      <td>34382</td>\n",
       "      <td>1006234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1217</td>\n",
       "      <td>ReutersAfrica</td>\n",
       "      <td>738848</td>\n",
       "      <td>341</td>\n",
       "      <td>Reuters Africa provides a feed of our latest n...</td>\n",
       "      <td>39437</td>\n",
       "      <td>738507</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>116 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         screen_name  followers  friends  \\\n",
       "43      RoadAlertsKE     162406      490   \n",
       "79        coldtusker     147140     3716   \n",
       "91      pressecitron     649614     2160   \n",
       "105         ntvkenya    2551233      262   \n",
       "110   CofekRebranded     139997    19373   \n",
       "...              ...        ...      ...   \n",
       "957    TheKenyanPost      51940      418   \n",
       "1146     KBCChannel1     310081     1750   \n",
       "1151    swahilitimes     581701        0   \n",
       "1189        WMutunga    1007081      847   \n",
       "1217   ReutersAfrica     738848      341   \n",
       "\n",
       "                                            description  tweets  popularity  \n",
       "43    Official Twitter Account for Road Alerts KE. S...  123136      161916  \n",
       "79                                         Simply me...  361943      143424  \n",
       "91    Connectez-vous à l'innovation ! #Pressecitron1...   64786      647454  \n",
       "105   LIVE STREAM: https://t.co/BrEeOBOxu6 _________...  491479     2550971  \n",
       "110   COFEK is Kenya and Africa's Premier Consumer P...   44135      120624  \n",
       "...                                                 ...     ...         ...  \n",
       "957   News, Politics, Entertainment, Gossip, Feature...  326473       51522  \n",
       "1146  Kenya's National Broadcaster, bringing you sto...  203064      308331  \n",
       "1151  Habari | Elimu | Burudani | swahilitimes@gmail...   53538      581701  \n",
       "1189    INSPIRING THE YOUTH TO CREATE BETTER TOMORROWS.   34382     1006234  \n",
       "1217  Reuters Africa provides a feed of our latest n...   39437      738507  \n",
       "\n",
       "[116 rows x 6 columns]"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mega Df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_df = pd.concat([new_df, new_df_2])\n",
    "full_df = full_df.drop_duplicates(subset =\"screen_name\") \n",
    "full_df.to_csv('mega1.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [Back to top](#Tweeps-scraping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}