Beginner approach to web scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img style=\"float: right;\" src=\"pycon_africa_logo.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Beginner approach to Web Scraping\n",
    "\n",
    "![Webscraping](web-scraping.png)\n",
    "\n",
    "### * Why Web Scraping?\n",
    "### * What Is Web Scraping? Is Web Scraping Legal?\n",
    "### * Why Python For Web Scraping?\n",
    "### * How Does Web Scraping work?\n",
    "### * Libraries used for Web Scraping\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Imagine you want to pull a large amount of data from websites and you want to do it quickly as possible. How would you do it without manually going to each wbsite and getting the data?\n",
    "\n",
    "## Why Web Scraping?\n",
    "Price Comparison, Email address gathering, Social Media Scraping, Research and Development, Job listing etc.\n",
    "##  What is webscraping? Is it legal?\n",
    "It is an authomated method used to extract large amount of data. There are different ways of to scrape websites such as online services , API's or writing your own code.\n",
    "## Is it Legal?\n",
    "## Why python for web scraping?\n",
    "Ease of use, Large collection of libraries, Easily Understandable Syntax, Small code Large task, community.\n",
    "## How does web scraping works?\n",
    "To extract data using web scraping with python, you need to follow these basic steps:\n",
    "1. Find the URL that you want to scrape\n",
    "2. Inspecting the page\n",
    "3. Find the data you want to extract\n",
    "4. Write the code\n",
    "5. Run the code an dextract the data\n",
    "6. Store the data in the required format\n",
    "\n",
    "## Libraries used for webscraping\n",
    "* BeautifulSoup: is a python library for parsing HTML and XML document\n",
    "* Pandas: python library for data manipulation and analysis\n",
    "* Requests: is a popular alternative to the standard library's HTTP handling (e.g. urllib2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# library importation\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = requests.get('https://steemit.com/nigeria/@leopantro/50-yoruba-proverbs-and-idioms')\n",
    "soup = BeautifulSoup(res.content,'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extracting necessary informations\n",
    "yoruba = []\n",
    "english = []\n",
    "for ol in soup.findAll('ol'):\n",
    "    proverb = ol.find('li')\n",
    "    yoruba.append(proverb.text)\n",
    "\n",
    "for paragraph in soup.findAll('p'):\n",
    "    if paragraph.text.startswith(\"Meaning\"):\n",
    "        english.append(paragraph.text.split(':')[1].strip())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Yoruba</th>\n",
       "      <th>Meaning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ile oba t'o jo, ewa lo busi</td>\n",
       "      <td>Necessity is mother of invention, creativity i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Gbogbo alangba lo d'anu dele, a ko mo eyi t'in...</td>\n",
       "      <td>Everyone looks the same on the outside but eve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ile la ti n ko eso re ode</td>\n",
       "      <td>A man cannot give what he does not have good o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A pę ko to jęun, ki ję ibaję</td>\n",
       "      <td>It is more profitable to exercise patience whi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Eewu bę loko Longę, Longę fun ara rę eewu ni</td>\n",
       "      <td>You should be extremely careful of situations ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Bi Ēēgun nla ba ni ohùn o ri gontò, gontò na a...</td>\n",
       "      <td>If an important man does not respect those les...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Kò sí ęni tí ó ma gùn ęşin tí kò ní ju ìpàkó. ...</td>\n",
       "      <td>Your status in life dictates your attitude tow...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Bí abá so òkò sójà ará ilé eni ní bá;</td>\n",
       "      <td>Be careful what you do unto others it may retu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Agba ki wa loja, ki ori omo titun o wo.</td>\n",
       "      <td>Behave in a mature manner so avoid bad reputat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Adìẹ funfun kò mọ ara rẹ̀lágbà</td>\n",
       "      <td>Respect yourself</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Ọbẹ̀ kìí gbé inú àgbà mì</td>\n",
       "      <td>You should be able to keep secrets</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>À ń pe gbẹ́nàgbẹ́nà ẹyẹ àkókó ń yọjú</td>\n",
       "      <td>Never think too highly of yourself</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Díẹ̀ díẹ̀ nimú ẹlẹ́dẹ̀ẹ́ fi ń wọgbà</td>\n",
       "      <td>Attend to a small problem before it becomes un...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Ilé ọba tójó ẹwà ló bùsi</td>\n",
       "      <td>Every cloud has a silver lining</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>A ti kekee e ti p’eitan u’oko i, do ba d’agba ...</td>\n",
       "      <td>Training of a child commences when the child i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Oiho buuk’ e ka koolo bi s’upekun da le da ma.</td>\n",
       "      <td>Appearance is deceptive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Onen e n’oogun adoado ee gb’ ayinyan win.</td>\n",
       "      <td>One should avoid any thing or act that will ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>D’iaye ba a s’oko s’oja, o le la a ba ia l’onen.</td>\n",
       "      <td>avoid  doing  things  that  will  have  seriou...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Di ulu be e don panen panen, a ka jo ni melo m...</td>\n",
       "      <td>Avoid doing things that will  have serious rep...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Oju ya a b’onen t’ale e e m’owuo kutu s’ipen.</td>\n",
       "      <td>The success of a venture requires early planni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Iy’o wu mi e wue, oun e m’omaiye meji jeun oto...</td>\n",
       "      <td>Every human being has special characteristics ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Ijo je o we yon i.</td>\n",
       "      <td>There is joy in cooperative effort</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Omi yi iay’ a mon ee san gb’onen ee.</td>\n",
       "      <td>Whatever one is destined to become or achieve ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>O so si ko du ka, o wo si ko du see</td>\n",
       "      <td>The problem is enigmatic, defying solution.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Die e je ma n’aja o siwin ku i</td>\n",
       "      <td>One should not over do things. There should be...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Uun yi agbalagba ri n’oiho joko, omoiton le ri...</td>\n",
       "      <td>Experience is the best teacher</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Igi so so o y’a a ghonen ‘ju, a to ke iye e ti...</td>\n",
       "      <td>To be forewarned, is to be forearmed.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Onen p’ aoko s’ oja oun oja p’ aoko si.</td>\n",
       "      <td>Anyone who gives will receive.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Se s’ale oun e je Seike i.</td>\n",
       "      <td>Benevolence leaves a lasting impression.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Aja e e gbagbe l’oiye owuo.</td>\n",
       "      <td>It is good to show appreciation for any good d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Oj’ ole igbad’e gbo i</td>\n",
       "      <td>If a man does not work, he must not eat.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Li suu e fon wa’a keneun i.</td>\n",
       "      <td>He who is patient can perform an impossible task.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Ubee eese l’use, onen yo gbe jake oun o se l’u...</td>\n",
       "      <td>He who perseveres to the end will be rewarded</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Igbe ‘a ga o we s’ugwaju ipa’uun.</td>\n",
       "      <td>Pride goes before a fall.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Aja e e j, ogungun ya gbe ko ghon o ‘on.</td>\n",
       "      <td>It is one’s responsibility to take care of wha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>Onen ma nen a’e do fi ghon n’onen, uya a a je ...</td>\n",
       "      <td>He who does not help himself but relies on oth...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Ko omo e do le mu usimi ko e.</td>\n",
       "      <td>Train your child so that he can give you peace...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Den a’a e n’awo yo gba e.</td>\n",
       "      <td>Cut your coat according to your size.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>E e m’oiho Ade gwole.</td>\n",
       "      <td>This means that nobody disrespects a crowned i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>Li t’onen e e h’oi osan da m’aipan.</td>\n",
       "      <td>Someone in an influential position should use ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Uun yi wo ba a gbe oun wa ka i, we le gbe ila ...</td>\n",
       "      <td>Whatever you sow, that you will reap;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Ohun pele, o yo obi n’ apo; ohun lile yo uda n...</td>\n",
       "      <td>The tongue is a two-edged sword.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>A’a e e wowo di l’otu e m a le gbe un.</td>\n",
       "      <td>Each individual must undertake his responsibil...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>E e do da ma ja, sugbon aja i gb’uha o so o i.</td>\n",
       "      <td>Emphasizes the importance of closeness of indi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Omi yi iay’a a mon e e san gb’onen e.</td>\n",
       "      <td>God’s timing is always perfect</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Esin e e da onen da ma ton gun.</td>\n",
       "      <td>Never give up</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Di l’ejo ba a ma ejo n’ebi e ni i pe n’oiho uk...</td>\n",
       "      <td>If you own up to your guilt, you have tried to...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>Ole yo ji kakaki oba, o ku boon ti a fon.</td>\n",
       "      <td>Nemesis will always catch up with a wrongdoer.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>Inon ukoko didun oun eko fifun ti jade i.</td>\n",
       "      <td>Do not judge a book by its cover.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>O nukan je nukan pawo ma.</td>\n",
       "      <td>A selfish individual.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               Yoruba  \\\n",
       "0                         Ile oba t'o jo, ewa lo busi   \n",
       "1   Gbogbo alangba lo d'anu dele, a ko mo eyi t'in...   \n",
       "2                           Ile la ti n ko eso re ode   \n",
       "3                        A pę ko to jęun, ki ję ibaję   \n",
       "4        Eewu bę loko Longę, Longę fun ara rę eewu ni   \n",
       "5   Bi Ēēgun nla ba ni ohùn o ri gontò, gontò na a...   \n",
       "6   Kò sí ęni tí ó ma gùn ęşin tí kò ní ju ìpàkó. ...   \n",
       "7               Bí abá so òkò sójà ará ilé eni ní bá;   \n",
       "8             Agba ki wa loja, ki ori omo titun o wo.   \n",
       "9                      Adìẹ funfun kò mọ ara rẹ̀lágbà   \n",
       "10                           Ọbẹ̀ kìí gbé inú àgbà mì   \n",
       "11               À ń pe gbẹ́nàgbẹ́nà ẹyẹ àkókó ń yọjú   \n",
       "12                Díẹ̀ díẹ̀ nimú ẹlẹ́dẹ̀ẹ́ fi ń wọgbà   \n",
       "13                           Ilé ọba tójó ẹwà ló bùsi   \n",
       "14  A ti kekee e ti p’eitan u’oko i, do ba d’agba ...   \n",
       "15     Oiho buuk’ e ka koolo bi s’upekun da le da ma.   \n",
       "16          Onen e n’oogun adoado ee gb’ ayinyan win.   \n",
       "17   D’iaye ba a s’oko s’oja, o le la a ba ia l’onen.   \n",
       "18  Di ulu be e don panen panen, a ka jo ni melo m...   \n",
       "19      Oju ya a b’onen t’ale e e m’owuo kutu s’ipen.   \n",
       "20  Iy’o wu mi e wue, oun e m’omaiye meji jeun oto...   \n",
       "21                                 Ijo je o we yon i.   \n",
       "22               Omi yi iay’ a mon ee san gb’onen ee.   \n",
       "23                O so si ko du ka, o wo si ko du see   \n",
       "24                     Die e je ma n’aja o siwin ku i   \n",
       "25  Uun yi agbalagba ri n’oiho joko, omoiton le ri...   \n",
       "26  Igi so so o y’a a ghonen ‘ju, a to ke iye e ti...   \n",
       "27            Onen p’ aoko s’ oja oun oja p’ aoko si.   \n",
       "28                         Se s’ale oun e je Seike i.   \n",
       "29                        Aja e e gbagbe l’oiye owuo.   \n",
       "30                              Oj’ ole igbad’e gbo i   \n",
       "31                        Li suu e fon wa’a keneun i.   \n",
       "32  Ubee eese l’use, onen yo gbe jake oun o se l’u...   \n",
       "33                  Igbe ‘a ga o we s’ugwaju ipa’uun.   \n",
       "34           Aja e e j, ogungun ya gbe ko ghon o ‘on.   \n",
       "35  Onen ma nen a’e do fi ghon n’onen, uya a a je ...   \n",
       "36                      Ko omo e do le mu usimi ko e.   \n",
       "37                          Den a’a e n’awo yo gba e.   \n",
       "38                              E e m’oiho Ade gwole.   \n",
       "39                Li t’onen e e h’oi osan da m’aipan.   \n",
       "40  Uun yi wo ba a gbe oun wa ka i, we le gbe ila ...   \n",
       "41  Ohun pele, o yo obi n’ apo; ohun lile yo uda n...   \n",
       "42             A’a e e wowo di l’otu e m a le gbe un.   \n",
       "43     E e do da ma ja, sugbon aja i gb’uha o so o i.   \n",
       "44              Omi yi iay’a a mon e e san gb’onen e.   \n",
       "45                    Esin e e da onen da ma ton gun.   \n",
       "46  Di l’ejo ba a ma ejo n’ebi e ni i pe n’oiho uk...   \n",
       "47          Ole yo ji kakaki oba, o ku boon ti a fon.   \n",
       "48          Inon ukoko didun oun eko fifun ti jade i.   \n",
       "49                          O nukan je nukan pawo ma.   \n",
       "\n",
       "                                              Meaning  \n",
       "0   Necessity is mother of invention, creativity i...  \n",
       "1   Everyone looks the same on the outside but eve...  \n",
       "2   A man cannot give what he does not have good o...  \n",
       "3   It is more profitable to exercise patience whi...  \n",
       "4   You should be extremely careful of situations ...  \n",
       "5   If an important man does not respect those les...  \n",
       "6   Your status in life dictates your attitude tow...  \n",
       "7   Be careful what you do unto others it may retu...  \n",
       "8   Behave in a mature manner so avoid bad reputat...  \n",
       "9                                    Respect yourself  \n",
       "10                 You should be able to keep secrets  \n",
       "11                 Never think too highly of yourself  \n",
       "12  Attend to a small problem before it becomes un...  \n",
       "13                    Every cloud has a silver lining  \n",
       "14  Training of a child commences when the child i...  \n",
       "15                            Appearance is deceptive  \n",
       "16  One should avoid any thing or act that will ha...  \n",
       "17  avoid  doing  things  that  will  have  seriou...  \n",
       "18  Avoid doing things that will  have serious rep...  \n",
       "19  The success of a venture requires early planni...  \n",
       "20  Every human being has special characteristics ...  \n",
       "21                 There is joy in cooperative effort  \n",
       "22  Whatever one is destined to become or achieve ...  \n",
       "23        The problem is enigmatic, defying solution.  \n",
       "24  One should not over do things. There should be...  \n",
       "25                     Experience is the best teacher  \n",
       "26              To be forewarned, is to be forearmed.  \n",
       "27                     Anyone who gives will receive.  \n",
       "28           Benevolence leaves a lasting impression.  \n",
       "29  It is good to show appreciation for any good d...  \n",
       "30           If a man does not work, he must not eat.  \n",
       "31  He who is patient can perform an impossible task.  \n",
       "32      He who perseveres to the end will be rewarded  \n",
       "33                          Pride goes before a fall.  \n",
       "34  It is one’s responsibility to take care of wha...  \n",
       "35  He who does not help himself but relies on oth...  \n",
       "36  Train your child so that he can give you peace...  \n",
       "37              Cut your coat according to your size.  \n",
       "38  This means that nobody disrespects a crowned i...  \n",
       "39  Someone in an influential position should use ...  \n",
       "40              Whatever you sow, that you will reap;  \n",
       "41                   The tongue is a two-edged sword.  \n",
       "42  Each individual must undertake his responsibil...  \n",
       "43  Emphasizes the importance of closeness of indi...  \n",
       "44                     God’s timing is always perfect  \n",
       "45                                      Never give up  \n",
       "46  If you own up to your guilt, you have tried to...  \n",
       "47     Nemesis will always catch up with a wrongdoer.  \n",
       "48                  Do not judge a book by its cover.  \n",
       "49                              A selfish individual.  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({'Yoruba':yoruba,'Meaning':english})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}