Link-scraper-in-python

A Python script to scrap all links in a given website using requests and Beautiful soup

Detailed article

The detailed article on how to scrap all links in a given website can be found on my blog with an article titled How to extract all website link in Python.

Getting started

To get started exploring this code you might have to clone or download the repository just as shown below;

-> git clone https://github.com/Kalebu/Link-scraper-in-python
-> cd Link-scraper-in-python

Dependencies

To successfully run this code you're supposed to have requests and BeautifulSoup libary installed on your machine

-> pip install requests
-> pip install beautifulsoup4

Running

Now that we have everything already set up , lets run our code just as shown below;

Link-scraper-in-python -> python link_spider.py
Enter URL of the site : https://kalebujordan.com/
['#content', 'https://kalebujordan.com/', 'https://kalebujordan.com/', 'https://kalebujordan.com/category/projects/...]

Explore it

Now keep explore it by testing it with various input links to see what links it will scrap

Give it a star

Did you find this information useful, then give it a star

Credits

All the credits to kalebu