Store tweets from a Twitter search results, using browser scraping
This application uses Python and the browser (controlled through Selenium) to load a page and scrape the contents and save the data. The approach here loads the DOM (not possible when using plain HTTP request) and also adds in waiting and scrolling logic so that dynamic elements can be pulled in.
This project is aimed at scraping Tweets on a search, where scrolling is needed to load more. It could be used for a user timeline though.
- Python 3
- Chrome browser
$ git clone [email protected]:MichaelCurrin/scroll-and-scrape.git
$ cd scroll-and-scrapeInstall Python 3.
Create a virtual environment and activate it.
Install project packages:
$ pip install -r requirements.txt
$ pip install -r requirements-dev.txt$ cd scrollscrape
$ python main.pyThis is a simple Python 3 application, based on existing scripts which are included in the research directory. See also gist.
The goal is to get all the Twitter tweets for a search query, going back as far as possible.
Using the Twitter API is restrictive - it only gives a week worth of data. Note that to keep this application simple, only the tweet ID needs to be stored and none of the tweet or author data. Since, once you have a tweet ID no matter how old, you can use the Twitter API to get all tweet and author data for a given tweet ID.
So far the main script works okay, but it not getting all known tweets for a particular known so it needs improvement. The format can be improved - for now it is printing to stdout which can be redirected to a text file.
Also, this could be improved for efficiency using a headless browser. It could also use a more modern library such as requests-html.
Released under MIT by @MichaelCurrin.