Each crawler is built as part of another project. Different crawler techs are used:
- Selenium
- BeautifulSoup
- Scrapy
- Scholarly
Other possible crawlers that may speed up code flow (not used yet):
- serpAPI
- Octoparse
Dataset consists of 32,240 records of Google Scholar profiles from researchers affiliated with top 20 universities in Canada. Columns are GUID, full name, list of research interests, university name, and number of total citations per researcher.