bs4 implementation

[blacksmithop]

Change Overview:

Uses bs4 to scrape requirements mentioned in issue
Current format:

question, url and source_verified was acquired relatively easily. answer is the combination of commands being mentioned in the accepted answer. Due to the nature of the filter being applied (ranked by votes) only accepted answer is taken as the answer. manual_verified and final_answer defaults to False and None since they need to be verified manually.

Testing Overview:

Creating a dict for each url and appending to a list is memory intensive and takes a while to finish.
Can be replaced with writing to csv to remove that overhead.

Related Issues / PRs:

Further testing will have to be done remotely with larger PAGE_MAX.

[navan0]

@GopikrishnanSasikumar check and merge

[gksoriginals]

Wouldn't it be great if we can split url_to_data() into small methods. This violate SLAP

[blacksmithop]

Actually, that sounds better than having all the operations be done under one hood. I'll look into what can be moved around.

[blacksmithop]

Rewrote scraping logic to fit a class structure.
DataSetGen now has class variables PAGE_MAX and platform_to_tag . Use of class vars to share current data among methods.
All operations including collection of urls, and the operations under url_to_data are now class methods.
Namely get_question, is_verified, (returns a Boolean as well the url if applicable), get_answer.
This was run on a remote Linux environment with a PAGE_MAX of 1 and one tag for askubuntu. (screenshot)
Only accepted answers are considered by data_from_url (responsible for building the collection of data items)

[navan0]

Looks good