Create a Crawler tool to build a corpus

[sehsanm]

Create a Crawler tool to collect information from a set of websites and links to build a corpus

[abb4s]

I've created a simple one by 'scrapy' , you need to install scrapy and run this in directory of project
scrapy crawl quotes -o ham.json . and also you can change websites urls for crawling in 'spiders/quotes_spiders.py' . I found scrapy useful because it have features like html parsing , defining request rate, avoiding duplicate urls and ... . but we have some requirements that I think we should solve : 1.we want just Persian words 2.there are many phrases in websites which are not sentences and we don't want them in our corpus . 3.there are many repetitive phrases in all pages of each websits.

the scrapy crawler for hamshahri :
nlpcrawler.zip

[sehsanm]

Perfect this is kind of an implementation of this story. Can you move this to repository and write a small documentation for it?

Please also not that we need to have a corpus of size X10 million sentences

[abb4s]

I've sent you a pull request. I don't think that it would be good way to control sentences count while crawling . but we can limit size of crawled text in GB measure. I think 3GB is enough size for 10M sentences .