Skip to content

Commit

Permalink
modify sehsanm#2 crawler README file
Browse files Browse the repository at this point in the history
  • Loading branch information
Sahand504 committed Dec 11, 2018
1 parent 6155952 commit 6defc78
Showing 1 changed file with 8 additions and 8 deletions.
16 changes: 8 additions & 8 deletions scripts/crawler/crawler/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@
simple crawler using scrapy framework . scrapy have useful features like : avoiding duplicate urls, limitation on depth , defining request rate and ... . \n
you need to install scrapy :
`pip install scrapy`
and run the crawlers : <br /> <br />
and run the crawlers : <br />

`scrapy crawl hamshahri -o ham.json` <br />
`scrapy crawl blogspider -o ham.json` <br />
`scrapy crawl blogfa -o ham.json` <br />
`scrapy crawl blogsky -o ham.json` <br />
`scrapy crawl dorsablog -o ham.json` <br />
`scrapy crawl mihanblog -o ham.json` <br />
`scrapy crawl persianblog -o ham.json` <br /> <br />
`scrapy crawl hamshahri -o hamshahri.json` <br />
`scrapy crawl blog -o blog.json` <br />
`scrapy crawl blogfa -o blogfa.json` <br />
`scrapy crawl blogsky -o blogsky.json` <br />
`scrapy crawl dorsablog -o dorsablog.json` <br />
`scrapy crawl mihanblog -o mihanblog.json` <br />
`scrapy crawl persianblog -o persianblog.json` <br />

settings can be found in :'crawler/settings.py' <br />
spiders are available in 'crawler/spiders'
Expand Down

0 comments on commit 6defc78

Please sign in to comment.