A simple webcrawler that will scrape every URL linked to on a given site, and will put it into a links.txt file. It will then randomly pick a link from links.txt and will search that page for links, and will then add those to links.txt. This will be repeated indefinitely until the program is turned off
-
Notifications
You must be signed in to change notification settings - Fork 0
A simple webcrawler written in ruby
License
OpenSauce04/LinkShovel
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
A simple webcrawler written in ruby
Topics
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published