Skip to content

Latest commit

 

History

History
48 lines (36 loc) · 1.46 KB

README.md

File metadata and controls

48 lines (36 loc) · 1.46 KB
Raw

linkin

A minimal, async Clojure web crawling library.

("linkin" from Linkin Park's "Crawling", and because links)

Features

  • Uses http-kit and core.async for async fetching
  • Uses Jsoup to reliably extract links from scraped pages
  • Allows the user to pass in their own function to handle the body content of scraped pages
  • Respects the robots.txt of the target website (allow & disallow rules)

Usage

Right now it's not on clojars yet, so you'll need to build and install it locally using a method of your own choosing (lein-localrepo is a good choice)

Then include the following dependency in your project.clj:

[linkin "0.1.0-SNAPSHOT"]

Then:

(require 'linkin.core)
(linkin.core/crawl "http://example.com" linkin.core/simple-body-parser)

Todo

  • Throttling (ie don't DoS target sites...)
  • Control throttling using the spider delay directives in robots.txt
  • Control of max depth / number of pages crawled
  • Ability to spider across domains
  • Pass options through to http-kit (eg following redirects)
  • Filtering by content type
  • Stats while running
  • Better URL normalization (for detecting URLs we've seen before) - see http://en.wikipedia.org/wiki/URL_normalization

License

Copyright © 2014 Rory Gibson

Distributed under the Eclipse Public License version 1.0.