Home
This is the public wiki for the Heritrix archival crawler project.
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
Heritrix (sometimes spelled heretrix, or misspelled or mis-said as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt.
All topical contributions to this wiki (corrections, proposals for new features, new FAQ items, etc.) are welcome! Register using the link near the top-right corner of this page.
Heritrix is designed to respect the robots.txt exclusion directives† and META nofollow tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity.
If you notice our crawler behaving poorly – The Internet Archive uses archive.org_bot as User Agent when crawling – please send us email at [email protected].
(If you see a different User-Agent in your logs that still says 'heritrix', it may be someone else using this open-source software. In such a case, even if we can't directly change how your site is crawled, we are happy to help you interpret your logs and identify, contact, or block the source of any troublesome crawling.)
† The newer wildcard extension to robots.txt is not yet supported (feature request).
The most up to date release packages are the Heritrix 3.x dated releases. These are also available on Maven Central.
Heritrix is free software; you can redistribute it and/or modify it under the terms of the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0. Some individual source code files are subject to or offered under other licenses.
For more details see https://raw.github.com/internetarchive/heritrix3/master/dist/LICENSE.txt
Periodic dated releases are available from Github releases tab and Maven Central. These are currently maintained by the National Library of Australia and previously by the UK Web Archive.
A 3rd party build of Heritrix. This release uses Heritrix 3.3.0 master from May 2016. It was tested by several institutions and found to be very stable. It addresses many serious issues in 3.2.0. Blog post about this release.
Binary (-dist) and source (-src) distributions are available at: https://sbforge.org/nexus/index.html#view-repositories;thirdparty~browsestorage
Source fork can be found here: https://github.com/Landsbokasafn/heritrix3/tree/LBS-2016-02
Heritrix 3.2.0 final release is now available. See the following Release Notes page on the project wiki for full details: Release Notes - Heritrix 3.2.0
Binary (-dist) and source (-src) distributions are available at: http://builds.archive.org/maven2/org/archive/heritrix/heritrix/3.2.0/
Heritrix release 1.14.4 is now available at Sourceforge: https://sourceforge.net/projects/archive-crawler/files/
This is a 'micro' release with small bug fixes and improvements. Online release notes for 1.14.4 with a list of issues addressed are available at: Release Notes - 1.14.4
The 1.x-based Heritrix User Manual covers getting started with Heritrix and many advanced topics. This User Manual is generally focused on Heritrix 1.X versions, not fully updated for 1.12/1.14 or the larger changes in 2.0/3.0, but provides a reasonable basis for getting started with Heritrix, especially 1.14.4.
There is a FAQ from the 1.x era: http://archive-crawler.sourceforge.net/faq.html
The Knowledge Base contains topical articles documenting parts of the crawler and common problems or usage scenarios.
For developers, the 1.x-based Heritrix Developer Manual provides a guide to extending and customizing Heritrix code for your own purposes, though of course the source code itself, which is fairly well-commented, is the best guide.
For future documentation improvements, we have a [Documentation Wishlist](Documentation Wishlist).
An Introduction to Heritrix provides more detailed information on the structure and design of Heritrix.
Some very-old info can still be gleaned from the old wiki (http://web.archive.org/web/*/http://crawler.archive.org/cgi-bin/wiki.pl?HomePage).
As of October 2011, immediately following the 3.1.0 release, the Heritrix 3 source repository is hosted at github: https://github.com/internetarchive/heritrix3
Older source code can be found in sourceforge svn: http://sourceforge.net/scm/?type=svn&group_id=73833
Structured Guides:
User Guide
- Introduction
- New Features in 3.0 and 3.1
- Your First Crawl
- Checkpointing
- Main Console Page
- Profiles
- Heritrix Output
- Common Heritrix Use Cases
- Jobs
- Configuring Jobs and Profiles
- Processing Chains
- Credentials
- Creating Jobs and Profiles
- Outside the User Interface
- A Quick Guide to Creating a Profile
- Job Page
- Frontier
- Spring Framework
- Multiple Machine Crawling
- Heritrix3 on Mac OS X
- Heritrix3 on Windows
- Responsible Crawling
- Politeness parameters
- BeanShell Script For Downloading Video
- crawl manifest
- JVM Options
- Frontier queue budgets
- BeanShell User Notes
- Facebook and Twitter Scroll-down
- Deduping (Duplication Reduction)
- Force speculative embed URIs into single queue.
- Heritrix3 Useful Scripts
- How-To Feed URLs in bulk to a crawler
- MatchesListRegexDecideRule vs NotMatchesListRegexDecideRule
- WARC (Web ARChive)
- When taking a snapshot Heritrix renames crawl.log
- YouTube
- H3 Dev Notes for Crawl Operators
- Development Notes
- Spring Crawl Configuration
- Potential Cleanup-Refactorings
- Future Directions Brainstorming
- Documentation Wishlist
- Web Spam Detection for Heritrix
- Style Guide
- HOWTO Ship a Heritrix Release
- Heritrix in Eclipse