How To Create A Ripper for HTML websites

This guide explains how to rip from an unsupported website using RipMe.

If you like to learn by example, check out the simple ImgboxRipper.java.

Expectations

• Some knowledge of the Java programming language
• Some knowledge of the build tool Maven
• This is for dependency resolution & so you don't have to download a ton of .jar files
• Some knowledge of the HTML DOM, CSS selectors, and the like.

Step 0: Fork this repo

Step 1: Create a new .java file

Create the file within / src / main / java / com / rarchives / ripme / ripper / rippers

File should follow the naming scheme <Site>Ripper.java

Step 2: Extend the AbstractHTMLRipper class

public class YoursiteRipper extends AbstractHTMLRipper {

Step 3: Understand the fields available

By extending AbstractHTMLRipper, you have access to the this.url object containing the URL to be ripped.

Step 4: Constructors

We need to let the superclass know what URL we're working with.

Change the constructor's class name to your ripper's class name.

    public YoursiteRipper(URL url) throws IOException {
        super(url);
    }

Step 5: Override the required methods

The methods below are defined in AbstractHTMLRipper and must be overridden in your .java file.

---

String getHost()

Returns: The name of the website (no need for .com).
This String is used in naming the save directory.

    @Override
    public String getHost() {
        return "imgur";
    }

---

String getDomain()

Returns: The domain of the website.
This String is used in the canRip() method to determine if a URL can be ripped.

    @Override
    public String getDomain() {
        return "imgur.com";
    }

---

String getGID(URL)

Returns: A unique identifier for the album (Gallery ID or GID).

Note: The URL to every album on the website should return a different GID.
This is because the save directory will be named in the scheme HOST_GID

Most rippers use regex to strip out the GID.

Example: imgur.com/a/abc123 could return abc123

    @Override
    public String getGID(URL url) throws MalformedURLException {
        Pattern p = Pattern.compile("^https?://imgur\\.com/a/([a-zA-Z0-9]+).*$");
        Matcher m = p.matcher(url.toExternalForm());
        if (m.matches()) {
            // Return the text contained between () in the regex
            return m.group(1);
        }
        throw new MalformedURLException("Expected imgur.com URL format: " +
                        "imgur.com/a/albumid - got " + url + " instead");
    }

---

Document getFirstPage()

Returns: A Jsoup Document object containing the contents of the first page.

Tip: Use the Http class for easy methods of retrieving the page.

Most rippers just need to get the page, and do so with:

    @Override
    public Document getFirstPage() throws IOException {
        // "url" is an instance field of the superclass
        return Http.url(url).get();
    }

This works for the majority of websites (most sites don't require cookies, referrers, etc).

---

Document getNextPage(Document) // Optional!

Input: Jsoup Document retrieved in the getFirstPage() method.
Returns: The next page to retrieve images from.
Throws: IOException if no next page can be retrieved.

Note: By default, this method throws an IOException within AbstractHTMLRipper, meaning it assumes there is no next page. If you need to rip multiple pages, override this method & retrieve the next page. See ImagebamRipper.java for an example of how this is used.

---

List getURLsFromPage(Document)

Input: Jsoup Document retrieved in the getFirstPage() method (and optionally the getNextPage() method).
Returns: List of URLs to be downloaded or retrieved.

This is where the URLs are extracted from the page Document.
Some rippers return a list of subpages to be ripped in separate threads (e.g. ImagevenueRipper.java)

This is when CSS-Selectors come in handy. Say you wanted to grab every image that appears on the page:

    @Override
    public List<String> getURLsFromPage(Document doc) {
        List<String> result = new ArrayList<String>();
        for (Element el : doc.select("img")) {
            result.add(el.attr("src"));
        }
        return result
    }

This would return the source to all images on the page (although they will likely be thumbnails).

The URLs returned are passed into the next method...

---

void downloadURL(URL url, int index)

Input: URL: One of the URLs returned by getURLsFromPage()
Input: index: The number for this URL (whether it's the 1st image, 2nd image, etc).

This is where your ripper downloads the image/file.
Most rippers simply use the AlbumRipper's method addURLToDownload():

    @Override
    public void downloadURL(URL url, int index) {
        addURLToDownload(url, getPrefix(index));
    }

The above will download the URL to the appropriate save directory, guessing the filename to save based on the url and a given prefix (index).

The addURLToDownload() method is heavily overloaded with lots of options.
Variants of this method allow you to:

• Define the exact file name to save as,
• The subdirectory to save to,
• HTTP headers (such as cookies or referrers) that should be used while downloading the file

Other rippers, as mentioned before, start a separate Thread to retrieve the full-size image from the provided URL. Your implementation may vary.

Step 6: Test!

RipMe automatically detects new rippers without any other code changes required.

1. Execute the ripper.
2. Paste in a URL to the site you're trying to rip.
3. Click Rip
4. Look at the output for errors, warnings, Exceptions, etc.
5. Fix any bugs.
6. Repeat.