Support dual HTTP/HTTPS for a mirror

[elboulangero]

This is a feature request, plus a PR that implements it: #156

Background

With the term dual HTTP/HTTPS, I mean: make it possible for a mirror to support both HTTP and HTTPS.

At the moment: a mirror has only one field HttpUrl. It can be either a HTTP or a HTTPS URL. Mirrorbits handles that as such:

• 1. if the incoming request is HTTPS, only HTTPS mirrors are eligible for redirection
• 2. if the incoming request is HTTP, both HTTP and HTTPS mirrors are eligible for redirection

This behavior makes complete sense when distributing files via the web browser.

Now coming from a different background: for a Debian-based Linux distro, it's important to distribute packages over both HTTP and HTTPS. And it's important to respect the protocol of the incoming request: a HTTP request should NOT be redirected to HTTPS.

A bit more background

Just for comparison: mirrorbrain (another redirector, maybe we could say mirrorbits precursor) doesn't handle that either, a mirror can only have one URL (either HTTP or HTTPS). The workaround is to have two instances running: one for the HTTP mirrors, and another for the HTTPS mirrors. It has downsides though:

• each instance must scan the mirrors, so we basically duplicate the workload:
  • on the mirrors side, to do the scan, and
  • on the redirector side, to process the result of the scan
• it takes twice more RAM, since the database is duplicated

With mirrorbits, we can follow the same workaround, with the same downsides. However the point 2 "it takes twice more RAM" becomes a dealbreaker. For Kali Linux, we have a quite large repo (between 500k to 1 million files), and around 50 mirrors, so the database is quite big. Compared to mirrorbrain, mirrorbits needs around 6 times more RAM to operate... And that's why we can't afford to run two instances in parallel anymore, like we used to do with mirrorbrain.

The feature request

So we'd like that mirrorbits can support "Dual HTTP/HTTPS" for mirrors. The most obvious way to get there, IMO, is to add a new field HttpsUrl. Then adjust mirrorbits code all over the place so that it can support various combinations of HttpUrl (set/unset) and HttpsUrl (set/unset).

In practice, see the PR #156. It's not so tricky, but it requires modifying mirrorbits in various places, and it implies a database upgrade, leading to a rather larger PR.

I must add that this code is running on a staging instance at http://http-staging.kali.org/README?mirrorlist, and will be in production pretty soon, if all goes well.

Any feedback welcome, thanks for reading!