External link checking flaw and service

[hamishwillee]

Currently we have flaw checkers that can spot broken internal links. We have checkers that note http links that might possibly be https. What we don't have is anything to check whether external links actually work. That's a problem!

Flaw checker proposal

As we move to markdown, how about we add something like https://github.com/tcort/markdown-link-check to the flaw checker.
This is a good tool because you can feed it just the markdown pages you are interested in checking. Because it is parsing the markdown it can find the "intended links" in the page content, and not care about anything that might be elsewhere in sidebars.

The flaw checker should actually report only actual flaws, so I'd propose we only report things that give HTTP404 errors, and possibly those that give permanent redirects. We strip out all other errors since we don't know for sure they are actual errors.

Note Yes, this will miss a lot of potential problems - e.g. links where the page is being squatted etc. That's not the point - any flaw it catches should be valid.

CI based link checker service

On CI we use a service that can perhaps give more detailed information. For example there are services that can tell you if a domain is being resold - but it is a 404 for our purposes.

This report could be wrapped in a twistie so that it is hidden by default - with perhaps only the "real" problems being displayed.

FYI @peterbe
Yes, I know we have spoken of this before. I'm going to keep it alive until the problem is fixed :-)

[Ryuno-Ki]

I think, I side with @peterbe here when he introduced the flaw checker: the tool can't be made much cleverer than it is.
That is, if the content of the page changes, expires, gets riddled with advertisement etc. that's hard to figure out with tooling.
Instead, listing all external links and make a human take a look at it is the best thing to do for now.

What I could imagine, though, is to keep a file with meta information.
Like, this link was checked last by a human at date XYZ.
This could then prompt to a reevaluation for files that are older than a year (or so).

The HTTP status code might also be worth looking at (not only for 404, but also for redirects, for example).

[hamishwillee]

Hi @Ryuno-Ki

What @peterbe said is that it is hard to detect broken link that has been replaced by a squatter or "for sale" site. That's why the first part of the proposal is written as it is. If we're looking at 404s and redirects then that's a known bad page - here is no question they are legitimate flaws and a flaw check can be written to spot these.

Finding squatting pages is a tough challenge. There are services that can find these with some reliability. Peter suggested this :-)

I also quite like the idea of metadata for human checked links, but frankly that's an "as well" not an "instead".

[peterbe]

In general, I love the idea! This is going to significantly improve the quality of MDN because of SEO. Google hates it when you link to spam because they think you're encouraging that and it counts as a negative on our SEO. I.e. spam site "link farms".
It's much more than just SEO though. User hate clicking on links that don't work or are 200 OK but is just "junk". The junk can be nasty spammy stuff or it can be plainly not what the link set out to be. E.g. a perfectly normal and healthy site but due to old redirects or simply changed content, you might have thought it would link you to a JS demo but in fact, it took you to a company's product pricing page.

The other point I want to emphasize is that we should not tie this to Markdown.
Getting a list of URLs is absolutely trivial and it doesn't matter if the source is Markdown or if it's HTML. Actually, our Markdown processor basically converts the Markdown into HTML just like the legacy content and from there the HTML is post-processed and analyzed for things like flaws.

The tricky part is the flaws are meant to be "a slam dunk". It's supposed to be indisputable that something is a flaw. But many external links are not. They're almost entirely best judged by humans and not by scripts. One thing that might definitely be a flaw and automates well are:

1. Does the external link's HTTP error code >=400 and <500? And is that predictably so every time?
2. Is the external link's URL marked as unsafe in the same safe-browser list the Chrome and Firefox share when trying to open a link.
3. Does the use of HTTPS lead to an SSL problem. E.g. self-signed certs.

Currently, the PR review companion highlights external URLs but it's only a small start. I wish we could do more with that!
For example, a solid third-party service could help. Is there one where you say "Is this URL safe?". We can use a proprietary service that requires a private API key if that helps.
What about a service that gives you thumbnail screenshots from a URL? Perhaps we can post that as a PR comment instead of just listing URLs for the reviewer to have to open in a new tab.
What about, a basic sweep of all external URLs and simply highlight all the ones that either 3xx, 4xx, or 5xx? We can write a simple Python script that tries to open each URL and simply says which URLs to pay extra attention to because they might be bad. E.g. a 504 error might just mean temporary and starts to work after the review companion looked at it.

[hamishwillee]

@peterbe Thanks very much.

1. Very happy not to tie to markdown. As long as the check happens :-)
2. Flaw checker
   • For the flaw checker I am only proposing HTTP404 check. My assumption is that this is a slam dunk and hence that check can live in the flaw checker.
   • If there are other trustable status values then we should add them to the flaw checker too. My thinking was that the codes for a permanent redirect probably are reliable - e.g. 301. Completely up to you on that. Can't we trust a 404?
   • No, agree that we don't try report other cases as flaws because we don't know what they are (e.g. like >=400 and <500),
   • I'm aware we'll miss heaps of possible cases with just this check. But we'll catch a lot too. Incremental improvement is a good thing.
3. PR companion
   • We already know some people dislike what they consider too much information here. So whatever we provide should be optional to see (i.e. I think closed-by-default twisties are good)
   • I like the idea of highlighting the external links that are 3xx, 4xx, 5xx. Ideally with the actual code / string for the status. That's a simple first test. It makes it more likely I will actually check a link in PR companion.
   • I'm not so sure about the thumbnail. If I have the link and a hint that it might be broken from a service that is probably enough.
   • Love the idea of a service that can do more. I have no personal experience on any such service. Did find this link: https://www.makeuseof.com/tag/4-quick-sites-that-let-you-check-if-links-are-safe/

[peterbe]

Last but not least; I really like the PR review companion "framework". It's a nice place to write all sorts of scripts whose output becomes a (Markdown) GitHub issue comment.
BUT, it only works on PRs. It won't help a writer/reviewer who's opening the page on localhost:5000.
And obviously, it has the problem that it can only analyze pages you touch in a PR. It won't help you with existing content that slipped in from the old days of the Wiki before we had pre-publish review.

[peterbe]

I still think it would be relatively straight-forward to extend the review companion and do a bit more with the external links.
We need to be careful with HTTP GET and download because you might trip on a bomb that downloads too much data so the HTTP GET needs to streamed and capped.

But a basic sweep of each external URL and a list of its HTTP error codes I think would be a good start.

Now, if we do that in the review companion it won't solve for existing content. And it might cause confusion if it analyzes an external URL that was not introduced by the current PR author.

[Ryuno-Ki]

Most sites should be able to handle HTTP HEAD requests, which would save you bandwidth but still allow for checks of status codes …

[peterbe]

True, but I've found too many times that HEAD yields something very different from GET which might be a poorly configured web server or potentially malicious attempt to thwart an external site check.

[hamishwillee]

Hmmm

• no response here seems to answer why 404 in flaw checker is not a good idea. It is always a bug.
• Sweep on external URl and HTTP error codes would certainly be interesting. If we're concerned about confusing users then we just have to tell them what we're doing. That might simply be a note and a twistie hiding the you-do-not-absolutely-have-to-act-on-this information.