Node.js script that crawls a Sitemap and checks for broken links

[timcosgrove]

Requirements

We need a script that checks broken links on VA.gov, so that we can verify that our content works correctly for Veterans.

Acceptance criteria

Preview Give feedback

1. Given a domain as an argument, the script finds and processes the Sitemap for that domain, including any child sitemaps
2. Each URL in the Sitemap is checked for broken links
3. At the end of processing, a structured report is generated showing each page with broken links and the broken destinations

Background & implementation details

There are any number of broken link checking packages in the Node ecosystem: https://www.npmjs.com/search?q=broken%20link

Linkinator is a likely candidate. Note however that Linkinator does not currently crawl via Sitemap: JustinBeckwith/linkinator#346

A separate Sitemap reading package would likely be necessary to find URLs to pass to Linkinator for checking.

Parallelizing the checks is ideal, though if it adds significant complication it should not block completion of the ticket.