Treat manifest hops same as navlink hops

[kris-sigur]

Links from manifests (e.g. sitemaps) should not receive the preferential treatment sometimes accorded to "transitive" hops. Most commonly this is about giving priority to discovered (probable) embeds.

Manifests should be regarded as more analogous with a directory page.

[kris-sigur]

I suppose, alternatively (or additionally), the logic in FrontierPreparer#getSchedulingDirective could be amended to schedule M links ahead of L, but after E X R etc.

This seams a slightly more invasive change as it would require either adding a level or moving current "HIGH" to "HIGHEST" and current "MEDIUM" to "HIGH" to free up "MEDIUM" priority for manifests. Given that "HIGHEST" priority doesn't seem to be used anywhere, maybe this isn't much of an issue. The alternative would be to add a priority level constant, if we want to leave "HIGHEST" available.

If this approach is taken, Inferred links should maybe also be treated at this 'middle' level.

[ato]

I actually wonder if sitemap links should even be de-prioritised compared to the normal link structure because they effectively flatten the structure of the site. If a big site happens to put all their leaf pages at the beginning, which is very common because they're often the most recently updated articles that they want search engines to pick up, the crawl might miss important intermediate section pages that in a non-sitemap crawl would have priority because of being less deep.

[ato]

But yes, this is at least an improvement on the current situation.