Implications of merging Advanced Crawling Add-on to core?

[leonstafford]

@john-shaffer should make life easier.

Scanning through code, looks like it will replace the Sitemap parsing in core? That would be nice. HTML parser looks elegant, too.

This all makes more sense to you as you wrote it, do you feel keen to do the merging? I'll go work on something else for a bit to avoid conflicts.

[john-shaffer]

It would make sense to merge in the parts you mentioned. But the add-on as a whole is unstable and needs more refinement and testing. There are a lot of performance trade-offs, as an optimization for one site tends to slow down other sites that don't benefit from the optimization. It's still very experimental.

[leonstafford]

Can those trade-offs be made into configurable options within Core? Think it's about time to bring in anything from Static HTML Output, too, with same concerns.

[leonstafford]

Else, go the other direction and split this Advanced Crawling Add-on into more specific drop-in replacements for Crawling, Detection, etc.

[leonstafford]

I'm about ready to MergeAllTheThings back together, including add-ons. There hasn't been a lot of interest in ppl building 3rd party add-ons, so it's just extra overhead for me at the moment. As long as not blocking the extensible functions in core, impact for users should be net beneficial.

Regarding the core functions of Detect, Crawl, Post Process, Deploy, what are the variables we need to allow configuration for?

I'll start thinking:

Detect

• homepage only
• all the currently filterable detection options in ./src/URLDetection.php

Crawling

• discover (Detect) new URLs as we find them, parsing 0..n of HTML, CSS, XML
• parse CSS

Post processing

• ?

Deploy

• use DeployCache or not
• configure options specific to deployment provider

What other variables do you think we need to make configurable?

[john-shaffer]

The config values in advanced crawling are:

Detect

• Filenames to Ignore
• File Extensions to Ignore

Crawl

• Add URLs Discovered While Crawling
• Crawl Chunk Size
• Crawl Only Changed URLs
• Crawl Progress Report Interval
• Crawl Sitemaps
• Detect Redirects from the Redirection Plugin
• Additional Paths to Crawl

Post Process

• Additional Hosts to Rewrite

[john-shaffer]

Some of the functionality in Advanced Crawling just isn't ready to get locked in. The main thing is that we're adding a crawled_time column to the URLs table which causes extra writes, since crawl_cache also has that column. The problem is that there's no cheap way to cross-reference the urls table with the crawl cache. The most efficient solution that I can think of is to combine the two tables, and I think we should give that serious consideration since it would eliminate a lot of DB load. Failing that, I'd still like to test other approaches than the current monkey-patching.