Automatic URL fetching in xpath Documents (with guardrials and feature gated)

[phayes]

Hello again!

I'm wrapping up my work on xee-extract, and turning my attention to my next usecase need - automatic URL fetching in Documents.

My use case is I have a large set (about a million) of XML documents in a datastore that are accessible via http.

I would like to run XML extractions against this entire datastore. Any given extraction might only touch a few documents, but I dont necessarily know which documents an extraction will touch ahead of time, so I can't pre-load them.

What I'd like to do is open a PR where I add a feature-gated feature to xee_xpath::Documents. The idea would be someting like this:

let policy = HttpPolicy::new()
    .whitelist("https://store.myxmlstore.com/*")
    .header("https://*.myxmlstore.com/*", "User-Agent", "my-agent/1.0")
    .header("https://*.myxmlstore.com/*", "Authorization", "my-secret-auth-code")
    .relative_base("https://store.myxmlstore.com")
    .connect_timeout(Duration::from_secs(5))
    .total_timeout(Duration::from_secs(15))
    .max_response_size(10 * 1024 * 1024)
    .max_retries(3)
    .retry_on_status(vec![429, 503])

let mut documents = Documents::new();
documents.enable_doc_fetch(policy);

 // Add our default document, which I've pre-loaded
let user_doc = documents
    .add_string(
        "http://example.com/user/1.xml".try_into().unwrap(),
        r#"<users>
            <user id="user123">
                <name>John Doe</name>
                <email>john@example.com</email>
                <organization xlink:type="simple" xlink:href="/orgs/acme.xml" />
            </user>
        </users>"#,
    )
    .unwrap();

let extractor = Extractor::new();
let user: UserWithOrg = extractor.extract_from_docs(&mut documents, &user_doc).unwrap();

This would obviously introduce a bunch of dependencies, so it would need to be feature-gated (default disabled).

Thoughts?

[faassen]

This would be an awesome feature to have!

I would imagine the main dependency would be reqwest, right? Sounds like a good idea to feature gate that, even though I've not really done anything with feature-gating in the past. But here we're really opening up a whole new set of interactions with the world so this may be wise. Besides feature gating you need an explicit way to turn this on, which I see you have with enable_doc_fetch.

It would be good to think about other aspects of XPath that want to access non-XML resources too; we want to configure that the same way I imagine.

The xee CLI tool should probably come with this configured to on for the whole web, as that supports experiments and the risks are pretty minor, though there should be a CLI option to turn this off.

I'm very curious whether Documents as I wrote it is up the job to support this, or whether it needs significant rewriting. At some point I introduced quite sophisticated URI (IRI) construction to Xee using the iri_string crate. I do recommend trying to use this crate for URL resolution, as though there's a learning curve to that crate, it does help ensure we get this right.

[bojidar-bg]

Note: having support for fetching documents on the fly would also be helpful for the XLST test suite, where it would enable us to just change a single XPath query to support .out files, rather than special-casing the code around it.

[gronke]

I'm working toward running xee's XSLT support in the browser via WebAssembly, where fn:doc/document() would have to resolve URIs without a filesystem (e.g. from an in-memory map), ideally lazily.

It appears that three document-loading variants could benefit from a shared design:

• HTTP fetching behind a policy, as proposed here
• in-memory/lazy resolution (my WASM case)
• filesystem loading, as the xee CLI does today by pre-loading

Instead of three parallel mechanisms, could these share a single resolution hook on Documents, with the HTTP policy as one implementation? And could that boundary be chosen so it stays compatible with the DocumentsRef sharing direction in #138 (e.g. a resolver stored in the collection would need to be Send + Sync if the Arc<RwLock<…>> route lands)?

I don't have the full picture of the internals, so treat this as a nudge rather than a design, but if a shared hook sounds right, I'd be glad to prototype the in-memory/wasm side against it and help move both threads forward.