Hybrid Mode: ask scrapeghost to write selectors

[jamesturk]

See FAQ: https://jamesturk.github.io/scrapeghost/faq/#why-not-ask-the-scraper-to-write-css-xpath-selectors

There's an alternate version of the long-page scraper that could generate extraction selectors and then apply them client-side. Would be a huge cost savings for simple list pages. I'm exploring ideas related to this and will start posting updates on it soon.

[jamesturk]

Revisiting this after a few failed attempts that didn't go anywhere.

Generating the XPath/etc. just isn't nearly as robust as going straight to the data, so many variations of this introduce a whole host of problems.

In the current request to GPT, the full HTML (mod cleaning) is sent as well as a JSON schema. This is usually enough to get the data itself out of the HTML without examples because the field names are good enough (first name, last name, etc. have semantic meaning that GPT can use to make the extraction call.

For list pages, it's common to wind up with something like this:

for link in tree.xpath(DIV_OR_TR_XPATH):
    item = {
          "field": FIELD_SUB_XPATH,
          "field2": FIELD2_SUB_XPATH,
    }

To make this work on list pages, we need to send a representative sample of the page (the entire page is often too long)
and then ask GPT for DIV_OR_TR_XPATH, FIELD_XPATH, FIELD2_XPATH, etc.

We don't want each field's XPath to include the parent (e.g. we don't want //tr[3]/td[@id=xyz], we just want ./td[@id=xyz].

The prompt needs to be quite different.

Ideas:

• template code like above and ask it to fill in the gaps
• require some data to be sent with the request (eg. first and third item) and use those to help it understand what to grab
• first pass: request "container" (how to word this?) / second pass: given just container HTML what XPath would be given

Potential Uses:

• huge page where only first chunk fits within window (but need to be careful with cleaned HTML here since it'll modify xpaths)
• pagination, where xpath can be determined by first page, and then repeated over and over
• the same could be true of templated pages, but they run into the issue where not all data is xpath-able at least with list pages the general result is a list of URLs