Dynamic Scraper maintain session across requests

[tedjt]

It seems like the dynamic scraper works by using the 'request' library to load content and then loads it in phantom with the setContent command.

https://github.com/ariya/phantomjs/wiki/API-Reference-WebPage#webpage-setContent

It seems to do navigation based scraping that depends on sessions we would have to manually parse out the appropriate data from the response object and pass it along in subsequent requests.

It would be nice to have a persistent session mode where the dynamic scraper would work at the level of a phantom WebPage instance (like a browser tab) and load/navigate across pages using js actions (like clicks). This is important for scraping ajax based pages. Tools like CasperJs work well for this, and the use cases for the Dynamic scraper seem a little limited without it.

[ruipgil]

Scraping is extracting information from the page, and scraperjs does it very good and very easily that's why there's no navigation system.
The purpose of the dynamic scraper is to get content that is loaded dynamically, like an angular app, or even to access js variables.
However there is support for things like sessions using request.

[vdraceil]

+1 to the issue/suggestion.
It will be really great if we can have a persistent mode as CasperJS does.

Also, as a first time user, I didn't know that the DynamicScraper doesn't support sessions until I tried and failed. May be, it would be useful to people if we put that explicitly on the README.

[chmac]

Is this now covered by the scraper factory functionality? Maybe this issue could be closed if so...

[ruipgil]

This is not covered by scraperjs in any way.