Scraping dynamic content

[JackyZhang1219]

Is your feature request related to a problem? Please describe.

Is it possible to get all the dynamically loaded content from a url? E.g. getting all the reviews from IMDB requires clicking "load more" over and over again, and Scrapegraph-ai currently only gets the reviews loaded first

[VinciGit00]

not yet

[PeriniM]

Hey @JackyZhang1219 thanks we'll look into It! if you have some suggestions write them in the Discord channel

[hogking]

This is a great feature request. or example, get a 'next page' button to jump to the next list page, like scrapy-slash?

[angelotc]

This is a scraper, not a crawler. The way I see it is you can integrate this with scrapy.

[VinciGit00]

hi please look at the deep scraper graph, is not the solution but it could be a temporal solution