Paginated Imports

[jonmifsud]

As of XML Importer v 3 (currently integration) one can pass url parameters to the importer itself which can be used for things such as pagination. I think it would be nice if there's an option to determine the pagination variable (if any) along with an xpath which determines if there's a next page or not.

If the condition matches it should show on the front end a link to import the next page.

example xpath:

/data/datasource/pagination/@current-page !=  /data/datasource/pagination/@total-pages

The user can then use the judgment to determine if they import the next page or not. Would be nice to give the option to automate the process through some Javascript but not necessary I presume. Willing to implement myself.

@brendo any objection?

[nilshoerrmann]

Just a thought, not sure if it's related: Couldn't the pagination option be used for throttling which would be useful for large imports?

[jonmifsud]

Yes that's more or less the idea. But I'm not sure how throttling would
work as such. I am thinking it could be something enabled through some
Javascript maybe, to avoid causing server side timeouts,
or excessively long loading times. This way you can still see step by step
the page which the import is on via the backend. Also if there's an error
it would give an error on a smaller subset of entries.

I have over 100k entries to import in a particular section and looking to
automate in some way or another, but in some cases if you're debugging it
might be useful to move page by page as well.

On 8 January 2015 at 14:39, Nils Hörrmann [email protected] wrote:

Just a thought, not sure if it's related: Couldn't the pagination option
be used for throttling which would be useful for large imports?

—
Reply to this email directly or view it on GitHub
#39 (comment)
.

[brendo]

Might be worthwhile seeing what the Import CSV or Resave Entries does to paginate through entries.

[jonmifsud]

Resave Entries does it all through Javascript hitting the same page via Ajax. When the resave entry triggers, it returns back a json with the status rather than the full page.

I think as long as we return the page, and the error messages to show to the user the approach seems fine.