You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Update RESTClient's PageNumberPaginator to optionally stop paging after receiving empty pages.
I'm unsure if other paginators need this same functionality. Perhaps it needs to be added to the base RangePaginator class.
Are you a dlt user?
Yes, I'm already a dlt user.
Use case
I am pulling data from a few REST APIs that do not include the information necessary to determine the total number of records or pages. Setting the max number of pages isn't sufficient because the REST helper will keep sending requests up to the max page limit, even if everything past the first page is empty.
Proposed solution
Add a boolean keyword argument to the PageNumberPaginator (or its base class RangePaginator) called stop_after_empty that is False by default.
When it is true, consider the paging complete when the first empty page is received.
Related issues
No response
The text was updated successfully, but these errors were encountered:
Thank you for your feature request @timoguin!
In the specification, you wrote that the default for this new flag should be False. I wonder whether defaulting to True would be more practical.
Have you encountered a case where a page delivers empty results but the next page is not empty?
@burnash@willi-mueller I only proposed setting it to False to retain the current default behavior, but I agree that it makes sense to stop after empty pages by default.
Feature description
Update RESTClient's PageNumberPaginator to optionally stop paging after receiving empty pages.
I'm unsure if other paginators need this same functionality. Perhaps it needs to be added to the base
RangePaginator
class.Are you a dlt user?
Yes, I'm already a dlt user.
Use case
I am pulling data from a few REST APIs that do not include the information necessary to determine the total number of records or pages. Setting the max number of pages isn't sufficient because the REST helper will keep sending requests up to the max page limit, even if everything past the first page is empty.
Proposed solution
Add a boolean keyword argument to the
PageNumberPaginator
(or its base classRangePaginator
) calledstop_after_empty
that isFalse
by default.When it is true, consider the paging complete when the first empty page is received.
Related issues
No response
The text was updated successfully, but these errors were encountered: