Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RESTClient: stops pagination after empty page (Feat/1637) #1677

Merged
merged 9 commits into from
Aug 19, 2024
Merged

RESTClient: stops pagination after empty page (Feat/1637) #1677

merged 9 commits into from
Aug 19, 2024

Conversation

willi-mueller
Copy link
Collaborator

@willi-mueller willi-mueller commented Aug 9, 2024 •
edited
Loading

Description

  • Adds a new option to RangePaginator and its subclasses OffsetPaginator and PageNumberPaginator to stop pagination upon receiving a page without data items
  • By default, stops pagination after an empty page
  • Every RangePaginator now requires either total_path or maximum_offset or stop_after_empty_page
  • If total_path is not provided, pagination is controlled by maximum_page and stop_after_empty_page or maximum_offset and stop_after_empty_page respectively.

Related Issues

@netlify Netlify
Copy link

netlify bot commented Aug 9, 2024 •
edited
Loading

✅ Deploy Preview for dlt-hub-docs canceled.

Name Link
🔨 Latest commit 83bab15
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/66c339cf80854200087f47ef

@willi-mueller willi-mueller changed the title Feat/1637 stop pagination after empty page rest_client: stops pagination after empty page (Feat/1637) Aug 9, 2024
@willi-mueller willi-mueller requested a review from burnash August 9, 2024 10:38
@willi-mueller willi-mueller mentioned this pull request Aug 9, 2024
Closed
@willi-mueller willi-mueller self-assigned this Aug 9, 2024
@willi-mueller willi-mueller changed the title rest_client: stops pagination after empty page (Feat/1637) RESTClient: stops pagination after empty page (Feat/1637) Aug 9, 2024
@burnash burnash added the enhancement New feature or request label Aug 9, 2024
@burnash burnash mentioned this pull request Aug 9, 2024
Closed
@burnash burnash linked an issue Aug 9, 2024 that may be closed by this pull request
Additional paginator for offset without a "total_param" #1530
Closed
burnash
burnash previously approved these changes Aug 15, 2024
View reviewed changes
Copy link
Collaborator

@burnash burnash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, @willi-mueller. I've left a couple of comments

dlt/sources/helpers/rest_client/paginators.py Outdated Show resolved Hide resolved
dlt/sources/helpers/rest_client/paginators.py Outdated Show resolved Hide resolved
burnash
burnash approved these changes Aug 19, 2024
View reviewed changes
Copy link
Collaborator

@burnash burnash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work, thank you @willi-mueller

@willi-mueller willi-mueller merged commit d448122 into devel Aug 19, 2024
56 checks passed
@willi-mueller willi-mueller deleted the feat/1637_stop-pagination-after-empty-page branch August 19, 2024 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@burnash burnash burnash approved these changes

Assignees

@willi-mueller willi-mueller

Labels
enhancement New feature or request
Projects
dlt core library
Status: Done
Milestone
No milestone
2 participants
@willi-mueller @burnash