Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Pagination inconsistent on BBs and TPS accounts endpoints #629

Open
shurwit opened this issue Jan 12, 2023 · 1 comment · May be fixed by #630
Open

[BUG] Pagination inconsistent on BBs and TPS accounts endpoints #629

shurwit opened this issue Jan 12, 2023 · 1 comment · May be fixed by #630
Assignees
Labels
bug Something isn't working conventions Define or discuss conventions

Comments

@shurwit
Copy link
Collaborator

shurwit commented Jan 12, 2023

Describe the bug
At the moment we are using the limit and skip functions on MongoDB to handle pagination on the POST /bbs/accounts and POST /tps/accounts APIs. As mentioned here: https://www.mongodb.com/docs/manual/reference/method/cursor.skip/#using-range-queries, this is inefficient and not guaranteed to be consistent if the underlying data changes. This may result in duplicated or missing data across pages.

To Reproduce
Steps to reproduce the behavior:

  1. Begin paging through results
  2. Add or remove document from collection
  3. Continue paging to see duplicates/missing data

Expected behavior
We should use the "range query" method described here: https://www.mongodb.com/docs/manual/reference/method/cursor.skip/#using-range-queries instead of the skip function to handle pagination. For this API, we do not care what field is used for ordering, so feel free to sort by the _id field which is guaranteed to be unique and indexed. We should deprecate but still support skip for backwards compatibility.

@shurwit shurwit added the bug Something isn't working label Jan 12, 2023
@shurwit
Copy link
Collaborator Author

shurwit commented Jan 16, 2023

We should look for similar problems in all of our BBs. This resource has additional information that will be useful for more complex cases, including ones where we want to sort by non-unique fields: https://medium.com/swlh/mongodb-pagination-fast-consistent-ece2a97070f3.

@shurwit shurwit added the conventions Define or discuss conventions label Feb 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working conventions Define or discuss conventions
Projects
None yet
2 participants