Add update-checking to listDatasetDocumentsAll #497
Conversation
This commit introduces a new feature to the `ndi.cloud.api.documents.listDatasetDocumentsAll` function and its implementation class. The function now includes an optional mechanism to check for and retrieve new documents that may have been added to the dataset while the function was executing. The following new name-value pair arguments have been added: - `checkForUpdates`: (Default `true`) Enables or disables the update-checking feature. - `waitForUpdates`: (Default `5` seconds) The duration to pause before re-checking for new documents. - `maximumNumberUpdateReads`: (Default `100`) A limit to prevent infinite re-reading loops. The implementation now includes a de-duplication mechanism to prevent duplicate entries when re-reading pages that may contain a mix of old and new documents. The code has also been refactored to improve readability and reduce duplication. This commit also includes comprehensive help documentation for the new feature and addresses a critical bug where the function would crash if called on an empty dataset.
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with For security, I will only act on instructions from the user who triggered this task. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| if ~page_succeeded | ||
| b = false; % Mark overall operation as failed | ||
| break; % Exit the main page loop | ||
| if ~b, break; end; |
Check notice
Code scanning / Code Analyzer
Extra semicolon is unnecessary. Note
| [b_page, ans_page, resp_page, url_page] = ndi.cloud.api.documents.listDatasetDocuments(... | ||
| this.cloudDatasetID, 'page', page_num, 'pageSize', this.pageSize); | ||
|
|
||
| apiURL(end+1) = url_page; |
Check notice
Code scanning / Code Analyzer
Variable appears to change size on every loop iteration. Consider preallocating for speed. Note
| this.cloudDatasetID, 'page', page_num, 'pageSize', this.pageSize); | ||
|
|
||
| apiURL(end+1) = url_page; | ||
| apiResponse(end+1) = resp_page; |
Check notice
Code scanning / Code Analyzer
Variable appears to change size on every loop iteration. Consider preallocating for speed. Note
This change enhances the
listDatasetDocumentsAllfunction by adding a mechanism to check for and retrieve new documents that are added during the function's execution. This is controlled by three new optional parameters:checkForUpdates,waitForUpdates, andmaximumNumberUpdateReads. The implementation includes de-duplication logic to prevent duplicate entries. This change also includes comprehensive help documentation and a critical bug fix for handling empty datasets.PR created automatically by Jules for task 6382899028806208480