Skip to content

Conversation

@google-labs-jules
Copy link
Contributor

This change introduces a new feature to the ndi.cloud.upload.uploadDocumentCollection function. A new optional argument, onlyUploadMissing (defaulting to true), has been added. When active, this feature prevents the re-upload of documents that already exist in the remote dataset.

The function now performs the following steps:

  1. Checks if the onlyUploadMissing option is enabled.
  2. If so, it calls ndi.cloud.api.documents.listDatasetDocumentsAll to retrieve a list of all document summaries from the remote dataset.
  3. It then compares the local document list with the remote list and filters out any documents that already exist.
  4. The upload process then continues only with the documents that are not present on the remote.
  5. If all documents are filtered out (meaning no new documents to upload), the function returns early with a success status.

This enhancement improves the efficiency of the document upload process by avoiding redundant data transfer.


PR created automatically by Jules for task 16961829823943544003

Modified `ndi.cloud.upload.uploadDocumentCollection` to include a new
optional name-value argument, `onlyUploadMissing`, which defaults to `true`.

When this option is enabled, the function will first query the remote
dataset for a list of existing documents and filter the local
`documentList` to exclude any documents that are already present. This
prevents unnecessary re-uploads and improves efficiency.
@google-labs-jules
Copy link
Contributor Author

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!


For security, I will only act on instructions from the user who triggered this task.

@stevevanhooser stevevanhooser marked this pull request as ready for review October 31, 2025 14:19
@stevevanhooser stevevanhooser self-requested a review October 31, 2025 14:19
Copy link
Contributor

@stevevanhooser stevevanhooser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, thanks

@stevevanhooser stevevanhooser merged commit 7b7b235 into main Oct 31, 2025
1 of 2 checks passed
@stevevanhooser stevevanhooser deleted the feature-upload-missing-docs branch October 31, 2025 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants