Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating data sync handlers based on mem leak updates #13

Merged
merged 2 commits into from
Nov 12, 2024

Conversation

rpmcginty
Copy link
Collaborator

@rpmcginty rpmcginty commented Nov 9, 2024

What's in this Change?

Primarily meant to address the following bug - https://alleninstitute.atlassian.net/browse/OCSDV-354

  • caching of botocore config
  • introducing DataSyncResult and BatchDataSyncResult
    • summarizes data that is transferred
    • allows to conditionally update
  • for batch jobs, we add the ability for partial success of transfers (by default this is disabled)

Primarily meant to address the following bug - https://alleninstitute.atlassian.net/browse/OCSDV-354

This also introduces a new approach to gathering results during data sync. Previously the data sync operations returned nothing and the lambda handlers would simply return the requests in the response. For a few reasons, this is not great and so I have introduced a more informative and cohesive result object that summarizes data sync operations based on bytes transferred, files transferred and for the batch case, the number of succeeded and failed tasks.

Connected PRs

Testing

  • unit tests
  • e2e tests in merscope analysis dev

pyproject.toml Outdated Show resolved Hide resolved
@rpmcginty rpmcginty marked this pull request as ready for review November 9, 2024 00:53
@rpmcginty rpmcginty requested a review from njmei November 12, 2024 01:13
@rpmcginty rpmcginty force-pushed the bugfix/data-sync-memory-leak branch from 2a0e5be to b9aa7dd Compare November 12, 2024 20:45
@rpmcginty rpmcginty merged commit d155d57 into main Nov 12, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants