add support for s3 references to batch data sync requests #10
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rpmcginty
force-pushed
the
feature/update-batch-data-sync
branch
from
ea65c7c to
49bcafb
Compare
aamster
approved these changes
Jun 27, 2024
| for _ in request.requests: | ||
| if isinstance(request.requests, S3URI): | ||
| self.logger.info(f"Request is stored at {request.requests}... fetching content.") | ||
| _ = download_to_json(request.requests) |
There was a problem hiding this comment.
not a huge fan of using _ and __ instead of variable name, but I can still understand what the code is doing. _ is generally used when we don't use that variable at all, not when you can't think of a name for it
There was a problem hiding this comment.
yeah you are right, I can change it.
There was a problem hiding this comment.
I will clean up in a follow up change
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's in this Change?
Why is this needed? there is a rather small max payload allowed in step functions (256 KB). This is problematic for our lambda and batch lambda functions that are plugged into the system. This change allows us to write part of the larger parts of the lambda responses and requests to s3 and load it.
The way it works is that if you specify
intermediate_s3_pathin the prepare batch data sync request, the function will write the batch data sync requests to s3 objects under that path specified.Testing