Fixed hackathon issues #8891
Open
Fixed hackathon issues #8891
+53
−14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Unable to set Settings due to no interface that allows it - No ability for Mongo to do ignore_duplicates and updates - No passing of kwargs between set methods in the Settings service - Fixed issue with Jobs not working with data > 16MB in mongo because data was fetched from blob storage and then re-saved
shubham3121
reviewed
Jun 7, 2024
| @@ -291,14 +291,25 @@ def _set( | |||
| f"The fields that should be unique are {keys}." | |||
| ) | |||
| else: | |||
| # TODO: h4x fix ?????? how is this supposed to work? This returns if | |||
There was a problem hiding this comment.
Yeah we were not throwing error because ignore duplicates is set to True, but we're not overwriting the data. +1 to override the data in case ignore duplicates is True during .set.
shubham3121
reviewed
Jun 7, 2024
| context=self.auth_context, action_object=new_action_object | ||
| ) | ||
|
|
||
| # TODO: do this elsewhere, because here it breaks mongo by setting data back |
There was a problem hiding this comment.
+1 Ideally, we shoudn't have nested action objects, but even in the case if were to unwrap nested action object, and save it again to the Database, we should do the following:
data = action_object.syft_action_data
new_data = self.unwrap_nested_actionobjects(data)
new_action_object = ActionObject.from_obj(new_data, id=action_object.id)
# this will save the actual data to blob storage and only save the metadata in mongo store.
new_action_object._save_to_blob_storage()
res = self.action_service.set(
context=self.auth_context, action_object=new_action_object
)
There was a problem hiding this comment.
the idea was that you can do something like this
@syft_function
def map():
...
@syft_function
def reduce():
...
@syft_function
def outer():
map_results=[]
for x in [1,2,3]:
map_job = domain.launch_job(map, x)
map_results.append(map.job.results) # this does not exist yet, but we do have a pointer
reduce_job = domain.launch_job(reduce, map_results) # this is a list of ActionObjects that do not exist yet
return reduce_job.result
There was a problem hiding this comment.
agreed on how it should be stored @shubham3121. I think this was (partially?) created before we had blob storage so it was probably not changed when we introduced blob storage
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Please include a summary of the change, the motivation, and any additional context that will help others understand your PR. If it closes one or more open issues, please tag them as described here.
Affected Dependencies
List any dependencies that are required for this change.
How has this been tested?
Checklist