Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add descriptions without modifying model and protocol digest #111

Closed
wants to merge 25 commits into from

Conversation

zmezei
Copy link
Contributor

@zmezei zmezei commented Jul 3, 2023

No description provided.

@zmezei zmezei requested a review from qati July 3, 2023 22:08
@zmezei zmezei changed the title feat: add descriptions without modifying model digest feat: add descriptions without modifying model and protocol digest Jul 6, 2023
orig_descriptions[field_name] = {}
Model._remove_descriptions(field.type_, orig_descriptions[field_name])

@staticmethod
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my point of view these methods shouldn't be static most of the times the model is even passed in why not use self then?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The model param passed to these newly created methods is actually not an instance of a Model subclass but rather a Model subclass type (so a class object) that's why I couldn't implement these methods as instance methods (so using self.). However you are right in the sense that these method could be class methods using cls.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the refactored version I created class methods instead of static methods where possible, please check it.

Copy link
Contributor

@lrahmani lrahmani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this feature needs to be rethought as it can produce different digests for the same model. Also removing model fields may not be the best approach to skip them in digest computation. One can simply delete the fields from the manifest dict.

@@ -176,7 +176,7 @@ def manifest(self) -> Dict[str, Any]:

for schema_digest, model in all_models.items():
manifest["models"].append(
{"digest": schema_digest, "schema": model.schema()}
{"digest": schema_digest, "schema": model.schema_no_descriptions}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it guaranteed that this value won't be None?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the refactored version I call the schema_no_descriptions(), that method also uses a class attribute as a cache and that class attribute could possibly be None, but in the schema_no_descriptions() method I guarantee to set the value of that class variable now if that is None.

@@ -199,6 +199,12 @@ def manifest(self) -> Dict[str, Any]:
encoded = json.dumps(manifest, indent=None, sort_keys=True).encode("utf8")
metadata["digest"] = f"proto:{hashlib.sha256(encoded).digest().hex()}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

digest is also computed by the compute_digest static method.
It takes as input the manifest which will contain the descriptions hence producing a different digest.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I modified this method accordingly and also created test for it in test_fields_descr.py, please check it.

@zmezei zmezei requested review from lrahmani and dominic22 July 6, 2023 23:42
@jrriehl
Copy link
Contributor

jrriehl commented Aug 31, 2023

Closing this because we will soon be revising the way the model digests are done anyway.

@jrriehl jrriehl closed this Aug 31, 2023
@Archento Archento deleted the feature/upload-field-desc branch April 12, 2024 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants