SNOWVATION: Replace parameter boilerplate with utility classes. #2751
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sfc-gh-jrose
added
the
NO-CHANGELOG-UPDATES
sfc-gh-jrose
force-pushed
the
jrose_snow_vation_config_refactor
branch
from
72c910b to
0f2d9d2
Compare
sfc-gh-jrose
requested review from
sfc-gh-yixie,
sfc-gh-aling and
sfc-gh-aalam
Comment on lines
-763
to
+820
| def conf(self) -> RuntimeConfig: | ||
| def conf(self) -> SettingStore: |
There was a problem hiding this comment.
we should probably keep the name same for migration sake: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.conf.RuntimeConfig.html
Comment on lines
-638
to
620
| ], | ||
| "parameter_name", ["auto_clean_up_temp_table_enabled", "cte_optimization_enabled"] | ||
| ) |
Comment on lines
+256
to
+259
| session.auto_clean_up_temp_table_enabled = True | ||
| assert session.auto_clean_up_temp_table_enabled is True | ||
| assert "auto_clean_up_temp_table_enabled is experimental" in caplog.text | ||
| session.auto_clean_up_temp_table_enabled = False |
There was a problem hiding this comment.
do we not raise warning only when we enable the param?
There was a problem hiding this comment.
The warning is only raised when setting to a non-default value.
| """ | ||
|
|
||
| name: str | ||
| description: str | None = field(default=None) |
There was a problem hiding this comment.
Suggested change
| description: str | None = field(default=None) | |
| description: Optional[str] = field(default=None) |
nit: we follow this style in the repo.
|
would it be easy to cover changes made in this PR: #2673 |
That wouldn't be too hard. I think I'd recommend downgrading from an error to a warning though so that users can be aware that they should clean things up. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-NNNNNNN
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
The goal of this refactor is to provide a central location for defining parameters and other snowpark-python configuration. This should make it easier to add additional parameters in the future as well as gate more deprecated or exerimental logic behind parameters.
Session parmeters are now all added in
Session._initialize_config. All definition needed for them should happen there.Package level confuguration settings should similarly all be defined in src/snowflake/snowpark/_internal/config.py
Parameters and package level configs can all be accessed via session.conf.
As part of this change I've moved all of the session level parameters over to the new settings style, but there are likely package level configuration options that I have missed. For now the parameters that are exposed as properties of the session object remain available in this manner, but in the future we should standardize the access method to be through the config object.