SNOW-1829870: Allow structured types to be enabled by default

[sfc-gh-jrose]

1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

   Fixes SNOW-1829870

2. Fill out the following pre-review checklist:

   • I am adding a new automated test(s) to verify correctness of my new code
     • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
   • I am adding new logging messages
   • I am adding a new telemetry message
   • I am adding new credentials
   • I am adding a new dependency
   • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
   • I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines

3. Please describe how your code solves the related issue.

   This PR addresses a few misc issue involved with enabling structured types by default.

• Change to_sql to support structured types. There were some edge cases where it was generating incorrect sql.
• When the structured type semantic mode is enabled structured types are assumed to be structured and to have semi-structured types you have to set the element types to None for ArrayType, MapType, and StructType.
• In cases where StringType was being passed to init for those types None is now passed by default allowing the type to decide if it should default to StringType.
• When the new mode is enabled types will prefer to be structured if element typing is available.

Before this change the in order to have a structured schema you would have to set the structured parameter to True on each type that was expected to be structured:

structured_schema = StructType([
    StructField("array", ArrayType(IntegerType(), structured=True)),
    StructField("map", MapType(IntegerType(), IntegerType(), structured=True)),
    StructField("struct", StructType([StructField("field", IntegerType())], structured=True)),
])

semi_structured_schema = StructType([
    StructField("array", ArrayType(IntegerType())),
    StructField("map", MapType(IntegerType(), IntegerType())),
    StructField("struct", StructType([StructField("field", IntegerType())])),
])

also_semi_structured_schema = StructType([
    StructField("array", ArrayType()),
    StructField("map", MapType()),
    StructField("struct", StructType()),
])

# Both of the semi-structured schemas actually materialize to this:
StructType([
	StructField('ARRAY', ArrayType(StringType()), nullable=True),
	StructField('MAP', MapType(StringType(), StringType()), nullable=True),
	StructField('STRUCT', MapType(StringType(), StringType()), nullable=True)
])

After this change the second case above will instead be inferred to be structured which should be more understandable to users. We've seen confusion in the past with the inner types being converted to StringType being non-inuitive.

After the change the definitions will look like this:

structured_schema = StructType([
    StructField("array", ArrayType(IntegerType())),
    StructField("map", MapType(IntegerType(), IntegerType())),
    StructField("struct", StructType([StructField("field", IntegerType())])),
])

semi_structured_schema = StructType([
    StructField("array", ArrayType()),
    StructField("map", MapType()),
    StructField("struct", StructType()),
])

# Semi structured actually materializes to what it is set to.

[github-actions]

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

[github-actions]

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

[sfc-gh-aalam]

can element_type be None here? What does it mean for the column type to be so?

[sfc-gh-jrose]

If element_type is None than this is a semi-structured array column and could contain anything.

[sfc-gh-aling]

is there an example to illustrate how the code change would work? I'm wondering changing StringType to None would cause any change in non-structured type cases

[sfc-gh-jrose]

See here:

snowpark-python/src/snowflake/snowpark/types.py

Lines 405 to 406 in 0dd7b91

	self.key_type = key_type if key_type else StringType()
	self.value_type = value_type if value_type else StringType()

The None here passes responsibility to the type initializer which still defaults to StringType when dealing with non-structured cases.

[github-actions]

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing