refactor: Update conversion from StructType to Avro Schema to include docs and default values

[the-other-tim-brown]

Describe the issue this Pull Request addresses

Currently there are two methods that developers must call to ensure that a StructType is properly converted to an Avro. This is error prone and can be simplified by unifying the code.

Summary and Changelog

• Set doc field when first constructing the Avro from the StructType
• Set default null value when field is nullable
• Set doc string in StructType for round trip conversion

Impact

Simplifies the code and eliminates risk of developers forgetting to call method to add doc strings or null values to the schema

Risk Level

None

Documentation Update

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[yihua]

LGTM

[yihua]

In Spark 3.5 + Scala 2.13, the test failed because of mismatched ArrayBuffer and Array, likely due to difference in array type between Scala 2.12 and 2.13.

- Test Create External Hoodie Table *** FAILED ***
  Expected ArrayBuffer(StructField(_hoodie_commit_time,StringType,true), StructField(_hoodie_commit_seqno,StringType,true), StructField(_hoodie_record_key,StringType,true), StructField(_hoodie_partition_path,StringType,true), StructField(_hoodie_file_name,StringType,true), StructField(id,IntegerType,true), StructField(name,StringType,true), StructField(price,DoubleType,true), StructField(ts,LongType,true), StructField(dt,StringType,true)), but got Array(StructField(_hoodie_commit_time,StringType,true), StructField(_hoodie_commit_seqno,StringType,true), StructField(_hoodie_record_key,StringType,true), StructField(_hoodie_partition_path,StringType,true), StructField(_hoodie_file_name,StringType,true), StructField(id,IntegerType,true), StructField(name,StringType,true), StructField(price,DoubleType,true), StructField(ts,LongType,true), StructField(dt,StringType,true)) (TestCreateTable.scala:232)

[hudi-bot]

CI report:

• 298f3ae Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build