[HUDI-8889] Trim unnecessary columns during MoR snapshot read

[TheR1sing3un]

Consider following case when we perform snapshot-read on MoR table:

1. full schema: (A, B, C, D)
2. primary key field: A
3. pre-combine key field: B
4. query with select D from table

for each file group

data file	log files	operation	payload allow projection pushdown	actual read schema
exist	empty	...	...	(_hoodie_record_key, B, D)
empty	exist	...	...	(_hoodie_record_key, B, D)
exist	exist	skip_merge	...	(_hoodie_record_key, B, D)
exist	exist	payload_combine	Y	(_hoodie_record_key, B, D)
exist	exist	payload_combine	N	(_hoodie_record_key, A, B, D)

However, except for the last two case, we only need to read column D on the file in other cases.

Change Logs

1. Trim unnecessary columns during MoR snapshot read

Impact

Improves performance when MoR snapshot-read with not actually merged

Risk level (write none, low medium or high below)

low

Documentation Update

none

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[TheR1sing3un]

@hudi-bot run azure

[hudi-bot]

CI report:

• 39f927a UNKNOWN
• 4842151 UNKNOWN
• 2ad28a1 Azure: FAILURE Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[danny0405]

Not sure how the fields got set up and whether they are required or not.

[TheR1sing3un]

Not sure how the fields got set up and whether they are required or not.

mandatory is for fields that are required to be read regardless of any read behavior, such as _hoodie_commit_time in incremental-read, and optional is for fields that are required to be merged during snapshot reads, such as _hoodie_record_key and precombine-key

[danny0405]

So why name the fields optional?