feat(schema): phase 5 - Perform Java Client Core Migration

[voonhous]

Describe the issue this Pull Request addresses

This Pull Request addresses the ongoing effort to refactor the Hudi codebase to use a unified HoodieSchema type system for schema handling, migrating away from direct usage of Avro's Schema in client-side code where possible.

This specifically completes Phase 5: Java Client Core Migration as described in #14270.

NOTE: All changes made here avoids FileGroup* classes.

Summary and Changelog

This PR migrates key Hudi Java client components to use HoodieSchema for internal schema handling, enhancing maintainability without altering the Avro on-disk format.

1. Core Migration: DeleteContext, RecordContext and HoodieReaderContext and their engine-specific implementations are updated to use HoodieSchema (instead of Avro Schema) for internal operations like getValue, toBinaryRow, and getOrderingValue.
2. Utilities: HoodieSchemaUtils gains new client-side utilities, including getFieldPosition, projectSchema, and isReadCompatible.
3. Integrates the HoodieSchema changes from the previous Column Statistics Migration phase into core utility classes (FileFormatUtils, HoodieTableMetadataUtil, etc.).

This ensures records and metadata are consistently processed using the HoodieSchema abstraction in memory.

Impact

Low

Risk Level

Low, risk is low due to refactoring maintaining Avro serialization compatibility via HoodieSchema.toAvroSchema().

Documentation Update

none

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[voonhous]

Still working on this, mergeBootstrapReaders has not been migrated yet.

[voonhous]

Okay, should be ready for review now.

[the-other-tim-brown]

I've reviewed some of the files but the scope of this PR is much larger than the ticket calls for. Is that intentional or is this combined with another branch?

[the-other-tim-brown]

Instead of the createUnion we can use the HoodieSchema.createNullable

[voonhous]

Swapped them out, will search for other references that i modified too.

[voonhous]

Just want to call this out: Avro has a strict validation rule, where the default value for a union field must match the type of the FIRST element in the union type array.

• ["null", "int"] with default null (Valid)
• ["null", "int"] with default 0 (Invalid)
• ["int", "null"] with default 0 (Valid)
• ["int", "null"] with default null (Invalid)

If we use HoodieSchema.createNullable for int fields with default 0, the following error will be thrown for the affected tests:

Caused by: org.apache.avro.AvroTypeException: Invalid default for field age: 0 not a ["null","int"]
	at org.apache.avro.Schema.validateDefault(Schema.java:1635)
	at org.apache.avro.Schema.access$500(Schema.java:94)
	at org.apache.avro.Schema$Field.<init>(Schema.java:561)
	at org.apache.avro.Schema$Field.<init>(Schema.java:607)
	at org.apache.hudi.avro.HoodieAvroUtils.createNewSchemaField(HoodieAvroUtils.java:381)
	at org.apache.hudi.common.schema.HoodieSchemaField.of(HoodieSchemaField.java:113)
	at org.apache.hudi.common.schema.HoodieSchemaField.of(HoodieSchemaField.java:92)
	at org.apache.hudi.functional.TestBufferedRecordMerger.getSchema1(TestBufferedRecordMerger.java:815)

[the-other-tim-brown]

Do we still want to seal in these cases?

[voonhous]

This is in the original code. AM just adding an additional check to ensure that schema is not null.

org.apache.hudi.common.engine.RecordContext#getSchemaFromBufferRecord calls
org.apache.hudi.common.engine.RecordContext#decodeAvroSchema, which is a nullable method.

[voonhous]

I've reviewed some of the files but the scope of this PR is much larger than the ticket calls for. Is that intentional or is this combined with another branch?

Nope, not combined with another branch. It's large because of how many other classes are using them. It cascades very quickly in a way where:

Let's change the variable to a HoodieSchema type -> Change parameter of method signature to HoodieSchema type.

The changes mainly affect the *Handles and the signature of *Handles changes, the same thing cascades.

I am trying not to touch the *Iterators and HoodieFileGroup* classes.

Most of the changes are in the hudi-client/hudi-client-common modules, and that's where everything starts cascading quickly.

[hudi-bot]

CI report:

• 9366433 UNKNOWN
• c35119a UNKNOWN
• d5de6fe UNKNOWN
• 8b743f7 UNKNOWN
• 8e63408 UNKNOWN
• 0be8a7e UNKNOWN
• 0e062be UNKNOWN
• 93ff6a2 UNKNOWN
• 3775f38 UNKNOWN
• 3f2a7b6 UNKNOWN
• d4a06dc UNKNOWN
• 64bc734 UNKNOWN
• 8ff89c2 UNKNOWN
• fb1da7d UNKNOWN
• bdb9eac Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build