[MINOR] Corrected the reading of passed in properties when using DeltaStreamer #17467
+3
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…eamer. The code depends on the existence of DEFAULT_DFS_SOURCE_PROPERTIES file. Overridden properties are specified for HoodieStreamer using --props=<filepath>. The code uses the default value of --props=DEFAULT_DFS_SOURCE_PROPERTIES. Since not everyone would want to use and publish the default properties file, we can pass an empty --props value. This fix ensures that when --props is an empty string we simply use the properties supplied as the argument to the HUDIStreamer class.
prashantwason
changed the title
Collaborator
yihua
reviewed
Dec 3, 2025
Comment on lines
+191
to
+194
| } else if (!StringUtils.isNullOrEmpty(cfg.propsFilePath)) { | ||
| hoodieConfig.setAll(readConfig(hadoopConf, new Path(cfg.propsFilePath), cfg.configs).getProps()); | ||
| } else { | ||
| hoodieConfig.setAll(UtilHelpers.getConfig(cfg.configs).getProps()); |
Contributor
There was a problem hiding this comment.
Let's add test cases on these new branches.
Contributor
|
@prashantwason Could you follow the new PR title format: https://hudi.apache.org/contribute/how-to-contribute#pr-title-format |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
This PR fixes an issue where properties passed as arguments to DeltaStreamer are not properly handled when the
--propsparameter is empty or not specified.The OSS code has a default value for
--propspointing toDEFAULT_DFS_SOURCE_PROPERTIES. However, when--propsis intentionally set to an empty string (or left unspecified), the passed-in properties should still be used instead of being ignored.Summary and Changelog
Summary: Fixed DeltaStreamer to correctly use properties passed as arguments when
--propsis empty.Changelog:
HoodieStreamer.javato handle the case whencfg.propsFilePathis null or emptycfg.configsviaUtilHelpers.getConfig()Impact
This is a bug fix that affects users who:
configsparameter--propsfile path or set it to emptyThe fix ensures that properties are correctly applied in all scenarios, improving the reliability of property configuration in DeltaStreamer.
Risk Level
Low
This is a small, targeted fix that adds a missing else clause to handle an edge case. The change:
UtilHelpers.getConfig()method that is already used in the first branch of the conditionalDocumentation Update
None. This is a bug fix that restores expected behavior without introducing new features or configs.
Contributor's checklist