[SPARK-54651][CORE] Driver should delete temporary files when exiting during commit process #53406
Conversation
… during commit process
|
Gentle. ping @cloud-fan @HyukjinKwon @dongjoon-hyun Could you take a look? |
| committer.setupJob(jobContext) | ||
| try { | ||
| val fs = stagingDir.getFileSystem(jobContext.getConfiguration) | ||
| fs.deleteOnExit(stagingDir) |
There was a problem hiding this comment.
For the streaming Spark jobs, this increases indefinitely, doesnt' it, @AngersZhuuuu ?
IMO, we had better consider this kind of behavior change as Improvement instead of Bug to avoid any side-effect. WDYT?
cc @mridulm
There was a problem hiding this comment.
I don't think streaming job need to look at file source staging dir when restarting, this is not state store or checkpoint, cc @HeartSaVioR
There was a problem hiding this comment.
For streaming jobs, given their really long duration and large number of commits, we will keep accumulating the deleteOnExit metadata to be tracked - and eventually OOM.
I am assuming that is what @dongjoon-hyun is referring to - and I agree it is a risk.
At a minimum, we can derisk it by calling cancelDeleteOnExit for successful commits.
Btw, please do not catch Throwable
4 participants
What changes were proposed in this pull request?
During spark commit file process, if spark exit,
may remain temporary staging dir under table path, This is very troubling.
After
setupJob, we should markstagingDiras deleteOnExit()Why are the changes needed?
Fix bug
Does this PR introduce any user-facing change?
Clean remaining staging dir automatic when unexpected exit
How was this patch tested?
No
Was this patch authored or co-authored using generative AI tooling?
No