Skip to content

[spark] Close the commit and release the cached RDD in remove_unexisting_files - #9706

Open
cxzl25 wants to merge 1 commit into
apache:masterfrom
cxzl25:spark-remove-unexisting-files-close-commit
Open

[spark] Close the commit and release the cached RDD in remove_unexisting_files#9706
cxzl25 wants to merge 1 commit into
apache:masterfrom
cxzl25:spark-remove-unexisting-files-close-commit

Conversation

@cxzl25

@cxzl25 cxzl25 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Purpose

SparkRemoveUnexistingFiles creates a TableCommit on the executor inside foreachPartition but never closes it. TableCommitImpl.close() shuts down the underlying commit and the maintainExecutor thread pool, so every invocation of the procedure leaks a thread pool on a long-lived executor. Other call sites such as PaimonPartitionManagement already wrap the commit in try/finally.

The intermediate RDD is also cached and never unpersisted. buildRDD() returned a child of the cached RDD to the caller, so the cache could only be released after the caller finished, which never happened. This change turns it into execute(), which collects the result inside the method and unpersists the cache in a finally block.

Behaviour of the procedure is unchanged; only resource lifecycle is affected.

Tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant