[Improvement]: Eliminate AMS Full GC impactions deriving from local terminal clean spark context

[nicochen]

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

We experienced and suffered from AMS full GC for a long period. we dig into this case and finally found out local spark session terminal triggers java.lang.System.gc() every 30 minutes to clean contexts. As arthas stack traces as shown below:

How should we improve?

For external kyuubi session terminal, there is no need to revises anything. The problem is only related to local session which is embedding into AMS itself. Simply add config 'spark.cleaner.referenceTracking' = 'false' to solve the problem.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Subtasks

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[nicochen]

@czy006 Due to my working network issue, the same issues seem to been wrongly submitted twice. Sry about that. This one is duplicated with #2969 and has already been merged. The fix logic is same. I will close this issue ad duplicated.

[nicochen]

Duplicated with #2969. Closed

[czy006]

@nicochen thanks for your callback