-
Notifications
You must be signed in to change notification settings - Fork 466
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DOC] Add document introducting StageLevel Resource Profile Adjust #8908
base: main
Are you sure you want to change the base?
Conversation
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format?
See also: |
docs/get-started/VeloxFeature.md
Outdated
--- | ||
|
||
### **Overview** | ||
Apache Gluten introduces a stage-level resource auto-adjustment framework to mitigate heap Out-of-Memory (OOM) issues caused by varying memory demands across stages in Spark applications. This feature dynamically adjusts task and executor resource profiles (e.g., heap/off-heap memory allocation) based on stage characteristics, such as the presence of fallback operators or heavy shuffle workloads(to be supported). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
by chatGPT:
"One major advantage of Apache Gluten is its ability to significantly reduce memory requirements per executor—potentially by up to half—when entire stages are offloaded to the native engine. This engine primarily relies on off-heap memory with minimal on-heap usage. However, when stages contain fallback operators that utilize the JVM engine, the on-heap memory size must be increased, leading to even higher memory demands per executor. This challenge has posed significant barriers during the adoption of Apache Gluten.
To address this issue, Apache Gluten introduces a stage-level resource auto-adjustment framework. This feature dynamically optimizes task and executor resource profiles, such as heap and off-heap memory allocation, based on the specific characteristics of each stage, including the presence of fallback operators. Additionally, this framework is designed with future enhancements in mind, allowing for adjustments to accommodate other requirements, such as heavy shuffle workloads."
Thank you, @zjuwangg |
@FelixYBW Thanks for your detailed review, just address the commnets. |
What changes were proposed in this pull request?
Add document about how to use StageLevelResource Auto adjust introduced in #8209
How was this patch tested?
NA