-
Notifications
You must be signed in to change notification settings - Fork 236
feat: Improve some confusing fallback reasons #2301
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2301 +/- ##
============================================
+ Coverage 56.12% 58.00% +1.87%
- Complexity 976 1294 +318
============================================
Files 119 146 +27
Lines 11743 13378 +1635
Branches 2251 2378 +127
============================================
+ Hits 6591 7760 +1169
- Misses 4012 4356 +344
- Partials 1140 1262 +122 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Thanks @wForget would you mind attach how reasons look before and after |
Thanks, I have edited description to add more test information. |
@@ -540,8 +540,13 @@ case class CometExecRule(session: SparkSession) extends Rule[SparkPlan] { | |||
// these cases specially here so we do not add a misleading 'info' message | |||
op | |||
case _ => | |||
// An operator that is not supported by Comet | |||
withInfo(op, s"${op.nodeName} is not supported") | |||
if (!hasExplainInfo(op)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Already has fallback reason, we don't need to overwrite it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @wForget this is LGTM
Which issue does this PR close?
Closes #2300.
Rationale for this change
We do not need to add fallback info for spark plan node when Comet is not enabled
Comet Scan is not enabled
fallback reason should be placed in the scan nodeditto
CometShuffleManager class name should be
org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
What changes are included in this PR?
Improve some confusing fallback reasons
How are these changes tested?
disable comet shuffle manager conf in
CometTestBase
and run the following test case:before this:
after this: