Skip to content

Commit

Permalink
[SPARK-38526][SQL] Fix misleading function alias name for RuntimeRepl…
Browse files Browse the repository at this point in the history
…aceable

### What changes were proposed in this pull request?

This PR uses a manual recursion to replace `RuntimeReplaceable` expressions instead of `transformAllExpressionsWithPruning`. The problem of `transformAllExpressionsWithPruning` is it will automatically make the replacement expression inherit  the function alias name from the parent node, which is quite misleading. For example, `select date_part('month', c) from t`, the optimized plan in EXPLAIN before this PR is
```
Project [date_part(cast(c#18 as date)) AS date_part(month, c)#19]
+- Relation default.t[c#18] parquet
```
Now it's
```
Project [month(cast(c#9 as date)) AS date_part(month, c)#10]
+- Relation default.t[c#9] parquet
```

### Why are the changes needed?

fix misleading EXPLAIN result

### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

new test

Closes apache#35821 from cloud-fan/follow2.

Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
cloud-fan authored and dongjoon-hyun committed Mar 12, 2022
1 parent 60334d7 commit c91c2e9
Show file tree
Hide file tree
Showing 2 changed files with 15 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,14 @@ import org.apache.spark.util.Utils
* we use this to replace Every and Any with Min and Max respectively.
*/
object ReplaceExpressions extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan.transformAllExpressionsWithPruning(
def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning(
_.containsAnyPattern(RUNTIME_REPLACEABLE)) {
case e: RuntimeReplaceable => e.replacement
case p => p.mapExpressions(replace)
}

private def replace(e: Expression): Expression = e match {
case r: RuntimeReplaceable => replace(r.replacement)
case _ => e.mapChildren(replace)
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ class ExplainSuite extends ExplainSuiteHelper with DisableAdaptiveExecutionSuite
keywords = "InMemoryRelation", "StorageLevel(disk, memory, deserialized, 1 replicas)")
}

test("optimized plan should show the rewritten aggregate expression") {
test("optimized plan should show the rewritten expression") {
withTempView("test_agg") {
sql(
"""
Expand All @@ -125,6 +125,13 @@ class ExplainSuite extends ExplainSuiteHelper with DisableAdaptiveExecutionSuite
"Aggregate [k#x], [k#x, every(v#x) AS every(v)#x, some(v#x) AS some(v)#x, " +
"any(v#x) AS any(v)#x]")
}

withTable("t") {
sql("CREATE TABLE t(col TIMESTAMP) USING parquet")
val df = sql("SELECT date_part('month', col) FROM t")
checkKeywordsExistsInExplain(df,
"Project [month(cast(col#x as date)) AS date_part(month, col)#x]")
}
}

test("explain inline tables cross-joins") {
Expand Down

0 comments on commit c91c2e9

Please sign in to comment.