Skip to content

[SPARK-54669][SQL] Remove redundant casting in rCTEs #53428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Pajaraja wants to merge 3 commits into from
Closed

[SPARK-54669][SQL] Remove redundant casting in rCTEs #53428

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3dcd7ef
[SQL][MINOR] Remove redundant casting in rCTEs
Pajaraja Dec 10, 2025
ad20f69
revert testing scope change/add time sensitive test
Pajaraja Dec 10, 2025
bef9c8a
change golden file
Pajaraja Dec 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionBase.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,11 @@ abstract class TypeCoercionBase extends TypeCoercionHelper {
case (attr, dt) =>
val widerType = findWiderTypeForTwo(attr.dataType, dt)
if (widerType.isDefined && widerType.get == dt) {
Alias(Cast(attr, dt), attr.name)()
if (attr.dataType != dt) {
Alias(Cast(attr, dt), attr.name)()
} else {
attr
}
} else {
throw cannotMergeIncompatibleDataTypesError(dt, attr.dataType)
}
Expand Down
12 changes: 11 additions & 1 deletion sql/core/src/test/resources/sql-tests/analyzer-results/cte-recursion.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1907,7 +1907,7 @@ WithCTE
: +- UnionLoop xxxx
: :- Project [1 AS 1#x, cast(1 as bigint) AS CAST(1 AS BIGINT)#xL]
: : +- OneRowRelation
: +- Project [cast((n + 1)#x as int) AS (n + 1)#x, cast((n + 1)#x as bigint) AS (n + 1)#xL]
: +- Project [(n + 1)#x, cast((n + 1)#x as bigint) AS (n + 1)#xL]
: +- Project [(n#x + 1) AS (n + 1)#x, (n#x + 1) AS (n + 1)#x]
: +- Filter (n#x < 5)
: +- SubqueryAlias t1
Expand Down Expand Up @@ -1936,6 +1936,16 @@ org.apache.spark.SparkException
}


-- !query
WITH RECURSIVE t1(n, stamp) AS (
SELECT 1, CAST(DATE '2024-01-15' AS TIMESTAMP)
UNION ALL
SELECT n+1, stamp FROM t1 WHERE n < 5)
SELECT * FROM t1
-- !query analysis
[Analyzer test output redacted due to nondeterminism]
Copy link
Contributor

@peter-toth peter-toth Dec 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this non-deterministic? If it is really non-deterministic, can you add a deterministic test?

Copy link
Contributor

@cloud-fan cloud-fan Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question

Copy link
Contributor Author

@Pajaraja Pajaraja Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was nondeterministic because the original relation was Date. I changed into string now.
It's deterministic now.



-- !query
WITH RECURSIVE t1(n) AS (
SELECT 1
Expand Down
7 changes: 7 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/cte-recursion.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -728,6 +728,13 @@ WITH RECURSIVE t1(n, m) AS (
SELECT n+1, CAST(n+1 AS BIGINT) FROM t1 WHERE n < 5)
SELECT * FROM t1;

-- Type coercion with timeZone sensitive type
WITH RECURSIVE t1(n, stamp) AS (
SELECT 1, CAST(DATE '2024-01-15' AS TIMESTAMP)
UNION ALL
SELECT n+1, stamp FROM t1 WHERE n < 5)
SELECT * FROM t1;

-- Recursive CTE with nullable recursion and non-recursive anchor
WITH RECURSIVE t1(n) AS (
SELECT 1
Expand Down
16 changes: 16 additions & 0 deletions sql/core/src/test/resources/sql-tests/results/cte-recursion.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1997,6 +1997,22 @@ org.apache.spark.SparkException
}


-- !query
WITH RECURSIVE t1(n, stamp) AS (
SELECT 1, CAST(DATE '2024-01-15' AS TIMESTAMP)
UNION ALL
SELECT n+1, stamp FROM t1 WHERE n < 5)
SELECT * FROM t1
-- !query schema
struct<n:int,stamp:timestamp>
-- !query output
1 2024-01-15 00:00:00
2 2024-01-15 00:00:00
3 2024-01-15 00:00:00
4 2024-01-15 00:00:00
5 2024-01-15 00:00:00


-- !query
WITH RECURSIVE t1(n) AS (
SELECT 1
Expand Down