-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-54442][SQL] Add numeric conversion functions for TIME type #53147
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dongjoon-hyun
approved these changes
Nov 21, 2025
Member
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM. Thank you for proposing this for Apache Spark 4.2.0, @vinodkc .
Member
|
Merged to master for Apache Spark 4.2.0. |
zifeif2
pushed a commit
to zifeif2/spark
that referenced
this pull request
Nov 25, 2025
### What changes were proposed in this pull request? This PR adds six numeric conversion functions for the TIME type, mirroring the existing pattern for TIMESTAMP: Constructor Functions (Numeric → TIME): - time_from_seconds(seconds) - Supports fractional seconds via NumericType - time_from_millis(millis) - IntegralType input - time_from_micros(micros) - IntegralType input Extractor Functions (TIME → Numeric): - time_to_seconds(time) - Returns DECIMAL(14,6) to preserve fractional seconds - time_to_millis(time) - Returns BIGINT - time_to_micros(time) - Returns BIGINT eg ``` -- Constructor functions (Numeric → TIME) SELECT time_from_seconds(52200); -- 14:30:00 SELECT time_from_seconds(52200.123456); -- 14:30:00.123456 SELECT time_from_millis(52200123); -- 14:30:00.123 SELECT time_from_micros(52200123456); -- 14:30:00.123456 -- Extractor functions (TIME → Numeric) SELECT time_to_seconds(TIME'14:30:00.123456'); -- 52200.123456 SELECT time_to_millis(TIME'14:30:00.123'); -- 52200123 SELECT time_to_micros(TIME'14:30:00.123456'); -- 52200123456 ``` ### Why are the changes needed? The TIME type lacks numeric conversion functions, making it difficult to: - Create TIME values from numeric representations (common in data ingestion) - Extract numeric values for calculations or external system integration TIMESTAMP has equivalent functions (timestamp_seconds(), unix_seconds(), etc.), and TIME should achieve feature parity. ### Does this PR introduce _any_ user-facing change? Yes, adds six new SQL functions: ### How was this patch tested? Unit Tests (TimeExpressionsSuite.scala): SQL Integration Tests (time.sql) ### Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Claude 3.5 Sonnet AI assistance was used for: - Code pattern analysis and design discussions - Implementation guidance following Spark conventions - Test case generation and organization - Documentation and examples ### Additional context Q: Why does time_to_seconds() return DECIMAL(14,6) instead of BIGINT? A: To preserve fractional seconds and enable exact round-trip conversions: ``` SELECT time_to_seconds(time_from_seconds(52200.123456)); -- Returns: 52200.123456 (exact match) ✓ ``` If we returned BIGINT, fractional seconds would be lost. Q: Why does time_from_seconds() accept NumericType instead of just IntegralType? A: To support fractional seconds for maximum flexibility: ``` SELECT time_from_seconds(52200.5); -- DECIMAL: 14:30:00.5 SELECT time_from_seconds(52200.5::float); -- FLOAT: 14:30:00.5 ``` This mirrors timestamp_seconds() which also accepts NumericType. Closes apache#53147 from vinodkc/br_time_numeric_conversion. Authored-by: vinodkc <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
huangxiaopingRD
pushed a commit
to huangxiaopingRD/spark
that referenced
this pull request
Nov 25, 2025
### What changes were proposed in this pull request? This PR adds six numeric conversion functions for the TIME type, mirroring the existing pattern for TIMESTAMP: Constructor Functions (Numeric → TIME): - time_from_seconds(seconds) - Supports fractional seconds via NumericType - time_from_millis(millis) - IntegralType input - time_from_micros(micros) - IntegralType input Extractor Functions (TIME → Numeric): - time_to_seconds(time) - Returns DECIMAL(14,6) to preserve fractional seconds - time_to_millis(time) - Returns BIGINT - time_to_micros(time) - Returns BIGINT eg ``` -- Constructor functions (Numeric → TIME) SELECT time_from_seconds(52200); -- 14:30:00 SELECT time_from_seconds(52200.123456); -- 14:30:00.123456 SELECT time_from_millis(52200123); -- 14:30:00.123 SELECT time_from_micros(52200123456); -- 14:30:00.123456 -- Extractor functions (TIME → Numeric) SELECT time_to_seconds(TIME'14:30:00.123456'); -- 52200.123456 SELECT time_to_millis(TIME'14:30:00.123'); -- 52200123 SELECT time_to_micros(TIME'14:30:00.123456'); -- 52200123456 ``` ### Why are the changes needed? The TIME type lacks numeric conversion functions, making it difficult to: - Create TIME values from numeric representations (common in data ingestion) - Extract numeric values for calculations or external system integration TIMESTAMP has equivalent functions (timestamp_seconds(), unix_seconds(), etc.), and TIME should achieve feature parity. ### Does this PR introduce _any_ user-facing change? Yes, adds six new SQL functions: ### How was this patch tested? Unit Tests (TimeExpressionsSuite.scala): SQL Integration Tests (time.sql) ### Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Claude 3.5 Sonnet AI assistance was used for: - Code pattern analysis and design discussions - Implementation guidance following Spark conventions - Test case generation and organization - Documentation and examples ### Additional context Q: Why does time_to_seconds() return DECIMAL(14,6) instead of BIGINT? A: To preserve fractional seconds and enable exact round-trip conversions: ``` SELECT time_to_seconds(time_from_seconds(52200.123456)); -- Returns: 52200.123456 (exact match) ✓ ``` If we returned BIGINT, fractional seconds would be lost. Q: Why does time_from_seconds() accept NumericType instead of just IntegralType? A: To support fractional seconds for maximum flexibility: ``` SELECT time_from_seconds(52200.5); -- DECIMAL: 14:30:00.5 SELECT time_from_seconds(52200.5::float); -- FLOAT: 14:30:00.5 ``` This mirrors timestamp_seconds() which also accepts NumericType. Closes apache#53147 from vinodkc/br_time_numeric_conversion. Authored-by: vinodkc <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
cloud-fan
reviewed
Nov 28, 2025
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR adds six numeric conversion functions for the TIME type, mirroring the existing pattern for TIMESTAMP:
Constructor Functions (Numeric → TIME):
Extractor Functions (TIME → Numeric):
eg
Why are the changes needed?
The TIME type lacks numeric conversion functions, making it difficult to:
TIMESTAMP has equivalent functions (timestamp_seconds(), unix_seconds(), etc.), and TIME should achieve feature parity.
Does this PR introduce any user-facing change?
Yes, adds six new SQL functions:
How was this patch tested?
Unit Tests (TimeExpressionsSuite.scala):
SQL Integration Tests (time.sql)
Was this patch authored or co-authored using generative AI tooling?
Yes.
Generated-by: Claude 3.5 Sonnet
AI assistance was used for:
Additional context
Q: Why does time_to_seconds() return DECIMAL(14,6) instead of BIGINT?
A: To preserve fractional seconds and enable exact round-trip conversions:
If we returned BIGINT, fractional seconds would be lost.
Q: Why does time_from_seconds() accept NumericType instead of just IntegralType?
A: To support fractional seconds for maximum flexibility:
This mirrors timestamp_seconds() which also accepts NumericType.