Add `DATE_TRUNC` Optimizer #14385

ashishjayamohan · 2024-11-05T05:59:16Z

Closes: Time Pruner should support queries where we're operating on the time column itself #14362
Addresses: Short circuit queries where no rows will be returned #13563
Adds optimizer for dateTrunc function

codecov-commenter · 2024-11-05T07:02:58Z

Codecov Report

Attention: Patch coverage is 0% with 98 lines in your changes missing coverage. Please review.

Project coverage is 33.99%. Comparing base (59551e4) to head (f38e322).
Report is 1625 commits behind head on master.

Files with missing lines	Patch %	Lines
...optimizer/filter/TimePredicateFilterOptimizer.java	0.00%	97 Missing and 1 partial ⚠️

❗ There is a different number of reports uploaded between BASE (59551e4) and HEAD (f38e322). Click for more details.

HEAD has 19 uploads less than BASE

Flag BASE (59551e4) HEAD (f38e322)

integration 7 5

integration2 3 2

temurin 12 8

java-21 7 5

skip-bytebuffers-true 3 2

skip-bytebuffers-false 7 6

unittests 5 3

unittests1 2 0

java-11 5 3

custom-integration1 2 0

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #14385       +/-   ##
=============================================
- Coverage     61.75%   33.99%   -27.76%     
- Complexity      207      675      +468     
=============================================
  Files          2436     2708      +272     
  Lines        133233   151642    +18409     
  Branches      20636    23413     +2777     
=============================================
- Hits          82274    51555    -30719     
- Misses        44911    95927    +51016     
+ Partials       6048     4160     -1888

Flag	Coverage Δ
custom-integration1	`?`
integration	`100.00% <ø> (+99.99%)`	⬆️
integration1	`100.00% <ø> (+99.99%)`	⬆️
integration2	`0.00% <ø> (ø)`
java-11	`33.98% <0.00%> (-27.73%)`	⬇️
java-21	`33.99% <0.00%> (-27.64%)`	⬇️
skip-bytebuffers-false	`33.99% <0.00%> (-27.76%)`	⬇️
skip-bytebuffers-true	`33.98% <0.00%> (+6.25%)`	⬆️
temurin	`33.99% <0.00%> (-27.76%)`	⬇️
unittests	`33.99% <0.00%> (-27.76%)`	⬇️
unittests1	`?`
unittests2	`33.99% <0.00%> (+6.26%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ashishjayamohan · 2024-11-06T01:02:12Z

@jadami10 Made that optimizer enhancement here. Let me know if anything looks off!

...test/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizerTest.java

...src/main/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizer.java

...test/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizerTest.java

...src/main/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizer.java

jadami10

first, thank you for doing this!

took an initial look and left some nit comments, some testing ideas, and some ideas about stuff that looks like it might break.

I haven't reviewed the actual algorithm yet

...src/main/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizer.java

...test/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizerTest.java

jadami10 · 2024-11-07T02:58:56Z

...test/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizerTest.java

+
+  @Test
+  public void testDateTruncOptimizer() {
+    testDateTrunc("datetrunc('DAY', col) < 1620777600000", new Range("0", true, "1620777600000", false));


do we need a test with an INT instead of long? I believe that's also a valid time type in pinot

can we have test with week or month truncation

can we have a test case where it's not a support function. as in IN to make sure nothing is erroring

also a test case where the time granularity is unsupported? I'm not sure if calcite will catch that before the optimizer, but we do use DAY as an input

Hey @jadami10. Thanks for this insight! I've been working on this for the past couple days (specifically on the time zone test). I've introduced several time zone tests and my implementation seems to work only for some time zone usages.
If you're willing, would you be able to write up a quick draft of what the algorithm should look like to convert the date_trunc function with time zones to a range query (essentially, the floor and ceiling inverse of date trunc). I think it would be beneficial to hear it from another perspective to find what I'm missing.

jadami10

i think your approach to convert the date trunc predicates to floor/ceiling sounds good. I left some comments where I'm confused about why the implementation drifts for similar functions

...src/main/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizer.java

jadami10 · 2024-11-12T03:38:53Z

...src/main/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizer.java

+        upperMillis = dateTruncFloor(operands);
+        if (upperMillis != TimeUnit.MILLISECONDS.convert(getLongValue(filterOperands.get(1)), TimeUnit.valueOf(outputTimeUnit.toUpperCase()))) {
+          upperInclusive = true;
+          upperMillis = dateTruncCeil(operands);


why is this recomputed here?

jadami10 · 2024-11-12T03:40:41Z

...src/main/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizer.java

+        lowerMillis = Long.MIN_VALUE;
+        upperInclusive = false;
+        upperMillis = dateTruncFloor(operands);
+        if (upperMillis != TimeUnit.MILLISECONDS.convert(getLongValue(filterOperands.get(1)), TimeUnit.valueOf(outputTimeUnit.toUpperCase()))) {


why are we checking this here but not in in GREATER_THAN?

jadami10 · 2024-11-12T03:42:55Z

...src/main/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizer.java

+        break;
+      case GREATER_THAN_OR_EQUAL:
+        operands.set(1, getExpression(getLongValue(filterOperands.get(1)), new DateTimeFormatSpec("TIMESTAMP")));
+        lowerMillis = dateTruncFloor(operands);


should this be ceil?

jadami10 · 2024-11-12T03:45:06Z

...test/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizerTest.java

+//    testDateTrunc("datetrunc('DAY', col, 'DAYS', 'CET', 'MILLISECONDS') = 39193714800000",
+//        new Range("453631", true, "453631", true));
+    testDateTrunc("datetrunc('DAY', col, 'MILLISECONDS', 'UTC', 'DAYS') = 453630",
+        new Range("39193632000000", true, "39193718399999", true));


this looks strange. 39193632000000 is like 1000 years from now?

ashishjayamohan · 2025-01-25T02:52:51Z

@jadami10 Following up on this, sorry for it taking a while! I think there are some implementation-level nuances that I'm running into. Specifically, this optimizer works 100% of the time without timezones but struggles when moving into a different time zone, outputted in a lower granularity unit than the bucketing unit. If you're free and willing, I'd love to schedule a quick call to help walk through some cases since this is my first time working on a feature like this. Let me know if that's a possibility!

ashishjayamohan and others added 5 commits November 4, 2024 14:38

add floor and ceil func

e3efa65

Merge branch 'apache:master' into optimize-date-trunc

731b943

add date trunc optimization

749bbbb

fix style

c2dbe08

fix range query column expr

8dda15e

ashishjayamohan added 8 commits November 5, 2024 11:20

add support for between

2edad2a

refactor test functions

52490a3

refactor back and add test

cb387da

add tests for range pred

0b41d25

style

9391b48

fix algo and tests

36e7639

fix between operator

bc08f03

add tests

b09fc49

ashishjayamohan changed the title ~~[draft] Optimize DATETRUNC in Predicate~~ Add DATE_TRUNC Optimizer Nov 6, 2024

ashishjayamohan marked this pull request as ready for review November 6, 2024 00:59

ashishjayamohan commented Nov 6, 2024

View reviewed changes

...test/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizerTest.java Outdated Show resolved Hide resolved

ashishjayamohan commented Nov 6, 2024

View reviewed changes

...src/main/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizer.java Outdated Show resolved Hide resolved

ashishjayamohan added 4 commits November 5, 2024 22:01

use correct output format

eec0e7f

add helper function

614e176

make unit step less brittle

624e360

correct conversion to output unit

5e87ed7

Jackie-Jiang added the enhancement label Nov 6, 2024

Jackie-Jiang reviewed Nov 6, 2024

View reviewed changes

...test/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizerTest.java Show resolved Hide resolved

Jackie-Jiang reviewed Nov 6, 2024

View reviewed changes

...src/main/java/org/apache/pinot/core/query/optimizer/filter/TimePredicateFilterOptimizer.java Show resolved Hide resolved

add javadocs

4b836c7

jadami10 reviewed Nov 7, 2024

View reviewed changes

ashishjayamohan added 2 commits November 7, 2024 12:46

fix short circuit

470d4bb

fix range settings

321cc6a

everything but time zones

e3a6651

ashishjayamohan marked this pull request as draft November 12, 2024 00:43

just logging for future

2e6bb78

jadami10 reviewed Nov 12, 2024

View reviewed changes

update, still not working

8b58e9e

Merge branch 'apache:master' into optimize-date-trunc

f38e322

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `DATE_TRUNC` Optimizer #14385

Add `DATE_TRUNC` Optimizer #14385

ashishjayamohan commented Nov 5, 2024 •

edited

Loading

codecov-commenter commented Nov 5, 2024 •

edited

Loading

ashishjayamohan commented Nov 6, 2024

jadami10 left a comment •

edited

Loading

jadami10 Nov 7, 2024

ashishjayamohan Nov 11, 2024

jadami10 left a comment

jadami10 Nov 12, 2024

jadami10 Nov 12, 2024

jadami10 Nov 12, 2024

jadami10 Nov 12, 2024

ashishjayamohan commented Jan 25, 2025

Add DATE_TRUNC Optimizer #14385

Are you sure you want to change the base?

Add DATE_TRUNC Optimizer #14385

Conversation

ashishjayamohan commented Nov 5, 2024 • edited Loading

codecov-commenter commented Nov 5, 2024 • edited Loading

Codecov Report

ashishjayamohan commented Nov 6, 2024

jadami10 left a comment • edited Loading

Choose a reason for hiding this comment

jadami10 Nov 7, 2024

Choose a reason for hiding this comment

ashishjayamohan Nov 11, 2024

Choose a reason for hiding this comment

jadami10 left a comment

Choose a reason for hiding this comment

jadami10 Nov 12, 2024

Choose a reason for hiding this comment

jadami10 Nov 12, 2024

Choose a reason for hiding this comment

jadami10 Nov 12, 2024

Choose a reason for hiding this comment

jadami10 Nov 12, 2024

Choose a reason for hiding this comment

ashishjayamohan commented Jan 25, 2025

Add `DATE_TRUNC` Optimizer #14385

Add `DATE_TRUNC` Optimizer #14385

ashishjayamohan commented Nov 5, 2024 •

edited

Loading

codecov-commenter commented Nov 5, 2024 •

edited

Loading

jadami10 left a comment •

edited

Loading