Skip to content

Commit c5a44fa

Browse files
antiguruclaude
andauthored
expr: recover pushdown for day-only interval predicates (MaterializeInc#36706)
### Motivation After MaterializeInc#36702 (merged) marked `add/sub_timestamp{,_tz}_interval` as `(false, false)` to fix the persist-filter-pushdown audit panic (database-issues#9656), filter pushdown was lost for **literal day-only interval** temporal predicates like `t_col - INTERVAL '1' day < literal` — exactly the pattern temporal filters were built to optimize. The non-monotonicity that motivated the parent fix only manifests when the interval has a non-zero `months` component (calendar-month arithmetic with day-clamping); for `months == 0` the operation reduces to adding a fixed number of microseconds and is straightforwardly monotone in `t`. ### Description Adds a dynamic-monotonicity escape hatch to the abstract interpreter so the `(false, false)` static annotation can be safely upgraded at runtime when the spec proves the interval is harmless. `SpecialBinary` (renamed to `AbstractFunc` per review) grows from a single `map_fn` override into an enum with two variants: ```rust struct AbstractFunc { handler: AbstractFuncHandler, pushdownable: (bool, bool), // (left, right): see Trace } enum AbstractFuncHandler { /// Existing: replace the spec computation entirely. Override(fn(ResultSpec, ResultSpec) -> ResultSpec), /// New: use the default flat-map machinery, but compute the per-argument /// monotonicity verdict from the input specs. DynamicMonotone(fn(&ResultSpec, &ResultSpec) -> (bool, bool)), } ``` `AddTimestampInterval`, `AddTimestampTzInterval`, `SubTimestampInterval`, `SubTimestampTzInterval` now get an `AbstractFunc` whose `DynamicMonotone` handler returns `(months_zero, months_zero)` when the right argument is a single known interval with `months == 0`, and `(false, false)` otherwise. The static `is_monotone()` annotation from MaterializeInc#36702 stays `(false, false)`, so anywhere this dynamic check isn't consulted (e.g. `Trace`'s static pushdownability gate) the conservative answer still applies. `AbstractFunc::pushdownable` is set to `(true, false)` so `Trace` routes `t_col +/- INTERVAL_lit` predicates through pushdown regardless of months — at runtime the dynamic check decides whether to narrow. A `Values::as_single() -> Option<Datum>` helper centralises the "is this a single known value?" check, so the test against `Interval::months` is robust to future variants of `Values` (e.g. a small-discrete-set representation). ### Verification - `interpret::tests::test_timestamp_plus_interval_dynamic_monotone` — three-scenario regression test covering tight narrowing (day-only interval, range straddling the literal), provable elimination (day-only, range strictly above), and conservative fallback (month-bearing literal). - `interpret::tests::proptest_timestamp_plus_interval_monotone_when_months_zero` — proptest that exercises the monotonicity claim against the actual `add_timestamp_interval` impl by sampling random timestamps and zero-month intervals, checking that input ordering is preserved in the output. - `test/sqllogictest/filter-pushdown.slt` — end-to-end test asserting `pushdown=...` is emitted for `WHERE t - INTERVAL '1' day < literal`. ### Known limitations The dynamic upgrade only triggers when the interval is a *constant* with statically-zero months. Day-only column-valued intervals don't get pushdown today — that's a smaller win to leave for later. CASE expressions over multiple day-only interval literals also don't recover (the CASE folds into a `Values::Within` range, not a single value), but that's the right cost trade-off given the conservative static annotation: the existing slt tests at lines 322-353 still reflect that. Co-authored-by: Claude <noreply@anthropic.com>
1 parent d69f688 commit c5a44fa

2 files changed

Lines changed: 327 additions & 29 deletions

File tree

src/expr/src/interpret.rs

Lines changed: 303 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,21 @@ impl<'a> Values<'a> {
123123
},
124124
}
125125
}
126+
127+
/// Returns the sole datum in this value set, if it is known to be a single
128+
/// value. Returns `None` otherwise (for empty sets, ranges with distinct
129+
/// endpoints, structured constraints, and the unconstrained set).
130+
///
131+
/// Prefer this over pattern-matching on [Values::Within] directly when you
132+
/// only need the "single known value" case: it's robust against future
133+
/// variants of [Values] (e.g. a small-set representation) automatically
134+
/// degrading to "not a single value" rather than silently mis-matching.
135+
fn as_single(&self) -> Option<Datum<'a>> {
136+
match self {
137+
Values::Within(a, b) if a == b => Some(*a),
138+
_ => None,
139+
}
140+
}
126141
}
127142

128143
/// An approximation of the set of values an expression might have, including whether or not it
@@ -602,18 +617,43 @@ impl SpecialUnary {
602617
}
603618
}
604619

605-
/// A binary function we've added special-case handling for; including:
606-
/// - A two-argument function, taking and returning [ResultSpec]s. This overrides the
607-
/// default function-handling logic entirely.
620+
/// The abstract-domain counterpart of a [BinaryFunc]: a binary function
621+
/// we've added special-case handling for; including:
622+
/// - Either a complete override of [ResultSpec] computation, or a way to
623+
/// compute monotonicity dynamically from the input specs.
608624
/// - Metadata on whether / not this function is pushdownable. See [Trace].
609-
struct SpecialBinary {
610-
map_fn: for<'a> fn(ResultSpec<'a>, ResultSpec<'a>) -> ResultSpec<'a>,
625+
///
626+
/// Note: today a function can have *either* a handler override *or* a
627+
/// dynamic-monotonicity verdict, but not both. If a future function wants
628+
/// both, promote [AbstractFuncHandler] from an enum to a struct with two
629+
/// optional fields.
630+
struct AbstractFunc {
631+
handler: AbstractFuncHandler,
632+
/// `(left, right)`: per-argument pushdownability hint consumed by
633+
/// [Trace]. `true` for an argument means the function preserves enough
634+
/// structure that, with sufficient information about that argument's
635+
/// range, the output spec can be predicted — i.e. the predicate is a
636+
/// pushdown candidate when that argument is constant or a tight range.
611637
pushdownable: (bool, bool),
612638
}
613639

614-
impl SpecialBinary {
640+
/// How an [AbstractFunc] computes the output [ResultSpec].
641+
enum AbstractFuncHandler {
642+
/// Completely override the spec computation; the default flat-map machinery
643+
/// is bypassed.
644+
Override(for<'a> fn(ResultSpec<'a>, ResultSpec<'a>) -> ResultSpec<'a>),
645+
/// Use the default flat-map machinery, but with a monotonicity verdict that
646+
/// depends on the input specs. This lets us claim monotonicity for cases
647+
/// the static `LazyBinaryFunc::is_monotone` annotation can't safely claim:
648+
/// for instance, `t + INTERVAL '1' day` is monotone in `t`, but `t + i`
649+
/// generally isn't (the calendar-month / day-clamping arithmetic in
650+
/// `add_timestamp_interval` is non-monotone when `i.months != 0`).
651+
DynamicMonotone(fn(&ResultSpec<'_>, &ResultSpec<'_>) -> (bool, bool)),
652+
}
653+
654+
impl AbstractFunc {
615655
/// Returns the special-case handling for a particular function, if it exists.
616-
fn for_func(func: &BinaryFunc) -> Option<SpecialBinary> {
656+
fn for_func(func: &BinaryFunc) -> Option<AbstractFunc> {
617657
/// Eager in the same sense as `func.rs` uses the term; this assumes that
618658
/// nulls and errors propagate up, and we only need to define the behaviour
619659
/// on values.
@@ -693,19 +733,51 @@ impl SpecialBinary {
693733
})
694734
}
695735

736+
/// `add_timestamp_interval` and friends do calendar-month arithmetic
737+
/// with day-clamping, which is non-monotone in either argument when
738+
/// `interval.months != 0`. But when `interval.months == 0` the
739+
/// operation reduces to adding a fixed number of microseconds, which
740+
/// *is* monotone in both arguments. The static `is_monotone`
741+
/// annotation has to pick the conservative answer; this dynamic check
742+
/// recovers filter pushdown for the common case of literal
743+
/// `INTERVAL '<N>' day`-style predicates.
744+
fn timestamp_plus_interval_monotone(
745+
_left: &ResultSpec<'_>,
746+
right: &ResultSpec<'_>,
747+
) -> (bool, bool) {
748+
let months_zero = matches!(
749+
right.values.as_single(),
750+
Some(Datum::Interval(i)) if i.months == 0,
751+
);
752+
(months_zero, months_zero)
753+
}
754+
696755
match func {
697-
BinaryFunc::JsonbGetString(_) => Some(SpecialBinary {
698-
map_fn: |l, r| jsonb_get_string(l, r, false),
756+
BinaryFunc::JsonbGetString(_) => Some(AbstractFunc {
757+
handler: AbstractFuncHandler::Override(|l, r| jsonb_get_string(l, r, false)),
699758
pushdownable: (true, false),
700759
}),
701-
BinaryFunc::JsonbGetStringStringify(_) => Some(SpecialBinary {
702-
map_fn: |l, r| jsonb_get_string(l, r, true),
760+
BinaryFunc::JsonbGetStringStringify(_) => Some(AbstractFunc {
761+
handler: AbstractFuncHandler::Override(|l, r| jsonb_get_string(l, r, true)),
703762
pushdownable: (true, false),
704763
}),
705-
BinaryFunc::Eq(_) => Some(SpecialBinary {
706-
map_fn: eq,
764+
BinaryFunc::Eq(_) => Some(AbstractFunc {
765+
handler: AbstractFuncHandler::Override(eq),
707766
pushdownable: (true, true),
708767
}),
768+
BinaryFunc::AddTimestampInterval(_)
769+
| BinaryFunc::AddTimestampTzInterval(_)
770+
| BinaryFunc::SubTimestampInterval(_)
771+
| BinaryFunc::SubTimestampTzInterval(_) => Some(AbstractFunc {
772+
handler: AbstractFuncHandler::DynamicMonotone(timestamp_plus_interval_monotone),
773+
// For [Trace]: we *might* be pushdownable in the first argument
774+
// (we are when the interval is a literal with no months). The
775+
// interval argument is reported as non-pushdownable so that
776+
// `t_col +/- col_interval` doesn't get routed through pushdown
777+
// for no benefit; if both sides are constants the predicate
778+
// collapses anyway.
779+
pushdownable: (true, false),
780+
}),
709781
_ => None,
710782
}
711783
}
@@ -885,24 +957,36 @@ impl<'a> Interpreter for ColumnSpecs<'a> {
885957
left: Self::Summary,
886958
right: Self::Summary,
887959
) -> Self::Summary {
888-
let (left_monotonic, right_monotonic) = func.is_monotone();
889960
let fallible = func.could_error() || left.range.fallible || right.range.fallible;
890961

891-
let mapped_spec = if let Some(special) = SpecialBinary::for_func(func) {
892-
(special.map_fn)(left.range, right.range)
893-
} else {
894-
let mut expr = MirScalarExpr::CallBinary {
895-
func: func.clone(),
896-
expr1: Box::new(Self::placeholder(left.col_type.clone())),
897-
expr2: Box::new(Self::placeholder(right.col_type.clone())),
898-
};
899-
left.range.flat_map(left_monotonic, |left_result| {
900-
Self::set_argument(&mut expr, 0, left_result);
901-
right.range.flat_map(right_monotonic, |right_result| {
902-
Self::set_argument(&mut expr, 1, right_result);
903-
self.eval_result(expr.eval(&[], self.arena))
962+
let special = AbstractFunc::for_func(func);
963+
let (left_monotonic, right_monotonic) = match &special {
964+
Some(AbstractFunc {
965+
handler: AbstractFuncHandler::DynamicMonotone(monotone_fn),
966+
..
967+
}) => monotone_fn(&left.range, &right.range),
968+
_ => func.is_monotone(),
969+
};
970+
971+
let mapped_spec = match special {
972+
Some(AbstractFunc {
973+
handler: AbstractFuncHandler::Override(f),
974+
..
975+
}) => f(left.range, right.range),
976+
_ => {
977+
let mut expr = MirScalarExpr::CallBinary {
978+
func: func.clone(),
979+
expr1: Box::new(Self::placeholder(left.col_type.clone())),
980+
expr2: Box::new(Self::placeholder(right.col_type.clone())),
981+
};
982+
left.range.flat_map(left_monotonic, |left_result| {
983+
Self::set_argument(&mut expr, 0, left_result);
984+
right.range.flat_map(right_monotonic, |right_result| {
985+
Self::set_argument(&mut expr, 1, right_result);
986+
self.eval_result(expr.eval(&[], self.arena))
987+
})
904988
})
905-
})
989+
}
906990
};
907991

908992
let col_type = func.output_type(&[left.col_type, right.col_type]);
@@ -1124,7 +1208,7 @@ impl Interpreter for Trace {
11241208
left: Self::Summary,
11251209
right: Self::Summary,
11261210
) -> Self::Summary {
1127-
let (left_pushdownable, right_pushdownable) = match SpecialBinary::for_func(func) {
1211+
let (left_pushdownable, right_pushdownable) = match AbstractFunc::for_func(func) {
11281212
None => func.is_monotone(),
11291213
Some(special) => special.pushdownable,
11301214
};
@@ -2115,6 +2199,196 @@ mod tests {
21152199
);
21162200
}
21172201

2202+
/// Companion test to `test_add_timestamp_interval_non_monotone`: when the
2203+
/// interval argument is a literal with `months == 0`, the function reduces
2204+
/// to a pure linear shift in microseconds and *is* monotone in the
2205+
/// timestamp. The dynamic-monotonicity handler in `AbstractFunc` should
2206+
/// recover the tight output range in that case, so that filter pushdown
2207+
/// can still narrow predicates like `t - INTERVAL '1' day < literal`.
2208+
#[mz_ore::test]
2209+
#[cfg_attr(miri, ignore)]
2210+
fn test_timestamp_plus_interval_dynamic_monotone() {
2211+
use chrono::NaiveDateTime;
2212+
use mz_repr::adt::interval::Interval;
2213+
use mz_repr::adt::timestamp::CheckedTimestamp;
2214+
use mz_repr::{Datum, Row};
2215+
2216+
let arena = RowArena::new();
2217+
2218+
let ts = |s: &str| {
2219+
Datum::Timestamp(
2220+
CheckedTimestamp::from_timestamplike(
2221+
NaiveDateTime::parse_from_str(s, "%Y-%m-%dT%H:%M:%S").unwrap(),
2222+
)
2223+
.unwrap(),
2224+
)
2225+
};
2226+
let interval_lit = |months: i32, days: i32, micros: i64| {
2227+
let mut row = Row::default();
2228+
row.packer().push(Datum::Interval(Interval {
2229+
months,
2230+
days,
2231+
micros,
2232+
}));
2233+
MirScalarExpr::Literal(Ok(row), ReprScalarType::Interval.nullable(false))
2234+
};
2235+
2236+
let relation = ReprRelationType::new(vec![ReprScalarType::Timestamp.nullable(false)]);
2237+
2238+
// (a) `t_col - INTERVAL '1' day < 2024-01-15`, with `t_col` ranging
2239+
// over `[2024-01-15, 2024-01-20]`. With the days-only interval, the
2240+
// subtraction is monotone, so endpoints alone determine the output:
2241+
// [2024-01-14, 2024-01-19]. Only `2024-01-14` satisfies `< 2024-01-15`,
2242+
// so both True and False are reachable.
2243+
{
2244+
let expr = MirScalarExpr::column(0)
2245+
.call_binary(interval_lit(0, 1, 0), SubTimestampInterval)
2246+
.call_binary(
2247+
MirScalarExpr::Literal(
2248+
Ok({
2249+
let mut r = Row::default();
2250+
r.packer().push(ts("2024-01-15T00:00:00"));
2251+
r
2252+
}),
2253+
ReprScalarType::Timestamp.nullable(false),
2254+
),
2255+
Lt,
2256+
);
2257+
let mut interpreter = ColumnSpecs::new(&relation, &arena);
2258+
interpreter.push_column(
2259+
0,
2260+
ResultSpec::value_between(ts("2024-01-15T00:00:00"), ts("2024-01-20T00:00:00")),
2261+
);
2262+
let range_out = interpreter.expr(&expr).range;
2263+
assert!(
2264+
range_out.may_contain(Datum::True),
2265+
"day-only interval should preserve tight bounds",
2266+
);
2267+
assert!(
2268+
range_out.may_contain(Datum::False),
2269+
"day-only interval should preserve tight bounds",
2270+
);
2271+
}
2272+
2273+
// (b) Same predicate, but with `t_col` strictly *after* the literal:
2274+
// `[2024-01-17, 2024-01-20]`. Output of `t - 1 day`:
2275+
// `[2024-01-16, 2024-01-19]`, none of which is `< 2024-01-15`. The
2276+
// interpreter must rule out `True`.
2277+
{
2278+
let expr = MirScalarExpr::column(0)
2279+
.call_binary(interval_lit(0, 1, 0), SubTimestampInterval)
2280+
.call_binary(
2281+
MirScalarExpr::Literal(
2282+
Ok({
2283+
let mut r = Row::default();
2284+
r.packer().push(ts("2024-01-15T00:00:00"));
2285+
r
2286+
}),
2287+
ReprScalarType::Timestamp.nullable(false),
2288+
),
2289+
Lt,
2290+
);
2291+
let mut interpreter = ColumnSpecs::new(&relation, &arena);
2292+
interpreter.push_column(
2293+
0,
2294+
ResultSpec::value_between(ts("2024-01-17T00:00:00"), ts("2024-01-20T00:00:00")),
2295+
);
2296+
let range_out = interpreter.expr(&expr).range;
2297+
assert!(
2298+
!range_out.may_contain(Datum::True),
2299+
"day-only interval should narrow out impossible matches",
2300+
);
2301+
}
2302+
2303+
// (c) With a *month*-bearing literal interval, the operation is no
2304+
// longer monotone (day-clamping), so the dynamic-monotonicity handler
2305+
// must fall back to `anything()` — the interpreter cannot rule out
2306+
// either outcome even when the column range is narrow.
2307+
{
2308+
let expr = MirScalarExpr::column(0)
2309+
.call_binary(interval_lit(1, 0, 0), SubTimestampInterval)
2310+
.call_binary(
2311+
MirScalarExpr::Literal(
2312+
Ok({
2313+
let mut r = Row::default();
2314+
r.packer().push(ts("2024-01-15T00:00:00"));
2315+
r
2316+
}),
2317+
ReprScalarType::Timestamp.nullable(false),
2318+
),
2319+
Lt,
2320+
);
2321+
let mut interpreter = ColumnSpecs::new(&relation, &arena);
2322+
interpreter.push_column(
2323+
0,
2324+
ResultSpec::value_between(ts("2024-01-17T00:00:00"), ts("2024-01-20T00:00:00")),
2325+
);
2326+
let range_out = interpreter.expr(&expr).range;
2327+
assert!(
2328+
range_out.may_contain(Datum::True),
2329+
"month-bearing interval must conservatively admit True",
2330+
);
2331+
assert!(
2332+
range_out.may_contain(Datum::False),
2333+
"month-bearing interval must conservatively admit False",
2334+
);
2335+
}
2336+
}
2337+
2338+
/// Proptest companion to [`test_timestamp_plus_interval_dynamic_monotone`]:
2339+
/// the dynamic-monotonicity handler in [`AbstractFunc`] claims that
2340+
/// `add_timestamp_interval(t, i)` is monotone in `t` whenever `i.months == 0`
2341+
/// (the only case it actually claims monotonicity for at runtime: the
2342+
/// matches above require the right argument to be a single value with
2343+
/// `months == 0`). This proptest verifies that claim directly against the
2344+
/// function impl by sampling random timestamps and zero-month intervals
2345+
/// and checking that input ordering is preserved in the output.
2346+
#[mz_ore::test]
2347+
#[cfg_attr(miri, ignore)]
2348+
fn proptest_timestamp_plus_interval_monotone_when_months_zero() {
2349+
use mz_repr::adt::interval::Interval;
2350+
use mz_repr::{Datum, RowArena, SqlScalarType, arb_datum_for_scalar};
2351+
use proptest::prelude::*;
2352+
2353+
let timestamp_strat = || arb_datum_for_scalar(SqlScalarType::Timestamp { precision: None });
2354+
// Lex order on `Interval` does *not* match total-microseconds order when
2355+
// both days and micros vary independently (e.g. `{0, 0, 86_400_000_001}`
2356+
// is lex-less than `{0, 1, 0}` but evaluates to a strictly larger
2357+
// timestamp), so we only claim monotonicity for *fixed* zero-month
2358+
// intervals — which is exactly what the DynamicMonotone handler does.
2359+
// The proptest accordingly varies `t` with `i` held constant.
2360+
let zero_month_interval_strat =
2361+
(any::<i32>(), any::<i64>()).prop_map(|(days, micros)| Interval {
2362+
months: 0,
2363+
days,
2364+
micros,
2365+
});
2366+
2367+
let expr = MirScalarExpr::CallBinary {
2368+
func: AddTimestampInterval.into(),
2369+
expr1: Box::new(MirScalarExpr::column(0)),
2370+
expr2: Box::new(MirScalarExpr::column(1)),
2371+
};
2372+
let arena = RowArena::new();
2373+
2374+
proptest!(|(
2375+
t1 in timestamp_strat(),
2376+
t2 in timestamp_strat(),
2377+
i in zero_month_interval_strat,
2378+
)| {
2379+
let t1 = match t1 { PropDatum::Timestamp(t) => t, _ => unreachable!() };
2380+
let t2 = match t2 { PropDatum::Timestamp(t) => t, _ => unreachable!() };
2381+
let i = Datum::Interval(i);
2382+
let r1 = expr.eval(&[Datum::Timestamp(t1), i], &arena);
2383+
let r2 = expr.eval(&[Datum::Timestamp(t2), i], &arena);
2384+
// Only compare when both calls succeed; the monotonicity claim
2385+
// applies only within the success domain.
2386+
if let (Ok(Datum::Timestamp(r1)), Ok(Datum::Timestamp(r2))) = (r1, r2) {
2387+
prop_assert_eq!(t1.cmp(&t2), r1.cmp(&r2));
2388+
}
2389+
});
2390+
}
2391+
21182392
/// Regression test for `date_bin_timestamp`, which is non-monotone in the
21192393
/// `stride` argument: a larger stride can bin a source timestamp to an
21202394
/// *earlier* result than a smaller stride, because the bin alignment to

0 commit comments

Comments
 (0)