Skip to content

Conversation

@wtn
Copy link
Contributor

@wtn wtn commented Nov 22, 2025

Fixes #25410.

group_by_dynamic panicked when window boundaries fell on non-existent datetimes during DST transitions, even when the actual data didn't contain those times.

Reproduction

import polars as pl
import datetime as dt

df = pl.DataFrame({
    "timestamp": pl.datetime_range(
        dt.datetime(2024, 2, 7, 9, 31),
        dt.datetime(2024, 2, 24, 16, 0),
        time_zone="America/New_York",
        eager=True,
        interval="1m"
    )
}).join(pl.DataFrame({"id": [1, 2, 3]}), how="cross")

df.group_by_dynamic(
    "timestamp",
    every="1m",
    period=dt.timedelta(days=17),
    closed="both",
    label="left",
    group_by="id"
).agg(ts_last=pl.col("timestamp").last())
# Panics: datetime '2024-03-10 02:00:00' is non-existent in time zone 'America/New_York'

Cause

The BoundsIter in polars-time calls .unwrap() on duration addition when computing window boundaries. With a 17-day period starting from Feb 7, the window boundaries include March 10, 2024 2:00 AM—a non-existent time in America/New_York when DST springs forward to 3:00 AM. This triggers try_localize_datetime to raise an error with NonExistent::Raise.

Fix

Added fallback helper functions that catch non-existent datetime errors during window boundary calculations. When a boundary falls on a DST gap, the code skips forward by 1 hour and retries. This maintains performance in the common case (just a Result check) while gracefully handling DST transitions.

@github-actions github-actions bot added fix Bug fix python Related to Python Polars rust Related to Rust Polars labels Nov 22, 2025
@wtn wtn force-pushed the dst branch 2 times, most recently from e2c2d32 to ff5d1e9 Compare November 22, 2025 04:16
@codecov
Copy link

codecov bot commented Nov 22, 2025

Codecov Report

❌ Patch coverage is 91.25964% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.70%. Comparing base (8ab1657) to head (a3832da).

Files with missing lines Patch % Lines
crates/polars-time/src/windows/window.rs 81.50% 27 Missing ⚠️
crates/polars-time/src/windows/test.rs 98.22% 4 Missing ⚠️
crates/polars-time/src/windows/group_by.rs 83.33% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25459      +/-   ##
==========================================
+ Coverage   79.58%   79.70%   +0.11%     
==========================================
  Files        1743     1743              
  Lines      240439   240779     +340     
  Branches     3038     3038              
==========================================
+ Hits       191347   191903     +556     
+ Misses      48310    48094     -216     
  Partials      782      782              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wtn wtn force-pushed the dst branch 3 times, most recently from 95938d5 to c18ab6e Compare November 23, 2025 19:48
@wtn wtn marked this pull request as ready for review November 23, 2025 20:17
@orlp
Copy link
Member

orlp commented Nov 25, 2025

I feel this is a bit dangerous. We don't check if the error is indeed a DST transition, and even if true I'm not sure if skipping a full hour is always the correct behavior.

@wtn wtn marked this pull request as draft November 25, 2025 16:18
@wtn wtn force-pushed the dst branch 2 times, most recently from cde2b19 to cf387fe Compare November 25, 2025 17:05
@wtn wtn marked this pull request as ready for review November 25, 2025 18:00
@wtn wtn force-pushed the dst branch 3 times, most recently from 7584916 to ca8042b Compare December 1, 2025 20:04
@wtn
Copy link
Contributor Author

wtn commented Dec 1, 2025

@orlp Applied your feedback. The fix now uses a fallback helper that catches non-existent datetime errors during window boundary calculations and skips forward past the DST gap when a boundary falls on it.

@wtn wtn force-pushed the dst branch 4 times, most recently from 30edecb to 0e6556e Compare December 5, 2025 05:16
@wtn wtn marked this pull request as draft December 5, 2025 05:54
@wtn wtn force-pushed the dst branch 7 times, most recently from eb34d02 to 253a382 Compare December 5, 2025 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Bug fix python Related to Python Polars rust Related to Rust Polars

Projects

None yet

Development

Successfully merging this pull request may close these issues.

group_by_dynamic panics over Daylight Savings transition

2 participants