Skip to content

Conversation

@kdn36
Copy link
Collaborator

@kdn36 kdn36 commented Nov 25, 2025

fixes #25434

This also fixes an undocumented issue whereby the rolling group sum kernel returns None on empty groups instead of 0.

This PR introduces the monotonic attribute for groups. It is used now -- in combination with overlapping -- for dispatch to the rolling kernel, but it can be leveraged for other use cases in the future.

The dispatch logic has been tested one-off for correct dispatch on rolling and group_by_dynamic, and against false misclassifications on the CI suite. Still, it is possible that it gets misclassified now or later, but the impact is limited.

Confirmed following the rolling agg kernel for sum:

  • group_by_dynamic
  • rolling
  • rolling in group_by
  • group_by_dynamic with by argument

This PR replaces #25470

@kdn36 kdn36 marked this pull request as draft November 25, 2025 22:10
@github-actions github-actions bot added internal An internal refactor or improvement python Related to Python Polars rust Related to Rust Polars labels Nov 25, 2025
@codecov
Copy link

codecov bot commented Nov 25, 2025

Codecov Report

❌ Patch coverage is 90.57592% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.65%. Comparing base (04e8c78) to head (b078afd).
⚠️ Report is 13 commits behind head on main.

Files with missing lines Patch % Lines
crates/polars-core/src/frame/group_by/position.rs 86.27% 7 Missing ⚠️
...ops/src/chunked_array/nan_propagating_aggregate.rs 0.00% 6 Missing ⚠️
...ates/polars-core/src/frame/group_by/into_groups.rs 66.66% 2 Missing ⚠️
crates/polars-expr/src/expressions/mod.rs 87.50% 2 Missing ⚠️
crates/polars-time/src/group_by/dynamic.rs 91.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25494      +/-   ##
==========================================
+ Coverage   79.39%   79.65%   +0.25%     
==========================================
  Files        1742     1743       +1     
  Lines      240100   240295     +195     
  Branches     3038     3038              
==========================================
+ Hits       190634   191408     +774     
+ Misses      48683    48105     -578     
+ Partials      783      782       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kdn36 kdn36 marked this pull request as ready for review November 26, 2025 12:41
Copy link
Member

@orlp orlp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we have these overlapping and monotonic attributes, setting them wrong somewhere can be disastrous. Can you replace all instances where we create a GroupsType with some sort of constructor (for example GroupStype::new_slice(groups, overlapping, monotonic)) which when #[cfg(debug_assertions)] is true actually checks those properties? That should allow us to catch bugs early in the CI.

@kdn36
Copy link
Collaborator Author

kdn36 commented Nov 26, 2025

Now that we have these overlapping and monotonic attributes, setting them wrong somewhere can be disastrous. Can you replace all instances where we create a GroupsType with some sort of constructor (for example GroupStype::new_slice(groups, overlapping, monotonic)) which when #[cfg(debug_assertions)] is true actually checks those properties? That should allow us to catch bugs early in the CI.

Yes, for sure. That said, I’m just as concerned about the scenario where we set it correctly initially, but the groups later change and the attribute isn’t updated. We could put the attribute behind a getter and add run-time verification in debug builds, to be exercsied in CI and when debugging issues. These attributes aren’t accessed often and are typically used only for dispatch between different implementation paths.

@kdn36 kdn36 force-pushed the refactor_rolling_groups_v2 branch from 32634fb to 0636e3b Compare December 5, 2025 10:25
@kdn36 kdn36 requested a review from orlp December 5, 2025 10:55
Copy link
Member

@orlp orlp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a small nit then it's ready to go.

false
}

assert!(!groups_overlap(&groups) || overlapping);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Swap the order, only do the test if it's claimed the groups aren't overlapping:

assert!(overlapping || !groups_overlap(&groups));

You always want to put cheap conditions before expensive conditions if possible.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent point.

}
true
}
assert!(groups_are_monotonic(&groups) || !monotonic);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

@orlp orlp merged commit 4742c6a into pola-rs:main Dec 5, 2025
28 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal An internal refactor or improvement python Related to Python Polars rust Related to Rust Polars

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Panic in rolling sum with varying window size

2 participants