Adoption of Pendulum package for datetime management

[Bridg109]

Proposal

Review and consider using the Pendulum package for all airflow datetime management functions. This is currently what airflow has in stock macros as class for datetime (the other options are strings).

Rationale

This seems to be the most native way to doing datetime things in airflow since the pendulum objects are used as standard macro/template elements. Pendulum community is not nearly as robust as datetime and might result in us having to develop more of our own code to do things. If we go this route could consider some blog post to add to the pendulum community as well as help cover a gap in airflow documentation.

[jrlegrand]

Would this obviate the need for a custom SageRx ds_datetime function? Or no?

[Bridg109]

Would this obviate the need for a custom SageRx ds_datetime function? Or no?

yes...if we can manage to covert most date management functions over. It will hopefully reduce the need to frequently make user defined macros and/or reduce having to do spaghetti long airflow macros referencing as well.

It seems that pendulum can do everything we need and cleaner then datetime...however it can be hard to find answers at datetime answers dominate results when googling.

as example (same number of lines but one is far more elegant to follow)...noting this was like the 4th answer in a stack overflow, datetime taking the first couple.

from datetime import date, timedelta

today = date.today()
start = today - timedelta(days=today.weekday())
end = start + timedelta(days=6)

vs

import pendulum
today = pendulum.now()
start = today.start_of('week')
end = today.end_of('week')

My guess is airflow uses pendulum as it is aimed at managing timezones. Which seems like it would be needed when managing airflow on multiple servers using celery (not an issue for us at the moment)

[Bridg109]

We can close this for now...the current method is working just longer syntax

[jrlegrand]

Re-opening to convert to discussion.