Additional holiday calendars for quadruple witching et cetera + other extensions.

[jenskeiner]

Hi all,

I've been using the exchange_calendars package for some time now. In one instance, I've used it in a professional capacity to build a small HTTP-based service that would allow, for example, to retrieve the next 10 regular business days relative to a given date for an exchange.

Originally, the service should have been a thin wrapper around the package, but over time, we were tasked with adding more special business days like quadruple witching, or last (regular) trading day of the month, as trading patterns on those days would merit special treatment. We added these additional days for select exchanges, but the logic was never properly pulled into a separate library.

Now, after some deliberation, I have finally started to develop a Python package exchange_calendars_extensions to add said functionality as transparently as possible to exchange_calendars. Rationale: I wanted to re-implement the functionality from scratch right now, but without having to try and add to exchange_calendars directly at the same time.

At this stage, I have added some additional special day types, like quadruple witching, and some convenience, e.g. a holiday calendar that combines regular and ad-hoc holidays.

Another aspect, with is not yet implemented, would be functionality to add/change holidays or other special days programmatically at runtime. This can be useful when correctness is mission-critical, you're made aware of short-term changes, and you can't wait for a new exchange_calendars release which may come after the fact. Until the changes are reflected upstream, it may be a good option to just apply patches to the calendars through a well-defined API.

I would be interested in feedback. It's fine for me to keep all the additional stuff in a separate package, but if anything would reasonably qualify for adding to exchange_calendars, I would be happy to contribute.

Let me know any thoughts.

[maread99]

Hi @jenskeiner

This looks absolutely great! I think all of this would be more than worthy additions to exchange_calendars. Indeed the combination of regular and ad-hoc holidays is something that should always have always been in here. I love the option to include the holiday name. The expiries and last_trading_days_of_months properties would be really great too.

My only quibble is the direct use of the HolidayCalendar class and holidays method to get the expiry sessions and last_trading_days_of_months, simply because they imply that these sessions are holidays. I'd suggest perhaps subclassing HolidayCalendar as SpecialSessionsCalendar or similar and overwriting the holidays method as sessions. Also, perhaps renaming 'last_trading_days_of_months' as 'monthly_last_sessions' just to keep the terminology in terms of 'sessions'.

It'll be @gerrymanoim's call, although for me, I'd say PLEASE do offer PRs to incorporate all of this in exchange_calendars. Would need docs and tests - although looks like you've got it all covered (only thing, exchange_calendars uses numpy style for docs).

Wonderful stuff!

[jenskeiner]

Hi @maread99,

thanks for the feedback which sounds reasonable. I'll try to incorporate that and add some more stuff I have been thinking about, before I'll consider creating PRs to move stuff over to exchange_calendars. Just want this to stabilise a bit so it's usable w/o having to wait for integration into the upstream package.

In the meantime, feel free to share more feedback with me and/or track progress over in exchange_calendars_extensions.

Thanks again.

[jenskeiner]

Hi @maread99,

regarding HolidayCalendar vs. SpecialSessionsCalendar, I noticed that e.g. the existing special_closes member also uses HolidayCalendar to define what are actually business days.

Also, defining a separate class may allow you to add another property like sessions, but you would need to inherit from pandas' AbstractHolidayCalendar at least anyway to avoid redundancies, thus keeping the holidays property.

Also, if you merge holidays and business days from different calendars into a new one, which type of calendar would that be?

In the end, I think it may be easier to just use HolidayCalendar, even to define (special) business days, because
a) exchange_calendars already does exactly that,
b) adding a new class either requires copy & pasting code from pandas, or doesn't achieve the strict interface separation,
c) cases where business days and holidays get mixed may arise, thus a common class is easier to handle.

What do you think?

@gerrymanoim: Any thoughts?

Cheers!

[maread99]

Hi @jenskeiner

Yup, scrap any suggestion of further subclassing HolidayCalendar. Sorry, I was barking up completely the wrong tree...

Having had a good look again over the existing interface, I don't feel that that the HolidayCalendar class, or any subclass of, should be the public facing object through which users should be expected to query anything. Bare with me...

I think it's fair to say that exchange_calendars has always been more orientated towards creating and interrogating schedules of trading sessions rather than holidays. The holiday info is there because it's required to create the trading schedules, but it's never been given much value in its own right. This is atested to by ExchangeCalendar lacking a way to query holidays in the manner that holidays_all provides for. Yes, regular_holdiays and adhoc_holidays can be queried although these are there more as a side effect of how the definition of calendars is implemented rather than an intentional public-facing interface. The same is true of special_opens, special_closes, close_times, close_offset etc. Thse are all required to evaluate the trading schedule, although they are not really intended for public consumption in their own right.

Considering in turn each of the properties that exchange_calendars_extensions is serving as a HolidayCalendar:

holidays_all

In order to keep the "public" interface consistent in terms of usage and terminology, I'd suggest that the combined holidays be explosed via:

• .holidays property which returns all holidays over the period covered by the calendar. This would be analogous to .sessions returning all the trading sessions.
• .holidays_in_range method, analogous with .sessions_in_range, to query holidays within a specific range.

Under-the-bonnet this could all be evaluated:

• without the need to create a dedicated HolidayCalendar
• lazily, only if requested although cached if it is.

special_opens_all and special_closes_all

The special opens and special closes are already exposed via the late_opens and early_closes properties, for example the following are equivalent:

xlon = get_calendar("XLON")
dti = xlon.special_closes_all.holidays(start=xlon.first_session, end=xlon.last_session)
dti = xlon.early_closes

The advantage of special_closes_all is being able to query over a range. If that functionality were required it could be offered via late_opens_in_range and early_closes_in_range methods.

Worth noting a big assumption (I believe) in the exchange_calendars implementation. On the "calendar definition side" the special_opens and special_closes properties provide for defining dates with 'non-standard' open and close times. However, on the "public interface" side these become late_opens and early_closes, i.e. it's assumed that there's no such thing as an 'early open' or a 'late close'.

weekend_days, quarterly_expiries, monthly_expiries, last_trading_days_of_months, last_regular_trading_days_of_months

In the same vein as above, all these could be exposed as simple properties returning DatetimeIndex that give the values over the full calendar. For each a ..._in_range method could be offered to query the property between specific dates.

Under-the-bonnet the HolidayCalendar, as defined in exchange_calendars_extensions, could be created lazily (and cached) and used to serve the public properties / methods.

Perhaps for each the HolidayCalendar could also be made public in its own right with a clear name, for example calendar_weekend_days, calendar_quarterly_expiries etc. Although I'm not sure that public access to the HolidayCalendar adds anything(?).

As I mentioned before, I think the properties your offering in exchange_calendars_extensions would be really great additions to exchange_calendars. I am keen to expose them in a way that's consistent with the existing interface. I'd love to hear your thoughts on all this. Am I being unreasonable in dismissing the value of exposing the HolidayCalendar in favor of a simple property and method?

Cheers
Marcus

[jenskeiner]

Hi @maread99,

thanks for the detailed thoughts and analysis. I think it should be a goal to add any new properties to exchange_calendars in a way that is consistent with the current interface, perhaps refactoring parts of the existing interface, where apropriate.

Thanks also for all the suggestions re naming conventions, data types, et cetera. I think I would agree with most, if not all of it.

At the current stage, I think there are two main considerations. The first is the build-out of the functionality I wanted to add on top of exchange_calendars through the new package. The second is whether some of that new functionality should be added to exchange_calendars eventually, and how to do it in a consistent and clean manner.

As said before, the build-out is not yet complete. At the moment, I'm adding functionality to programmatically add or remove holidays or special sessions. This is again driven by real-world experience where we often became aware of sudden ad-hoc changes to calendars (e.g. after the earthquake in Turkey) and couldn't wait for a new exchange_calendars release to reflect the changes. So we ended up sub-classing affected calendars to introduce or remove holidays. Being able to programmatically change the calendars allows to source ad-hoc changes from somewhere and inject into the running application without havinh to re-deploy.

Most of the work is finished and when done, I will put out a new pre-release of exchange_calendars_extensions. At that stage, I would consider the package feature complete, so eventually, there will be version 1.0.0, maybe after some minor fixes and/or improvements.

I will then probably start work on another application that will use the extensions, and it may become more clear why I've put the calendar extensions into a separate package.

Once the functionality is complete, I am very happy to actively work on bringing some or even all of the functionality over to the upstream exchange_calendars package as well. Even before, I'm happy to discuss necessary interface changes et cetera.

The benefit of working in this sequential order is, I think:

1. The functionality can be added (and used) right now, without having to make a single change to exchange_calendars. So it will get done quicker.
2. Interface changes and new functionality in exchange_calendars should rightly be discussed among a broader audience, if possible, to make sure we're on the right track. Maybe some changes would warrant a new major release since they could introduce breaking changes. If so, then it may be an opportunity to clean up or change other parts of the interface as well. All of this should be discussed and it may take some time.

So, to wrap up: I'm currently getting 1) done as quickly as possible, so 2) can get the focus after.

Thanks again for the input and feedback so far. Let me know any further thoughts.

Best,

Jens

[maread99]

Hi @jenskeiner

I'm adding functionality to programmatically add or remove holidays or special sessions. This is again driven by real-world experience where we often became aware of sudden ad-hoc changes to calendars (e.g. after the earthquake in Turkey) and couldn't wait for a new exchange_calendars release to reflect the changes.

I recall this functionality was also requested by someone in an issue. I'm slightly hesitant given that it might limit contributions if users can easily implement new holidays locally. Having said that, if as a repo we can't offer same-day merging and releasing of PRs offering calendar updates then it's perfectly reasonable, indeed necessary, to have this functionality.

I'm happy to discuss necessary interface changes etcetera

Maybe some changes would warrant a new major release since they could introduce breaking changes. If so, then it may be an opportunity to clean up or change other parts of the interface as well.

When you're in a position to, please do elaborate on what interface changes you think would be beneficial / necessary.

I very much hope you end up able to offer all this to the main repo - it's great stuff!

Cheers
Marcus

[jenskeiner]

Hi @maread99,

thanks for the above.

I would probably say that it may be best to keep the functionality to programmatically adjust calendars inside the exchange_calendars_extensions package and outside of exchange_calendars -- at least for now. It was just a second piece of functionality I wanted to add outside of a concrete app project, but without necessarily having to put it into exchange_calendars directly, hence the extra package.

I hope to wrap up the work on that functionality soon, so I can cut another pre-release. It's definitely a bit more tricky to implement and test, but progress is good.

For sure I will come forward with a proposal of what could be ported over in a first step, along with interface changes that may be advised or required. Let's discuss then.

As always, I value your input.

Best,

Jens