allow workflow config to unset global events config

[TomekTrzeciak]

This addresses a bug whereby it is not possible to reset list-type values like mail events from the global config to empty list by overriding them in the workflow config.

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• Changelog entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[MetRonnie]

Thanks @TomekTrzeciak 👍 it might be January before we have a chance to look at this however

[oliver-sanders]

This would change the meaning of mail events = from "reset to global default" to "reset to empty list".

The "reset to empty lisy" behaviour makes sense, but unfortunately, I think there may be valid use cases for both behaviours (especially when you take inheritance into account). I had a skim through workflows at our site to see what list items are being "unset" in this way and found the following:

• mail events (a lot of instances)
• (submission|execution) retry delays (a lot of instances)
• handler events (a few instances)
• collapsed families (a couple instances)

[MetRonnie]

I think this "reset to global default" behaviour only happens for events settings and no other settings?

[TomekTrzeciak]

I think this "reset to global default" behaviour only happens for events settings and no other settings?

My reading is that only events and mail settings under scheduler section would be affected by this change.

This would change the meaning of mail events = from "reset to global default" to "reset to empty list".

The "reset to empty lisy" behaviour makes sense, but unfortunately, I think there may be valid use cases for both behaviours (especially when you take inheritance into account). I had a skim through workflows at our site to see what list items are being "unset" in this way and found the following:

• mail events (a lot of instances)
• (submission|execution) retry events (a lot of instances)
• handler events (a few instances)
• collapsed families (a couple instances)

Out of this list only mail events and handler events would be affected by this change. On our site only the former (mail events) is set in the global config and thus would behave differently (and as expected IMHO).

[hjoliver]

So this change would mean:

• no mention in workflow config: you get the global defaults
• explicit key = # nothing: you get nothing

Seems good to me, no blockers as far as NIWA is concerned.

@oliver-sanders - would it be sufficient for you just to ping the owners of the workflows you found and tell them about the change?

[oliver-sanders]

@oliver-sanders - would it be sufficient for you just to ping the owners of the workflows you found and tell them about the change?

Will do. I'll confirm the scope of the change and try to work out the intention of the users impacted. So long as it's just [scheduler][events and [scheduler][mail], I'm not expecting any issues.

[oliver-sanders]

👍

I've taken a look into the change and am happy with the scope.

I had a look on site and found mail events = in 513 run dirs belonging to 47 users internally. I highly doubt that this change has the potential to cause any issues, but I've put up an internal bulletin to ensure that folks are aware. I don't think we need to go any further than that.

A quick changelog would be appreciated.

[MetRonnie]

Sorry, just gotten round to this.

Apparently though, it is already possible to unset events by using a value of None, i.e.

# flow.cylc
[scheduler]
    [[events]]
        mail events = None

(Apologies for not realising this when you asked me back in December!)

[oliver-sanders]

^ @TomekTrzeciak thoughts?

[hjoliver]

cylc-flow/cylc/flow/cfgspec/workflow.py

Lines 396 to 405 in 5b2b180

	with Conf('events',
	desc=global_default(EVENTS_DESCR, '[scheduler][events]')):
	for item, desc in EVENTS_SETTINGS.items():
	desc = global_default(desc, f"[scheduler][events]{item}")
	vdr_type = VDR.V_STRING_LIST
	default: Any = Conf.UNSET
	if item in {'handlers', 'handler events', 'mail events'}:
	# Note: default of None for V_STRING_LIST is used to
	# differentiate between not set vs set to empty
	default = None

I had forgotten about this too, sorry.

However, it doesn't seem to be documented, and use of "None" (being Pythonic) is weird in a config file.
So I'd prefer this PR.
And we should document it.

[MetRonnie]

Actually I've realised that mail events = None is no different from mail events = meow, giraffe, i.e. as long as you put a (list of) invalid values it will turn off mail events 😅 (The code Hillary linked is not actually relevant). Edit: an issue has been open for this for a while:

• handler events list not validated #2756

[MetRonnie]

I've pushed a changelog entry. Got one optional suggestion, take or leave it 👍

[MetRonnie]

Think this could be simplified to

Suggested change

	except (KeyError, TypeError, ValueError) as exc:
	except Exception as exc:

[MetRonnie]

Actually, might as well merge