Planned Outage to XCS / Monsoon Service due to Storm Éowyn

[Roger Milton (mo-roger-milton)]

Currently there is no UPS protection to the XCS disk systems (used by Monsoon) at Exeter Science Park. The science park data centre was only ever designed and built with limited resilience.

A fault was recently identified with the UPS requiring batteries to be replaced. The earliest this can happen is mid-February.

As the Met Office weather warning indicates increased risk of power disruption, there is an increased risk to XCS supplies (Compute and disks).

We will be stopping ALL compute work on XCS for the duration of the storm warning. This is specifically to limit damage to disks should we have a power problem and the system crashes. The expectation is that with limited read/write activity at the time of a crash, the recovery should be easier, and we reduce the chance of data corruption / loss.

IT Operations will manage the workload on XCS, informed by the weather guidance.

This process shall remain in place until the UPS repairs have been completed.

Apologies for further disruption to the Monsoon service. We'll aim to give more specifics about the timing of the outage shortly.

[Roger Milton (mo-roger-milton)]

I've had an update on this, and reservations are now in place on XCS from 00:00 to 08:00 tomorrow (Friday 24th January 2025). Work will begin to drain from the XCS system in advance of that tonight, and be restored in the morning. This will apply to both compute and shared nodes.