Skip to content

Commit cc30155

Browse files
authored
KVM: make storage heartbeat fence action configurable (graceful-reboot / restart-agent / log-only) (#13090)
* KVM: make storage heartbeat fence action configurable The KVM agent's storage heartbeat scripts (kvmheartbeat.sh and kvmspheartbeat.sh) hard-code an immediate kernel-level reboot via 'echo b > /proc/sysrq-trigger' when a heartbeat write to primary storage times out. This bypasses all OS-level shutdown protections, drops every running VM on the host instantly, and triggers HA cascades onto surviving hosts. For NFS shared storage the binary "heartbeat-write-failed = host-is-dead" heuristic is reasonable. For LINSTOR/DRBD or other replicated local storage, the same disk serves application I/O, replication I/O and heartbeat I/O simultaneously - so a transient I/O contention spike can time out the heartbeat write without the host actually being unhealthy. The result is false-positive sysrq fencing. Adds a new agent.properties option: kvm.heartbeat.fence.action = reboot | graceful-reboot | restart-agent | log-only Default value is "reboot" so existing deployments keep their current behavior. Operators on replicated storage backends can choose a less destructive action: - graceful-reboot: 'systemctl reboot' instead of sysrq, allowing VMs a chance to shut down cleanly - restart-agent: restart cloudstack-agent only, preserving running VMs - log-only: log + alert, no automatic action The existing 'reboot.host.and.alert.management.on.heartbeat.timeout' boolean continues to function as a complete Java-side bypass. Refs: #13089 * kvm-heartbeat: extract shared fence helper; add 'custom' action; rename to hard-reboot Per review on PR #13090: - Refactor common fence-action case into kvmha-fence.sh sourced by both kvmheartbeat.sh and kvmspheartbeat.sh (per @sureshanaparti). - Add 'custom' fence action that invokes an operator-supplied script (kvm.heartbeat.fence.custom.script, default /etc/cloudstack/agent/ heartbeat-fence-custom.sh) with the heartbeat script name as arg; falls back to hard-reboot if the script is missing/non-executable (per @NuxRo). - Rename canonical action 'reboot' -> 'hard-reboot' for clarity; keep 'reboot' accepted as alias so existing deployments don't break (per @DaanHoogland). Default behavior unchanged: sysrq-trigger reboot, required where a stale NFSv3 mount blocks systemctl reboot. * kvmsmpheartbeat.sh: honour kvm.heartbeat.fence.action Per @weizhouapache's review on #13090, kvmsmpheartbeat.sh was still hard-coding the immediate-reboot fence action (echo b > /proc/sysrq-trigger) while kvmheartbeat.sh and kvmspheartbeat.sh delegate to kvmha-fence.sh. Source kvmha-fence.sh and call fence_action so this script picks up the same hard-reboot / graceful-reboot / restart-agent / log-only / custom behaviour from agent.properties. Default behaviour (no property set) remains hard-reboot via sysrq-trigger, so existing deployments are unaffected. * address review (#13090): drop the 'reboot' alias, simplify the agent.properties lookup, clarify precedence - kvm.heartbeat.fence.action accepts hard-reboot (default), graceful-reboot, restart-agent, log-only and custom; the misleading 'reboot' alias is gone. - kvmha-fence.sh reads the property with a plain grep on the key=value form. - agent.properties comment states that reboot.host.and.alert.management.on.heartbeat.timeout stays the global on/off switch and this setting only chooses the action. Signed-off-by: James Peru <jmsperu@gmail.com> * address review (#13090): log-only only writes to syslog, drop the custom action - log-only is now described everywhere (agent.properties, AgentProperties Javadoc, kvmha-fence.sh) as writing a message to the host's syslog and taking no action: no reboot, no agent restart and no management server alert. The "log + alert" wording is gone. - The custom action and kvm.heartbeat.fence.custom.script are removed (agent.properties, AgentProperties.KVM_HEARTBEAT_FENCE_CUSTOM_SCRIPT, the custom branch and CUSTOM_SCRIPT handling in kvmha-fence.sh, the kvmsmpheartbeat.sh comment). Unknown values, including custom, still fall back to hard-reboot. - The AgentProperties Javadoc documents only hard-reboot (no reboot alias), names kvmsmpheartbeat.sh too, and states the precedence with reboot.host.and.alert.management.on.heartbeat.timeout. Claude-Session: https://claude.ai/code/session_01BhiE1Abhs315m3n3UPy6V7 * address review (#13090): restart-agent and log-only stop the host's VMs first With restart-agent or log-only the VMs kept running after the heartbeat went stale, so if the management server declared the host Down, HA could start the same VMs on another host and two copies would write to the same disks (raised by Damans227). - kvmha-fence.sh: restart-agent and log-only now stop every active VM on the host (virsh destroy) before acting; the host itself stays up. If libvirt cannot be queried or any VM is still active afterwards, the fence falls back to hard-reboot. - New agent property kvm.heartbeat.fence.keep.vms.running (default false) lets an operator keep the old behaviour; the fence then logs a WARNING and agent.properties / AgentProperties say it is only safe when no VM on the host can be restarted elsewhere by HA. - hard-reboot and graceful-reboot are unchanged. - The logger and sysrq trigger paths can be overridden through the environment (FENCE_LOGGER, FENCE_SYSRQ_TRIGGER), like AGENT_PROPS, so the new KvmHaFenceScriptTest runs the script with stand-ins for virsh, systemctl, logger and sysrq and never touches the machine running it. Agent module: 65 tests pass (6 new), bash -n clean on all heartbeat scripts. Claude-Session: https://claude.ai/code/session_01V82Ypx7SpAurn6ek1VXRAP * address review (#13090): let the fence leave system VMs running New agent property kvm.heartbeat.fence.include.system.vms (default true). With true, restart-agent and log-only still stop every VM first. With false they stop only user VMs; system VMs (r-, s-, v- and b- names, as generated by VirtualMachineName) keep running and a running system VM no longer forces the hard-reboot fallback. A syslog warning notes that a virtual router could then run twice with the same IP and MAC. kvm.heartbeat.fence.keep.vms.running=true stays the master opt-out: it wins over include.system.vms and its warning says so. Agent module: 69 tests pass (4 new in KvmHaFenceScriptTest). Claude-Session: https://claude.ai/code/session_01V82Ypx7SpAurn6ek1VXRAP * address review (#13090): bound the VM stop phase and guarantee the hard-reboot fallback Before restart-agent or log-only stop any VM, the fence now arms a detached deadman (own session via setsid, else nohup) that hard-reboots the host unless every selected VM is confirmed stopped within kvm.heartbeat.fence.stop.vms.timeout seconds (default 30), so the fallback still happens when the agent kills the script on heartbeat.update.timeout. Each virsh call is also bounded by coreutils timeout (10 s, or the budget if lower). Claude-Session: https://claude.ai/code/session_011DLKuGySDaq3JQs6SAw3TC * address review (#13090): exact property parsing, safe fallbacks, restart-agent loop guard - read_agent_property parses agent.properties the way java.util.Properties does: the key is compared as a plain string (dots are no longer regex wildcards), whitespace around '=' or ':' (or a whitespace separator) is accepted, the value is trimmed, '#' and '!' lines are comments and the last definition wins. - kvmheartbeat.sh, kvmspheartbeat.sh and kvmsmpheartbeat.sh fall back to the original sysrq hard-reboot, with an ERROR in syslog, if kvmha-fence.sh cannot be sourced or does not define fence_action. - restart-agent and graceful-reboot check for systemctl before acting and hard-reboot with an ERROR if it is missing or the systemctl call fails. - restart-agent records its time under /run (kept across agent restarts, cleared by a reboot). A new restart-agent fence within kvm.heartbeat.fence.restart.agent.cooldown seconds (default 600, 0 turns the guard off) escalates to hard-reboot, so a host whose storage stays down cannot loop on agent restarts. Documented in agent.properties and AgentProperties. - KvmHaFenceScriptTest: graceful-reboot (systemctl reboot, no VM stop by the fence), failing and missing systemctl, whitespace/colon separators, last definition wins, dotted key exact match, the cooldown guard, and the heartbeat scripts' fallback when the helper is missing or empty. Claude-Session: https://claude.ai/code/session_01EUU5skTaj95LfpkSbrWhPN --------- Signed-off-by: James Peru <jmsperu@gmail.com> Co-authored-by: jmsperu <jmsperu@users.noreply.github.com>
1 parent 53a6c0c commit cc30155

7 files changed

Lines changed: 1065 additions & 15 deletions

File tree

‎agent/conf/agent.properties‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -318,6 +318,80 @@ iscsi.session.cleanup.enabled=false
318318
# This parameter specifies if the host must be rebooted when something goes wrong with the heartbeat.
319319
#reboot.host.and.alert.management.on.heartbeat.timeout=false
320320

321+
# Action taken by kvmheartbeat.sh / kvmspheartbeat.sh / kvmsmpheartbeat.sh when a storage heartbeat
322+
# write fails persistently. Only consulted when fencing is enabled, i.e.
323+
# 'reboot.host.and.alert.management.on.heartbeat.timeout' is true: that boolean remains
324+
# the global on/off switch for fencing, this setting only chooses WHAT the fence does.
325+
#
326+
# Allowed values:
327+
# hard-reboot - immediate sysrq-trigger reboot (default).
328+
# Required default for setups where a stale NFSv3 mount can prevent
329+
# a graceful shutdown from completing.
330+
# graceful-reboot - 'systemctl reboot' instead of sysrq; allows VMs to stop cleanly.
331+
# Use only if a stale storage mount cannot block shutdown.
332+
# restart-agent - stop every VM on the host, then restart cloudstack-agent; the
333+
# host itself stays up. If it already fired within
334+
# kvm.heartbeat.fence.restart.agent.cooldown, hard-reboot instead.
335+
# log-only - stop every VM on the host, then only write a message to the
336+
# host's syslog (no reboot, no agent restart, no management server
337+
# alert). The operator must watch syslog and investigate.
338+
#
339+
# Any other value falls back to hard-reboot. restart-agent and graceful-reboot need
340+
# systemctl; if it is missing or the systemctl call fails, the fence hard-reboots.
341+
#
342+
# restart-agent and log-only stop the VMs because, once the heartbeat is stale, the
343+
# management server may declare the host Down and HA may start the same VMs on another
344+
# host; two copies would then write to the same disks. If any VM cannot be stopped,
345+
# the fence falls back to hard-reboot.
346+
#
347+
# The non-default values are recommended for setups using LINSTOR/DRBD or any local
348+
# storage with replication, where transient I/O contention can cause a heartbeat
349+
# write to time out without the host actually being unhealthy.
350+
#kvm.heartbeat.fence.action=hard-reboot
351+
352+
# Set to true to let restart-agent and log-only leave the VMs running. This is only
353+
# safe when no VM on this host can be started on another host by HA (VM HA off for all
354+
# of them and host HA not enabled). Otherwise a host the management server declares
355+
# Down can end up with the same VM running twice and corrupting its disks.
356+
# Has no effect on hard-reboot and graceful-reboot.
357+
# This is the master opt-out: when true, nothing is stopped and
358+
# kvm.heartbeat.fence.include.system.vms is ignored (the syslog warning says so).
359+
#kvm.heartbeat.fence.keep.vms.running=false
360+
361+
# Whether restart-agent and log-only also stop system VMs (virtual routers, console
362+
# proxy, secondary storage and internal LB VMs, recognised by their r-, v-, s- and b-
363+
# instance names). With true (default) every VM on the host is stopped. With false
364+
# only user VMs are stopped; system VMs keep running and a system VM that is still
365+
# running does not trigger the hard-reboot fallback.
366+
# Risk of false: if the management server declares this host Down it restarts the
367+
# system VMs on another host, so a virtual router could run twice with the same IP
368+
# and MAC addresses on the guest network, and a second console proxy or secondary
369+
# storage VM could run with the same addresses. Only set false if you accept that.
370+
# Ignored when kvm.heartbeat.fence.keep.vms.running is true. Has no effect on
371+
# hard-reboot and graceful-reboot.
372+
#kvm.heartbeat.fence.include.system.vms=true
373+
374+
# Seconds restart-agent and log-only may spend stopping VMs before the host is
375+
# hard-rebooted instead. Before stopping any VM the fence arms a detached deadman that
376+
# keeps running even if the agent kills the heartbeat script when
377+
# heartbeat.update.timeout runs out (stopping a VM can block for a long time on hung
378+
# storage). If the VMs are not all confirmed stopped within this budget, the deadman
379+
# hard-reboots the host. Each virsh call is also limited to 10 seconds (or this value
380+
# if lower) when coreutils timeout is installed. Keep it below heartbeat.update.timeout
381+
# (60 seconds by default). A value that is not a positive integer means 30.
382+
# Has no effect on hard-reboot and graceful-reboot, or when
383+
# kvm.heartbeat.fence.keep.vms.running is true.
384+
#kvm.heartbeat.fence.stop.vms.timeout=30
385+
386+
# Loop guard for restart-agent, in seconds. While storage keeps failing, the restarted
387+
# agent runs the heartbeat again, it fails again and restart-agent would restart the
388+
# agent again, over and over. Each restart-agent fence records its time under /run
389+
# (kept across agent restarts, cleared by a reboot); if restart-agent already fired less
390+
# than this many seconds ago, the fence hard-reboots the host instead. 0 turns the guard
391+
# off. A value that is not a whole number of seconds means 600.
392+
# Has no effect on hard-reboot, graceful-reboot and log-only.
393+
#kvm.heartbeat.fence.restart.agent.cooldown=600
394+
321395
# Enables manually setting CPU's topology on KVM's VM.
322396
#enable.manually.setting.cpu.topology.on.kvm.vm=true
323397

‎agent/src/main/java/com/cloud/agent/properties/AgentProperties.java‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -622,6 +622,96 @@ public class AgentProperties{
622622
public static final Property<Boolean> REBOOT_HOST_AND_ALERT_MANAGEMENT_ON_HEARTBEAT_TIMEOUT
623623
= new Property<>("reboot.host.and.alert.management.on.heartbeat.timeout", false);
624624

625+
/**
626+
* Action taken by the KVM agent's storage heartbeat scripts (kvmheartbeat.sh, kvmspheartbeat.sh
627+
* and kvmsmpheartbeat.sh) when a heartbeat write fails persistently. Only consulted when
628+
* {@link #REBOOT_HOST_AND_ALERT_MANAGEMENT_ON_HEARTBEAT_TIMEOUT} is true: that boolean remains the
629+
* on/off switch for fencing, this setting only chooses what the fence does. Allowed values:
630+
* <ul>
631+
* <li>{@code hard-reboot} (default): immediate sysrq-trigger reboot. Required default for setups
632+
* where a stale NFSv3 mount can prevent a graceful shutdown from completing.</li>
633+
* <li>{@code graceful-reboot}: {@code systemctl reboot} instead of sysrq; allows VMs
634+
* to stop cleanly. Use only if a stale storage mount cannot block shutdown.</li>
635+
* <li>{@code restart-agent}: stop every VM on the host, then restart cloudstack-agent; the host stays up. If it
636+
* already fired within {@link #KVM_HEARTBEAT_FENCE_RESTART_AGENT_COOLDOWN}, hard-reboot instead.</li>
637+
* <li>{@code log-only}: stop every VM on the host, then only write a message to the host's syslog (no
638+
* reboot, no agent restart, no management server alert); the operator must investigate.</li>
639+
* </ul>
640+
* Any other value falls back to {@code hard-reboot}. {@code restart-agent} and {@code graceful-reboot} need
641+
* systemctl; if it is missing or the systemctl call fails, the fence hard-reboots.<br>
642+
* {@code restart-agent} and {@code log-only} stop the VMs because, once the heartbeat is stale, the management
643+
* server may declare the host Down and HA may start the same VMs on another host, where both copies would write
644+
* to the same disks. If any VM cannot be stopped the fence falls back to {@code hard-reboot}. See
645+
* {@link #KVM_HEARTBEAT_FENCE_KEEP_VMS_RUNNING}.<br>
646+
* The non-default values are recommended for setups using LINSTOR/DRBD or other replicated
647+
* local storage, where transient I/O contention can cause a heartbeat write to time out
648+
* without the host actually being unhealthy.<br>
649+
* Read by the heartbeat shell scripts directly from agent.properties.<br>
650+
* Data type: String.<br>
651+
* Default value: {@code hard-reboot}
652+
*/
653+
public static final Property<String> KVM_HEARTBEAT_FENCE_ACTION
654+
= new Property<>("kvm.heartbeat.fence.action", "hard-reboot");
655+
656+
/**
657+
* When true, the {@code restart-agent} and {@code log-only} fence actions leave the host's VMs running instead of
658+
* stopping them first. Only safe when no VM on the host can be started on another host by HA (VM HA off for all
659+
* of them and host HA not enabled); otherwise a host the management server declares Down can end up with the
660+
* same VM running twice, writing to the same disks. No effect on {@code hard-reboot} and {@code graceful-reboot}.
661+
* This is the master opt-out: when true nothing is stopped and {@link #KVM_HEARTBEAT_FENCE_INCLUDE_SYSTEM_VMS}
662+
* is ignored.<br>
663+
* Read by the heartbeat shell scripts directly from agent.properties.<br>
664+
* Data type: Boolean.<br>
665+
* Default value: <code>false</code>
666+
*/
667+
public static final Property<Boolean> KVM_HEARTBEAT_FENCE_KEEP_VMS_RUNNING
668+
= new Property<>("kvm.heartbeat.fence.keep.vms.running", false);
669+
670+
/**
671+
* Whether the {@code restart-agent} and {@code log-only} fence actions also stop system VMs (virtual routers,
672+
* console proxy, secondary storage and internal LB VMs, recognised by their {@code r-}, {@code v-}, {@code s-} and
673+
* {@code b-} instance names). When true every VM on the host is stopped. When false only user VMs are stopped;
674+
* system VMs keep running and do not trigger the {@code hard-reboot} fallback. Risk of false: if the management
675+
* server declares the host Down it restarts the system VMs on another host, so a virtual router could run twice
676+
* with the same IP and MAC addresses. Ignored when {@link #KVM_HEARTBEAT_FENCE_KEEP_VMS_RUNNING} is true. No
677+
* effect on {@code hard-reboot} and {@code graceful-reboot}.
678+
* Read by the heartbeat shell scripts directly from agent.properties.<br>
679+
* Data type: Boolean.<br>
680+
* Default value: <code>true</code>
681+
*/
682+
public static final Property<Boolean> KVM_HEARTBEAT_FENCE_INCLUDE_SYSTEM_VMS
683+
= new Property<>("kvm.heartbeat.fence.include.system.vms", true);
684+
685+
/**
686+
* Seconds the {@code restart-agent} and {@code log-only} fence actions may spend stopping VMs before the host is
687+
* hard-rebooted instead. Before stopping any VM the fence arms a detached deadman that keeps running even if the
688+
* agent kills the heartbeat script when {@link #HEARTBEAT_UPDATE_TIMEOUT} runs out (stopping a VM can block for a
689+
* long time on hung storage). If the VMs are not all confirmed stopped within this budget, the deadman
690+
* hard-reboots the host. Each virsh call is also limited to 10 seconds, or this value if lower, when coreutils
691+
* {@code timeout} is installed. Keep it below {@link #HEARTBEAT_UPDATE_TIMEOUT}. A value that is not a positive
692+
* integer falls back to the default. No effect on {@code hard-reboot} and {@code graceful-reboot}, or when
693+
* {@link #KVM_HEARTBEAT_FENCE_KEEP_VMS_RUNNING} is true.<br>
694+
* Read by the heartbeat shell scripts directly from agent.properties.<br>
695+
* Data type: Integer.<br>
696+
* Default value: <code>30</code>
697+
*/
698+
public static final Property<Integer> KVM_HEARTBEAT_FENCE_STOP_VMS_TIMEOUT
699+
= new Property<>("kvm.heartbeat.fence.stop.vms.timeout", 30);
700+
701+
/**
702+
* Loop guard for the {@code restart-agent} fence action, in seconds. While storage keeps failing, the restarted
703+
* agent runs the heartbeat again, it fails again and {@code restart-agent} would restart the agent again, over and
704+
* over. Each {@code restart-agent} fence records its time under /run (kept across agent restarts, cleared by a
705+
* reboot); if {@code restart-agent} already fired less than this many seconds ago, the fence hard-reboots the host
706+
* instead. 0 turns the guard off. A value that is not a whole number of seconds falls back to the default. No
707+
* effect on {@code hard-reboot}, {@code graceful-reboot} and {@code log-only}.<br>
708+
* Read by the heartbeat shell scripts directly from agent.properties.<br>
709+
* Data type: Integer.<br>
710+
* Default value: <code>600</code>
711+
*/
712+
public static final Property<Integer> KVM_HEARTBEAT_FENCE_RESTART_AGENT_COOLDOWN
713+
= new Property<>("kvm.heartbeat.fence.restart.agent.cooldown", 600);
714+
625715
/**
626716
* Enables manually setting CPU's topology on KVM's VM. <br>
627717
* Data type: Boolean.<br>

0 commit comments

Comments
 (0)