Repository navigation
Commit cc30155
authored
KVM: make storage heartbeat fence action configurable (graceful-reboot / restart-agent / log-only) (#13090)
* KVM: make storage heartbeat fence action configurable
The KVM agent's storage heartbeat scripts (kvmheartbeat.sh and
kvmspheartbeat.sh) hard-code an immediate kernel-level reboot via
'echo b > /proc/sysrq-trigger' when a heartbeat write to primary storage
times out. This bypasses all OS-level shutdown protections, drops every
running VM on the host instantly, and triggers HA cascades onto
surviving hosts.
For NFS shared storage the binary "heartbeat-write-failed = host-is-dead"
heuristic is reasonable. For LINSTOR/DRBD or other replicated local
storage, the same disk serves application I/O, replication I/O and
heartbeat I/O simultaneously - so a transient I/O contention spike can
time out the heartbeat write without the host actually being unhealthy.
The result is false-positive sysrq fencing.
Adds a new agent.properties option:
kvm.heartbeat.fence.action = reboot | graceful-reboot
| restart-agent | log-only
Default value is "reboot" so existing deployments keep their current
behavior. Operators on replicated storage backends can choose a less
destructive action:
- graceful-reboot: 'systemctl reboot' instead of sysrq, allowing VMs
a chance to shut down cleanly
- restart-agent: restart cloudstack-agent only, preserving running VMs
- log-only: log + alert, no automatic action
The existing 'reboot.host.and.alert.management.on.heartbeat.timeout'
boolean continues to function as a complete Java-side bypass.
Refs: #13089
* kvm-heartbeat: extract shared fence helper; add 'custom' action; rename to hard-reboot
Per review on PR #13090:
- Refactor common fence-action case into kvmha-fence.sh sourced by both
kvmheartbeat.sh and kvmspheartbeat.sh (per @sureshanaparti).
- Add 'custom' fence action that invokes an operator-supplied script
(kvm.heartbeat.fence.custom.script, default /etc/cloudstack/agent/
heartbeat-fence-custom.sh) with the heartbeat script name as arg;
falls back to hard-reboot if the script is missing/non-executable
(per @NuxRo).
- Rename canonical action 'reboot' -> 'hard-reboot' for clarity; keep
'reboot' accepted as alias so existing deployments don't break (per
@DaanHoogland). Default behavior unchanged: sysrq-trigger reboot,
required where a stale NFSv3 mount blocks systemctl reboot.
* kvmsmpheartbeat.sh: honour kvm.heartbeat.fence.action
Per @weizhouapache's review on #13090, kvmsmpheartbeat.sh was still
hard-coding the immediate-reboot fence action (echo b > /proc/sysrq-trigger)
while kvmheartbeat.sh and kvmspheartbeat.sh delegate to kvmha-fence.sh.
Source kvmha-fence.sh and call fence_action so this script picks up the
same hard-reboot / graceful-reboot / restart-agent / log-only / custom
behaviour from agent.properties. Default behaviour (no property set)
remains hard-reboot via sysrq-trigger, so existing deployments are
unaffected.
* address review (#13090): drop the 'reboot' alias, simplify the agent.properties lookup, clarify precedence
- kvm.heartbeat.fence.action accepts hard-reboot (default), graceful-reboot,
restart-agent, log-only and custom; the misleading 'reboot' alias is gone.
- kvmha-fence.sh reads the property with a plain grep on the key=value form.
- agent.properties comment states that reboot.host.and.alert.management.on.heartbeat.timeout
stays the global on/off switch and this setting only chooses the action.
Signed-off-by: James Peru <jmsperu@gmail.com>
* address review (#13090): log-only only writes to syslog, drop the custom action
- log-only is now described everywhere (agent.properties, AgentProperties
Javadoc, kvmha-fence.sh) as writing a message to the host's syslog and
taking no action: no reboot, no agent restart and no management server
alert. The "log + alert" wording is gone.
- The custom action and kvm.heartbeat.fence.custom.script are removed
(agent.properties, AgentProperties.KVM_HEARTBEAT_FENCE_CUSTOM_SCRIPT,
the custom branch and CUSTOM_SCRIPT handling in kvmha-fence.sh, the
kvmsmpheartbeat.sh comment). Unknown values, including custom, still
fall back to hard-reboot.
- The AgentProperties Javadoc documents only hard-reboot (no reboot alias),
names kvmsmpheartbeat.sh too, and states the precedence with
reboot.host.and.alert.management.on.heartbeat.timeout.
Claude-Session: https://claude.ai/code/session_01BhiE1Abhs315m3n3UPy6V7
* address review (#13090): restart-agent and log-only stop the host's VMs first
With restart-agent or log-only the VMs kept running after the heartbeat
went stale, so if the management server declared the host Down, HA could
start the same VMs on another host and two copies would write to the same
disks (raised by Damans227).
- kvmha-fence.sh: restart-agent and log-only now stop every active VM on
the host (virsh destroy) before acting; the host itself stays up. If
libvirt cannot be queried or any VM is still active afterwards, the
fence falls back to hard-reboot.
- New agent property kvm.heartbeat.fence.keep.vms.running (default false)
lets an operator keep the old behaviour; the fence then logs a WARNING
and agent.properties / AgentProperties say it is only safe when no VM
on the host can be restarted elsewhere by HA.
- hard-reboot and graceful-reboot are unchanged.
- The logger and sysrq trigger paths can be overridden through the
environment (FENCE_LOGGER, FENCE_SYSRQ_TRIGGER), like AGENT_PROPS, so
the new KvmHaFenceScriptTest runs the script with stand-ins for virsh,
systemctl, logger and sysrq and never touches the machine running it.
Agent module: 65 tests pass (6 new), bash -n clean on all heartbeat
scripts.
Claude-Session: https://claude.ai/code/session_01V82Ypx7SpAurn6ek1VXRAP
* address review (#13090): let the fence leave system VMs running
New agent property kvm.heartbeat.fence.include.system.vms (default true).
With true, restart-agent and log-only still stop every VM first. With
false they stop only user VMs; system VMs (r-, s-, v- and b- names, as
generated by VirtualMachineName) keep running and a running system VM no
longer forces the hard-reboot fallback. A syslog warning notes that a
virtual router could then run twice with the same IP and MAC.
kvm.heartbeat.fence.keep.vms.running=true stays the master opt-out: it
wins over include.system.vms and its warning says so.
Agent module: 69 tests pass (4 new in KvmHaFenceScriptTest).
Claude-Session: https://claude.ai/code/session_01V82Ypx7SpAurn6ek1VXRAP
* address review (#13090): bound the VM stop phase and guarantee the hard-reboot fallback
Before restart-agent or log-only stop any VM, the fence now arms a detached deadman
(own session via setsid, else nohup) that hard-reboots the host unless every selected VM
is confirmed stopped within kvm.heartbeat.fence.stop.vms.timeout seconds (default 30), so
the fallback still happens when the agent kills the script on heartbeat.update.timeout.
Each virsh call is also bounded by coreutils timeout (10 s, or the budget if lower).
Claude-Session: https://claude.ai/code/session_011DLKuGySDaq3JQs6SAw3TC
* address review (#13090): exact property parsing, safe fallbacks, restart-agent loop guard
- read_agent_property parses agent.properties the way java.util.Properties
does: the key is compared as a plain string (dots are no longer regex
wildcards), whitespace around '=' or ':' (or a whitespace separator) is
accepted, the value is trimmed, '#' and '!' lines are comments and the
last definition wins.
- kvmheartbeat.sh, kvmspheartbeat.sh and kvmsmpheartbeat.sh fall back to
the original sysrq hard-reboot, with an ERROR in syslog, if kvmha-fence.sh
cannot be sourced or does not define fence_action.
- restart-agent and graceful-reboot check for systemctl before acting and
hard-reboot with an ERROR if it is missing or the systemctl call fails.
- restart-agent records its time under /run (kept across agent restarts,
cleared by a reboot). A new restart-agent fence within
kvm.heartbeat.fence.restart.agent.cooldown seconds (default 600, 0 turns
the guard off) escalates to hard-reboot, so a host whose storage stays
down cannot loop on agent restarts. Documented in agent.properties and
AgentProperties.
- KvmHaFenceScriptTest: graceful-reboot (systemctl reboot, no VM stop by
the fence), failing and missing systemctl, whitespace/colon separators,
last definition wins, dotted key exact match, the cooldown guard, and the
heartbeat scripts' fallback when the helper is missing or empty.
Claude-Session: https://claude.ai/code/session_01EUU5skTaj95LfpkSbrWhPN
---------
Signed-off-by: James Peru <jmsperu@gmail.com>
Co-authored-by: jmsperu <jmsperu@users.noreply.github.com>1 parent 53a6c0c commit cc30155
7 files changed
Lines changed: 1065 additions & 15 deletions
File tree
- agent
- conf
- src
- main/java/com/cloud/agent/properties
- test/java/com/cloud/agent
- scripts/vm/hypervisor/kvm
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
318 | 318 | | |
319 | 319 | | |
320 | 320 | | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
321 | 395 | | |
322 | 396 | | |
323 | 397 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
622 | 622 | | |
623 | 623 | | |
624 | 624 | | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
625 | 715 | | |
626 | 716 | | |
627 | 717 | | |
| |||
0 commit comments