Skip to content

Linstor shared storagepool support - #13910

Open
rp- wants to merge 16 commits into
apache:mainfrom
LINBIT:linstor-4.22-shared-storagepool-support
Open

rp- wants to merge 16 commits into
apache:mainfrom
LINBIT:linstor-4.22-shared-storagepool-support

Conversation

@rp-

@rp- rp- commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Description

This PR adds support for LINSTOR shared storage pools (thick LVM on a LUN that all
hypervisors can access, SAN-style) to the Linstor volume plugin.

Until now the plugin assumed every resource is DRBD replicated and thin provisioned. On a
shared storage pool the data exists once on the LUN, a resource is active on at most one
node at a time, there is no DRBD layer to make dual-primary, and the backing LVM is thick.
Functionally this changes:

  • Live migration uses the LINSTOR 1.29 API: make-available with
    auto_manage_dual_primary prepares the destination (DRBD dual-primary, or activating the
    resource on both nodes for shared pools) and unmake-available reverts it on the source.
    The controller REST API version is probed once per controller; against controllers older
    than 1.29 the previous manual allow-two-primaries handling is used unchanged, so
    existing DRBD/thin deployments behave exactly as before.
  • Resources of stopped VMs are INACTIVE on a shared pool and expose no device. They are
    now activated on demand (through the controller, so the shared-space lock is honored) for
    VM start, snapshot backup, snapshot revert and template cloning, and deactivated again
    afterwards.
  • Snapshots on thick LVM work: the snapshot path is resolved for thick pools, the
    snapshot LV is temporarily activated for the copy to secondary storage (without
    registering it with dmeventd, which could block the node wide LVM lock), and the copy is
    routed to the node the volume is active on, because dm-snapshot is not cluster aware.
  • Capacity reporting counts a shared space once instead of once per node (a 100 GiB LUN
    on three nodes was reported as 300 GiB), and thickly provisioned pools get a per pool
    storage.overprovisioning.factor of 1.0 at registration, since they cannot be over
    provisioned. Thin pools keep the current default.
  • Resources and clone targets are created with the layer stack configured on the resource
    group instead of the implicit default, which otherwise produced DRBD resources in a
    STORAGE-only resource group.

One change is outside the plugin: KVMStorageProcessor.copyVolumeFromPrimaryToSecondary()
now connects the source volume before reading it and disconnects it afterwards, mirroring
what the VM start and attach flows do. Migrating a detached volume between pools
otherwise reads a device that was never connected, which fails for any storage driver that
exposes devices on demand.

Requires java-linstor 0.8.0 (LINSTOR REST API 1.29.0, shipped with linstor-server 1.35).

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

n/a - no UI or API changes.

How Has This Been Tested?

Two 3-node KVM clusters (Ubuntu 24.04, CloudStack 4.22, one management server each), so
both storage models and both controller API versions are covered:

cluster primary storage LINSTOR REST API code path
A shared thick LVM: one 100 GiB LUN attached to all hosts, LVM VG with a LINSTOR shared space, resource group STORAGE layer only, place-count 1 1.29.0 (linstor-server 1.35) new make-available/unmake-available
B thin LVM (lvmthin) + DRBD, 2 replicas 1.28.0 version fallback to the previous handling

Marvin, test/integration/plugins/linstor:

  • test_linstor_volumes.py (15 tests): 15/15 pass on both clusters - includes volume
    attach/detach/reboot, snapshot create, template from snapshot, volume migration to the
    same and to a distinct pool, and VM snapshot create/revert/delete.
  • test_linstor_encrypted_snapshots.py (3 tests): 3/3 pass on cluster A
    (revert of an encrypted root snapshot, create-volume-from-encrypted-snapshot is rejected,
    and the qcow2 on secondary storage is verified to be LUKS encrypted at rest). On cluster B
    the third test skips itself because it needs DB access from the test runner to locate the
    backed-up file.
  • 9 plugin unit tests, incl. new ones for the shared-space capacity deduplication, thick
    provider-kind detection, diskful copy ordering and the controller version gate.

Manually verified on cluster A, checking the LINSTOR side after each step: VM deploy from
template, stop/start of a VM whose resource is INACTIVE, live migration (controller log
shows make-available on the destination and unmake-available on the source; the resource
ends up active on the destination only), snapshot of a running and of a stopped VM,
template from snapshot, volume migration between pools, and capacity (the 100 GiB LUN is
now reported as 100 GB instead of 599.98 GB).

How did you try to break this feature and the system with this change?

  • Old controller: ran the full suite against a 1.28.0 controller. This surfaced that the
    regenerated client serialized auto_manage_dual_primary on every make-available (the
    generated field is initialized with its schema default), and LINSTOR rejects unknown
    properties - so template copies and VM starts failed even though the feature is version
    gated. Fixed by only setting the flag for migrations (and, in java-linstor, by serializing
    only explicitly set properties). Full suite passes on 1.28.0 afterwards, i.e. no
    regression for existing deployments.
  • Cold start: rebooted all nodes, so every shared resource came back INACTIVE. That
    broke template cloning, VM start and snapshots, which all assumed an active resource;
    each path now activates on demand and was re-tested from a rebooted cluster.
  • Concurrency: parallel deploys from the same template - deactivating the template right
    after a clone raced other clones still using it, so the template is left active.
  • Host loss: a diskful copy whose host is down/disabled no longer forces a fallback to a
    temporary resource; all diskful copies are considered, in preference order.
  • Thin pools: verified capacity, over-provisioning and snapshot behavior are unchanged
    for lvmthin (dedup keys on the shared space, thick detection requires all pools of the
    resource group to be thick, mixed setups keep the current default).
  • Encrypted (LUKS) volumes on thick pools, snapshot on a stopped VM (resource inactive)
    and on a running VM, and repeated snapshot/delete cycles to check nothing is left behind
    in LINSTOR.
  • LVM edge cases: lvchange registering a snapshot LV with dmeventd could hang and
    block the node wide LVM lock, wedging all storage operations on that host - the temporary
    activation now uses --monitor n.

Testing this also uncovered several LINSTOR server side issues (satellite startup probe
racing on a shared VG, snapshot rollback with an inactive copy, an io-suspend hang on
encrypted devices, a clone that never left CLONING); those are fixed in linstor-server
1.35 and are not CloudStack changes.

@codecov

codecov Bot commented Aug 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 29.01235% with 230 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.82%. Comparing base (278bde2) to head (57ef99a).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
.../hypervisor/kvm/storage/LinstorStorageAdaptor.java 28.44% 74 Missing and 4 partials ⚠️
...cloudstack/storage/datastore/util/LinstorUtil.java 45.65% 68 Missing and 7 partials ⚠️
...tore/driver/LinstorPrimaryDataStoreDriverImpl.java 0.00% 48 Missing ⚠️
...e/wrapper/LinstorBackupSnapshotCommandWrapper.java 0.00% 18 Missing ⚠️
...ud/hypervisor/kvm/storage/KVMStorageProcessor.java 0.00% 5 Missing ⚠️
...ifecycle/LinstorPrimaryDataStoreLifeCycleImpl.java 0.00% 5 Missing ⚠️
...ck/storage/snapshot/LinstorVMSnapshotStrategy.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #13910      +/-   ##
============================================
+ Coverage     20.78%   20.82%   +0.04%     
- Complexity    20857    20962     +105     
============================================
  Files          6053     6053              
  Lines        548230   548694     +464     
  Branches      73237    73329      +92     
============================================
+ Hits         113928   114272     +344     
- Misses       420955   421043      +88     
- Partials      13347    13379      +32     
Flag Coverage Δ
uitests 3.86% <ø> (-0.01%) ⬇️
unittests 22.26% <29.01%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@DaanHoogland DaanHoogland added this to the 4.22.2 milestone Aug 18, 2026
@rp-
rp- force-pushed the linstor-4.22-shared-storagepool-support branch from 1759241 to 4150e40 Compare August 28, 2026 07:58
@DaanHoogland DaanHoogland moved this from Backlog to Ready in CloudStack Testing Aug 31, 2026
@rp-
rp- force-pushed the linstor-4.22-shared-storagepool-support branch from 4150e40 to 34a932d Compare September 7, 2026 09:26
@github-actions

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@weizhouapache

Copy link
Copy Markdown
Member

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 19419

@rp-
rp- force-pushed the linstor-4.22-shared-storagepool-support branch from 34a932d to c560105 Compare October 21, 2026 11:01
@rp- rp- modified the milestones: 4.22.2, 4.22.3 Oct 23, 2026
@github-actions

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@rp-
rp- force-pushed the linstor-4.22-shared-storagepool-support branch from c560105 to 246da04 Compare October 23, 2026 10:04
@github-actions

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@rp- rp- modified the milestones: 4.22.3, 24.0 Nov 16, 2026
@DaanHoogland
DaanHoogland requested review from DaanHoogland, slavkap and wido and removed request for DaanHoogland November 19, 2026 11:20

@wido wido left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this only hits Linstor customers and not existing other functionality I see no objections.

rp- added 13 commits November 20, 2026 16:35
Client for LINSTOR REST API 1.30.0. Since REST API 1.29 (shipped with
linstor-server 1.35.0) make-available has the auto_manage_dual_primary
option, there is a new unmake-available endpoint, resourceSnapshotDelete
grew a delete_empty_resource_definition parameter and the
resource-definition clone request accepts volume_sizes. Adapt the snapshot
delete call sites to the new signature (passing null keeps the previous
behavior).
LINSTOR REST API 1.29.0 moves the live-migration preparation into the
controller: make-available with auto_manage_dual_primary sets DRBD
allow-two-primaries (and protocol C) between the migration source and
destination, or activates shared storage pool resources on both nodes;
unmake-available reverts this on the source node and deletes diskless or
redundant shared copies while keeping tiebreaker and diskful resources.

connectPhysicalDisk now issues a single make-available with
auto_manage_dual_primary set for migrations, and tryDisconnectLinstor a
single unmake-available. The controller REST API version is queried per
operation (one GET against the controller); against controllers older than
1.29 the previous manual dual-primary, activate and delete handling is used
unchanged.
- getSnapshotPath: thick LVM snapshot LVs use the same naming scheme as
  thin ones
- backup: thick LVM snapshot LVs of inactive origins are inactive and
  carry the skip-activation flag; activate them (lvchange -ay -K) for the
  copy to secondary storage and deactivate them again afterwards
Shared storage pool resources of stopped VMs are INACTIVE and expose no
device on any node. Activate them through the Linstor controller (which
takes the shared-space activation lock) before sending the backup or
revert copy command to the host and deactivate them again afterwards.
The agent itself must only ever activate the snapshot LV, never the
resource, to not bypass the shared-space locking.
The KVM start flow builds the domain XML (createVbd) before
connectPhysicalDisk runs make-available, so getDevicePath must not fail
for resources that are inactive on every node - the start would spin on
the missing device path and never reach the activating connect. Return
the deterministic storage layer path instead; make-available activates
the resource before the domain is started.

Also add the plugin CHANGELOG entries for shared storage pool support.
Activating a thick LVM snapshot LV for the backup copy registers it with
dmeventd by default; that registration can block lvchange indefinitely
(observed stuck in pipe_read waiting for a dmeventd reply), and a blocked
lvm command holds the node-wide LVM lock. Pass --monitor n on the
temporary activation and deactivation: monitoring only exists to
auto-extend snapshots, and Linstor creates thick snapshot LVs at full
origin size, so they can never overflow. Also raise the command timeout.
Shared storage pool templates are INACTIVE while no clone is running and
Linstor's clone source selection skips inactive resources, failing with
'No suitable storage pools found for cloning'. Activate a diskful
template resource for the duration of the clone and deactivate it again
afterwards.
copyVolumeFromPrimaryToSecondary read the source device without
connecting it first. Detached volumes being migrated between pools are
not connected anywhere, and storage drivers that expose devices on
demand (e.g. Linstor shared storage pools, where resources of stopped
VMs are inactive without a device path) fail the copy. Connect the
source volume for the duration of the copy, mirroring what VM start and
attach flows do.
Clone requests without an explicit layer list can produce a target with
a default DRBD layer stack while the source on a shared storage pool is
STORAGE only, failing with 'no common clone strategy found'. Set the
resource group's configured layer stack on the clone request for
unencrypted volumes, mirroring what is already done for encrypted ones.
Rely on the explicit layer stack instead of the resource group default
when spawning unencrypted resources. A template spawned with a DRBD
stack cannot be used as a clone source for STORAGE or LUKS,STORAGE
resources of a shared storage pool later on (the clone either fails with
'no common clone strategy found' or hangs in state CLONING).
Every node accessing a shared space (e.g. a LUN with a shared LVM volume
group) reports the full capacity of that space, so summing all storage
pools of the resource group multiplied the reported capacity by the
number of nodes: a 100 GiB shared LUN on three nodes was reported as
300 GiB (and 600 GiB after the over-provisioning factor). Deduplicate
storage pools by their shared space when calculating total, used and
free capacity.
Thickly provisioned backends (thick LVM, ZFS with reservation, file)
allocate the full volume size, so the pool capacity is a hard limit and
the storage.overprovisioning.factor default of 2 makes CloudStack
allocate twice the space that exists. The Linstor storage pool type has
to allow over-provisioning for the thin backends, so set the factor to
1.0 per pool when all storage pools of the resource group are thick.
Mixed or unknown setups keep the default.
Linstor controllers older than 1.29 reject requests with unknown
properties, so a make-available carrying auto_manage_dual_primary fails
against them - which broke every make-available call (and with it
template copies and VM starts) even though the live-migrate API is
version gated: the generated client initializes fields with their schema
default (false here) and serialized everything non-null, so the property
was always sent.
@weizhouapache

Copy link
Copy Markdown
Member

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@rp-

rp- commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

@weizhouapache don't start tests yet, I'll address one of the points Damans raised.
Currently testing.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 20109

supportsLiveMigrateApi() treated a failed version query like an old
controller, so a single failed query silently dropped that call back to
the previous allow-two-primaries handling - a disk connected through
make-available could then be disconnected the old way.

Let the query fail instead: connectPhysicalDisk fails like on any other
controller error, and a disconnect leaves the resource untouched. The
previous handling is now only used for a controller that answers with a
REST API version older than 1.29.
@rp-
rp- force-pushed the linstor-4.22-shared-storagepool-support branch from b5305a0 to 9975085 Compare November 20, 2026 20:25
getDevicePath() returned /dev/<vg>/<rsc>_00000 when no node reported a
device path, i.e. for a shared storage pool resource that is inactive on
all its nodes. That is the storage layer device: for an encrypted volume
it is the LUKS ciphertext below the device the VM has to use.

Drop the fallback, getDevicePath() only returns what Linstor reports and
callers have to make sure the resource is active. All management server
callers ask right after creating or making the resource available. The
agent's getPhysicalDisk() is asked for the disk before connectPhysicalDisk
runs when a VM starts, so it makes a resource that is inactive everywhere
available on its node first; Linstor then reports the right device for
any layer stack.

DRBD resources keep their by-res device whether or not a path is reported,
as before the shared storage pool support: they are never inactive, and
the controller does not know the reported path until the satellites have
reconnected after a controller restart.
@rp-

rp- commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@rp- a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 20110

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

6 participants