systemvm-template: support on-demand download during setup and registration #11656

shwstppr · 2025-09-17T10:54:54Z

Description

Bundling all hypervisor SystemVM templates in release packages simplifies installs but inflates build time and pacakge size. This change enables downloading templates on demand when they’re not found after package installation. The download path is wired into both cloud-setup-management and the existing SystemVM template registration flow.
cloudstack-setup-management allows the argument --systemvm-templates to download the desired hypervisor system template during management server setup.

For air-gapped environments, a repository URL prefix can be provided to support air-gapped setups: pass --systemvm-templates-repository <URL-prefix> to cloudstack-setup-management, or set system.vm.templates.download.repository=<URL-prefix> in server.properties for post-setup registration. Currently, custom repository should mirror the file structure of the official repository, ie, <BASE_VERSION>/systemvmtemplate-<VERSION>-<ARCH>-<HYPERVISOR>.<FILE_EXTENSION>
If templates are already present (bundled or preseeded), behavior is unchanged and no download is attempted.

Documentation PR: apache/cloudstack-documentation#580

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2025-09-17T11:04:22Z

Codecov Report

❌ Patch coverage is 62.50000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.57%. Comparing base (bfc4f60) to head (29cd184).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...om/cloud/upgrade/SystemVmTemplateRegistration.java	40.00%	4 Missing and 2 partials ⚠️
.../cloudstack/utils/server/ServerPropertiesUtil.java	72.72%	4 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #11656   +/-   ##
=========================================
  Coverage     17.56%   17.57%           
- Complexity    15540    15546    +6     
=========================================
  Files          5909     5910    +1     
  Lines        529045   529088   +43     
  Branches      64615    64623    +8     
=========================================
+ Hits          92933    92962   +29     
- Misses       425657   425668   +11     
- Partials      10455    10458    +3

Flag	Coverage Δ
uitests	`3.58% <ø> (ø)`
unittests	`18.63% <62.50%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…ration Bundling all hypervisor SystemVM templates in release packages simplifies installs but inflates build time and artifact size. This change enables downloading templates on demand when they’re not found after package installation. The download path is wired into both cloud-setup-management and the existing SystemVM template registration flow. For connected or mirrored environments, a repository URL prefix can be provided to support air-gapped setups: pass --systemvm-templates-repository <URL-prefix> to cloud-setup-management, or set system.vm.templates.download.repository=<URL-prefix> in server.properties for post-setup registration. If templates are already present (bundled or preseeded), behavior is unchanged and no download is attempted. Signed-off-by: Abhishek Kumar <[email protected]>

github-actions · 2025-09-27T03:35:04Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

shwstppr · 2025-09-29T06:24:10Z

@blueorangutan package

blueorangutan · 2025-09-29T06:26:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-29T08:11:50Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15214

Signed-off-by: Abhishek Kumar <[email protected]>

Copilot

Pull Request Overview

This PR enhances CloudStack's SystemVM template handling by enabling on-demand downloading of templates during setup and registration, reducing package size and build time. The implementation supports both regular and air-gapped environments through configurable repository URLs.

Adds on-demand SystemVM template downloading capability with checksum verification
Refactors template metadata handling to support custom download repositories
Implements centralized server properties utility for configuration management

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
utils/src/main/java/org/apache/cloudstack/utils/server/ServerPropertiesUtil.java	New utility class for centralized server.properties access with thread-safe caching
plugins/hypervisors/external/src/main/java/org/apache/cloudstack/hypervisor/external/provisioner/ExternalPathPayloadProvisioner.java	Refactored to use new ServerPropertiesUtil instead of duplicated property loading logic
engine/schema/templateConfig.sh	Enhanced template metadata generation with repository URL support and refactored template specification structure
engine/schema/src/test/java/com/cloud/upgrade/SystemVmTemplateRegistrationTest.java	Updated test case to use VMware/arm64 combination instead of KVM/amd64
engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java	Added support for custom download repository configuration and URL replacement logic
client/conf/server.properties.in	Added configuration property for custom SystemVM template repository URL
client/bindir/cloud-setup-management.in	Implemented comprehensive template download functionality with metadata parsing and checksum verification

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

client/bindir/cloud-setup-management.in

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr · 2025-10-13T09:02:26Z

@blueorangutan package

blueorangutan · 2025-10-13T09:04:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Related apache/cloudstack#11656 Signed-off-by: Abhishek Kumar <[email protected]>

blueorangutan · 2025-10-13T11:28:21Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15431

shwstppr · 2025-10-27T07:40:26Z

@blueorangutan newpackage

blueorangutan · 2025-10-27T07:42:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Signed-off-by: Abhishek Kumar <[email protected]>

blueorangutan · 2025-10-27T08:39:02Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 15568

shwstppr · 2025-10-27T10:20:20Z

@blueorangutan newpackage

blueorangutan · 2025-10-27T10:22:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with no systemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-10-27T11:17:44Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15571

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr · 2025-10-28T09:20:31Z

@blueorangutan newpackage

blueorangutan · 2025-10-28T09:22:04Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with no systemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-10-28T10:20:18Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15576

shwstppr · 2025-10-28T11:01:07Z

@blueorangutan test

blueorangutan · 2025-10-28T11:02:04Z

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-10-29T06:12:00Z

[SF] Trillian test result (tid-14736)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 64859 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11656-t14736-kvm-ol8.zip
Smoke tests completed. 145 look OK, 2 have errors, 2 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestClusterDRS>:setup	`Error`	0.00	test_cluster_drs.py
test_01_redundant_vpc_site2site_vpn	`Failure`	501.30	test_vpc_vpn.py
all_test_human_readable_logs	`Skipped`	---	test_human_readable_logs.py
all_test_image_store_object_migration	`Skipped`	---	test_image_store_object_migration.py

shwstppr · 2025-10-30T07:04:53Z

@blueorangutan test keepEnv

blueorangutan · 2025-10-30T07:06:03Z

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-10-31T03:52:01Z

[SF] Trillian test result (tid-14746)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 70380 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11656-t14746-kvm-ol8.zip
Smoke tests completed. 146 look OK, 1 have errors, 2 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_redundant_vpc_site2site_vpn	`Failure`	456.19	test_vpc_vpn.py
all_test_human_readable_logs	`Skipped`	---	test_human_readable_logs.py
all_test_image_store_object_migration	`Skipped`	---	test_image_store_object_migration.py

shwstppr force-pushed the disjoint-systemvmtmpl branch from 5c41125 to df47ea1 Compare September 17, 2025 10:55

shwstppr force-pushed the disjoint-systemvmtmpl branch from df47ea1 to efdea06 Compare September 17, 2025 14:49

github-actions bot added the status:has-conflicts label Sep 27, 2025

Merge remote-tracking branch 'apache/main' into disjoint-systemvmtmpl

1a9ce76

github-actions bot removed the status:has-conflicts label Sep 29, 2025

refactor

9649e71

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr requested a review from Copilot October 9, 2025 10:35

Copilot AI reviewed Oct 9, 2025

View reviewed changes

client/bindir/cloud-setup-management.in Outdated Show resolved Hide resolved

shwstppr added 2 commits October 13, 2025 12:56

Merge branch 'main' into disjoint-systemvmtmpl

41b3658

add tests, fix comment

94417df

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr added a commit to shapeblue/cloudstack-documentation that referenced this pull request Oct 13, 2025

notes on system template auto-download, seeding

55cff9f

Related apache/cloudstack#11656 Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr mentioned this pull request Oct 13, 2025

notes on system template auto-download, seeding apache/cloudstack-documentation#580

Draft

shwstppr added 2 commits October 27, 2025 13:21

Merge remote-tracking branch 'apache/main' into disjoint-systemvmtmpl

0833a64

dont use fixed version

2fd18bd

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr mentioned this pull request Oct 27, 2025

systemvm template download change shapeblue/Trillian#212

Draft

shwstppr closed this Oct 27, 2025

shwstppr reopened this Oct 27, 2025

minor refactor

29cd184

Signed-off-by: Abhishek Kumar <[email protected]>

DaanHoogland added this to the 4.23 milestone Oct 28, 2025

systemvm-template: support on-demand download during setup and registration #11656

Are you sure you want to change the base?

systemvm-template: support on-demand download during setup and registration #11656

Uh oh!

Conversation

shwstppr commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Sep 27, 2025

Uh oh!

shwstppr commented Sep 29, 2025

Uh oh!

blueorangutan commented Sep 29, 2025

Uh oh!

blueorangutan commented Sep 29, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

shwstppr commented Oct 13, 2025

Uh oh!

blueorangutan commented Oct 13, 2025

Uh oh!

blueorangutan commented Oct 13, 2025

Uh oh!

shwstppr commented Oct 27, 2025

Uh oh!

blueorangutan commented Oct 27, 2025

Uh oh!

blueorangutan commented Oct 27, 2025

Uh oh!

shwstppr commented Oct 27, 2025

Uh oh!

blueorangutan commented Oct 27, 2025

Uh oh!

blueorangutan commented Oct 27, 2025

Uh oh!

shwstppr commented Oct 28, 2025

Uh oh!

blueorangutan commented Oct 28, 2025

Uh oh!

blueorangutan commented Oct 28, 2025

Uh oh!

shwstppr commented Oct 28, 2025

Uh oh!

blueorangutan commented Oct 28, 2025

Uh oh!

blueorangutan commented Oct 29, 2025

Uh oh!

shwstppr commented Oct 30, 2025

Uh oh!

blueorangutan commented Oct 30, 2025

Uh oh!

blueorangutan commented Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shwstppr commented Sep 17, 2025 •

edited

Loading

codecov bot commented Sep 17, 2025 •

edited

Loading