Add baremetal ABI support for NODES_PLATFORM=baremetal by fonta-rh · Pull Request #1922 · openshift-metal3/dev-scripts

fonta-rh · 2026-06-30T09:40:42Z

Summary

Add NODES_PLATFORM=baremetal support to the agent-based installation (make agent) pipeline for deploying on real baremetal nodes via a standalone provisioning host (no libvirt)
Bypass three libvirt-dependent code paths: host configuration (early return after platform-agnostic setup), MAC/IP discovery (get_baremetal_ips_and_macs() reads from NODES_FILE + config vars), and ISO delivery (mount_agent_iso_baremetal() via Redfish VirtualMedia)
Add dnsmasq DNS configuration from BAREMETAL_API_VIP/BAREMETAL_INGRESS_VIP config vars, and document all new BAREMETAL_* variables in config_example.sh
Add RHEL 10 / CentOS Stream 10 support to 01_install_requirements.sh (package availability fixes, metal3-dev-env compatibility patches)
Add BAREMETAL_MACS config var for multi-NIC servers where PXE boot MAC ≠ data network MAC

Context

This is part of OCPEDGE-2775 — enabling Two-Node with Fencing (TNF) deployment on real baremetal hardware. A companion tool (make baremetal-adopt from OCPEDGE-2774) generates the config_baremetal_fencing.sh that exports NODES_PLATFORM=baremetal, NODES_FILE, bridge overrides, and network vars.

Target topology: standalone provisioning host without libvirt (e.g., RDU2 HPE e920t nodes with iLO BMC via Redfish).

Validated with three end-to-end deploys on RDU2 baremetal (OCP 4.22.0-rc.5 and 4.22.4), both nodes Ready, all operators Available, full TNF pipeline (auth → setup → fencing → after-setup → update-setup) completed.

Changes

File	Change
`01_install_requirements.sh`	RHEL 10/CS10 support: distro case branch, CodeReady Builder repo, Go tarball arch fix for metal3-dev-env, remove unavailable packages (genisoimage → xorriso, python3-bcrypt)
`02_configure_host.sh`	Early return for baremetal — runs firewall, NTP, DNS, then `exit 0` (skips all libvirt VM/network/pool creation)
`network.sh`	Baremetal VIP branch in `set_api_and_ingress_vip()` — reads VIPs from config, calls existing `configure_dnsmasq()`
`agent/05_agent_configure.sh`	New `get_baremetal_ips_and_macs()` with `BAREMETAL_MACS` support for multi-NIC servers, conditional dispatch, guard `enable_isolated_baremetal_network` and `add_dns_entry` for baremetal
`agent/06_agent_create_cluster.sh`	New `mount_agent_iso_baremetal()` / `eject_agent_iso_baremetal()` via Redfish VirtualMedia (eject-before-insert, ISO staging for HTTP server), guard `get_node0_ip()`
`agent/common.sh`	Guard `getRendezvousIP()` — return first `BAREMETAL_IPS` instead of virsh
`config_example.sh`	Document `BAREMETAL_IPS`, `BAREMETAL_API_VIP`, `BAREMETAL_INGRESS_VIP`, `BAREMETAL_ISO_SERVER`, `BAREMETAL_MACS`

Design decisions

Early-return pattern for 02_configure_host.sh — cherry-picks only platform-agnostic operations, zero changes to existing libvirt code paths
User-provided BAREMETAL_ISO_SERVER — operator stages the ISO on an HTTP server reachable from BMCs; dev-scripts does not auto-start an HTTP server
Redfish VirtualMedia for ISO delivery — dynamically discovers CD slot from BMC's VirtualMedia collection to handle vendor differences (iLO, iDRAC, etc.)
BAREMETAL_MACS — optional override for data NIC MACs when they differ from PXE boot MACs (common on multi-NIC servers like HPE e920t where PXE=5C:BA:2C:*, data=b4:96:91:*)
RHEL 10 patches — metal3-dev-env pins genisoimage and python3-bcrypt which aren't available on RHEL 10; patched via conditional sed in the metal3-dev-env checkout

Test plan

shellcheck passes on all modified files (pre-existing warnings only)
Dry run on provisioning host: make agent with NODES_PLATFORM=baremetal config — verify early exit, dnsmasq, manifests
End-to-end on RDU2 HPE e920t baremetal cluster (OCP 4.22)
Regression: existing make agent with libvirt VMs still works (no changes to libvirt code paths)

🤖 Generated with Claude Code

openshift-ci · 2026-06-30T09:40:47Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign sadasu for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2026-06-30T09:40:54Z

Hi @fonta-rh. Thanks for your PR.

I'm waiting for a openshift-metal3 member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

dtantsur · 2026-07-06T15:41:58Z

/ok-to-test

fonta-rh · 2026-07-07T09:20:20Z

/retest-required

fonta-rh · 2026-07-07T13:58:12Z

/test e2e-metal-ipi-ovn-dualstack

fonta-rh · 2026-07-07T13:58:24Z

/test e2e-metal-ipi-ovn-ipv4

fonta-rh · 2026-07-07T13:58:28Z

/test e2e-metal-ipi-serial-ipv4-1of2

fonta-rh · 2026-07-07T13:58:33Z

/test e2e-metal-ipi-serial-ipv4-2of2

Enable the agent-based installation (make agent) pipeline to deploy on real baremetal nodes via a standalone provisioning host without libvirt. Three libvirt-dependent code paths are bypassed when NODES_PLATFORM=baremetal: - 02_configure_host.sh: early return after platform-agnostic setup (firewall, NTP, DNS, registry auth) — skips all VM/network creation - agent/05_agent_configure.sh: new get_baremetal_ips_and_macs() reads MACs from NODES_FILE and IPs from BAREMETAL_IPS instead of virsh - agent/06_agent_create_cluster.sh: new mount/eject_agent_iso_baremetal() delivers ISO via Redfish VirtualMedia instead of virt-xml Also adds a baremetal VIP branch in network.sh to configure dnsmasq DNS from BAREMETAL_API_VIP/BAREMETAL_INGRESS_VIP config vars, and documents all new BAREMETAL_* vars in config_example.sh. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The OS case statement only matched el9 variants, rejecting RHEL 10 with a hard exit. Add rhel10/centos10 as a supported DISTRO — same baseline as rhel9 (python3-pip, python symlink) without EPEL (unnecessary on el10). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

genisoimage is replaced by xorriso (which provides mkisofs) on RHEL 10, and python3-bcrypt is not available in base repos. Patch the metal3-dev-env package defaults to remove these entries on RHEL 10, following the same sed-patch pattern used for Go tarball arch and Ansible version fixes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The Redfish address from ironic_nodes.json includes the full API path (e.g., redfish://host/redfish/v1/Systems/1). After parsing, the 'system' variable contains 'host/redfish/v1/Systems/1'. Using ${system%%/Systems/*} strips /Systems/1 but leaves /redfish/v1 in the path, causing a doubled /redfish/v1/redfish/v1/Managers URL. Fix: extract just the BMC hostname (${system%%/*}) into bmc_base and build all Redfish URLs from that. The @odata.id values from Redfish responses are already absolute paths (/redfish/v1/Managers/1), so prepending scheme+hostname produces correct URLs. Fixes both mount_agent_iso_baremetal and eject_agent_iso_baremetal. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

When NODES_PLATFORM=baremetal and 02_configure_host.sh is skipped (no libvirt on the provisioning host), switch_to_internal_dns() was never called. This left configure_dnsmasq() writing entries to /etc/NetworkManager/dnsmasq.d/ that NetworkManager ignored because the dns=dnsmasq plugin wasn't active. The installer then failed to resolve api.<cluster>.<domain> and timed out waiting for bootstrap. Call switch_to_internal_dns() in 05_agent_configure.sh when NODES_PLATFORM=baremetal, before the configure_dnsmasq calls. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Two fixes for mount_agent_iso_baremetal: 1. Eject existing VirtualMedia before InsertMedia. iLO returns MaxVirtualMediaConnectionEstablished when a media is already mounted, silently failing the insert. The subsequent Inserted check returned true for the OLD image, masking the failure. 2. Symlink the agent ISO from OCP_DIR into WORKING_DIR/<cluster> so it's accessible via the HTTP server. The ISO is generated under ocp/<cluster>/ (relative to dev-scripts checkout) but BAREMETAL_ISO_SERVER points to a URL under the HTTP server root, which may be WORKING_DIR. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

On servers with multiple NICs, the PXE boot NIC (used in ironic_nodes.json ports) may differ from the data network NIC. When BAREMETAL_MACS is set, get_baremetal_ips_and_macs() uses those MACs for agent-config instead of ironic_nodes.json ports. Falls back to existing behavior when unset. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Drop BAREMETAL_GATEWAY from config_example.sh — nothing in dev-scripts reads it; nodes get gateway from DHCP. Remove registry /etc/hosts + login block from 02_configure_host.sh baremetal path — PROVISIONING_HOST_EXTERNAL_IP resolves to the wrong IP (computed from node subnet, not provisioning host address), and the baremetal flow uses CI_TOKEN for registry auth, not a local registry. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fonta-rh · 2026-07-10T08:28:31Z

/retest

dtantsur · 2026-07-14T08:55:24Z

+function redfish_curl() {
+    local user="$1" password="$2" url="$3"
+    shift 3
+    curl -s -u "${user}":"${password}" -k -H 'Content-Type: application/json' "$url" "$@"


Do you want -f here so that it fails if the call fails?

Added -f to redfish_curl. Also added || true on both EjectMedia calls — eject is best-effort and some BMCs return errors when nothing is inserted. With set -euxo pipefail, the -f flag makes HTTP 4xx/5xx propagate correctly through the pipe to jq.

dtantsur · 2026-07-14T08:56:34Z

+            sleep 2
+        fi
+
+        echo "Mounting ISO on ${node_name} via ${cd_slot}..."


If only we had a tool that can manage virtual media for you ;)

In all seriousness, don't expect this to work on all hardware. Ironic has 3 or 4 different ways to mount virtual media, as well as some more workarounds.

Fair point. This is currently tested on HPE iLO only — added a comment scoping it. Ironic's redfish-virtual-media boot interface with deploy_interface=ramdisk would be the right way to generalize this to other vendors. That would be a good follow-up once the basic baremetal ABI path is established.

dtantsur · 2026-07-14T08:57:03Z

+        bmc_base="${scheme}${system%%/*}"
+
+        local manager_id vmedia_uri cd_slot
+        manager_id=$(redfish_curl "$user" "$password" "${bmc_base}/redfish/v1/Managers" | jq -r '.Members[0]."@odata.id"')


Going through Managers is deprecated (but still required on HPE AFAIK)

Correct — our test hardware is HPE e920t (iLO), where Managers is the only path to VirtualMedia. Added a comment noting the deprecation and HPE requirement.

dtantsur · 2026-07-14T08:58:05Z

  sed -i '/name: "{{ packages.centos.common.packages }}"/a\      exclude: "openssl*"' vm-setup/roles/packages_installation/tasks/main.yml
+
+  # Go tarball defaults hardcode linux-amd64; use GOARCH passed as extra var.
+  # Upstream fix: https://github.com/metal3-io/metal3-dev-env/pull/1694


This is already merged, maybe update the hash?

Confirmed — PR #1694 is included in the current pin (dba726c, Jun 29). The grep guard made it a no-op, but it was dead code. Removed the entire block.

dtantsur · 2026-07-14T08:58:28Z

+  fi
+
+  # RHEL 10: genisoimage replaced by xorriso (provides mkisofs),
+  # python3-bcrypt not in base repos (soft dep of passlib, not required).


Do you have an upstream patch for this problem as well?

No upstream patch yet — genisoimage and python3-bcrypt are still in metal3-dev-env's defaults at the pinned hash. Will file an upstream PR.

dtantsur · 2026-07-14T08:58:46Z

    sudo ln -s /usr/bin/python3 /usr/bin/python || true
    PYTHON_DEVEL="python3-devel"
    ;;
+  "rhel10"|"centos10")


dtantsur · 2026-07-14T08:59:30Z

+# Comma-separated node IPs, order matches NODES_FILE entries.
+# Required when NODES_PLATFORM=baremetal.
+#
+#export BAREMETAL_IPS="10.1.155.10,10.1.155.11"


If any of these are ABI-specific, could you prefix them with AGENT_?

Renamed the ABI-specific vars:

BAREMETAL_IPS → AGENT_BAREMETAL_IPS

BAREMETAL_ISO_SERVER → AGENT_BAREMETAL_ISO_SERVER

BAREMETAL_MACS → AGENT_BAREMETAL_MACS

Kept BAREMETAL_API_VIP and BAREMETAL_INGRESS_VIP as-is — these are consumed in network.sh:340 (set_api_and_ingress_vip) which serves both IPI and ABI paths. They're platform-level, not installer-specific.

Reorganized config_example.sh to separate the two sections.

dtantsur · 2026-07-14T08:59:58Z

+# must provide an HTTP server reachable from the BMCs.
+# Required when NODES_PLATFORM=baremetal with agent installer.
+#
+#export BAREMETAL_ISO_SERVER="http://10.1.235.49:8080/agent.x86_64.iso"


At least this one is definitely ABI specific.

Renamed to AGENT_BAREMETAL_ISO_SERVER. See reply on the section comment above for the full breakdown.

- Rename ABI-specific vars to AGENT_BAREMETAL_* prefix (IPS, ISO_SERVER, MACS); keep platform-level BAREMETAL_API_VIP and BAREMETAL_INGRESS_VIP (used in network.sh for both IPI and ABI) - Add -f (--fail) to redfish_curl so HTTP errors propagate under set -euxo pipefail; add || true on EjectMedia calls (best-effort) - Add vendor scope comment on mount_agent_iso_baremetal (HPE iLO tested) and Managers path deprecation note - Remove dead Go tarball sed block (upstream fix PR openshift-metal3#1694 is included in the current metal3-dev-env pin dba726c) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

openshift-ci · 2026-07-16T14:31:06Z

@fonta-rh: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-agent-bad-dns	`b427fcc`	link	false	`/test e2e-agent-bad-dns`
ci/prow/e2e-metal-ipi-serial-ipv4-1of2	`b427fcc`	link	true	`/test e2e-metal-ipi-serial-ipv4-1of2`
ci/prow/e2e-metal-ipi-virtualmedia	`b427fcc`	link	false	`/test e2e-metal-ipi-virtualmedia`
ci/prow/e2e-agent-ha-dualstack	`b427fcc`	link	false	`/test e2e-agent-ha-dualstack`
ci/prow/e2e-metal-ipi-ovn-dualstack	`b427fcc`	link	false	`/test e2e-metal-ipi-ovn-dualstack`
ci/prow/e2e-metal-ipi-ovn-ipv4-bond	`b427fcc`	link	false	`/test e2e-metal-ipi-ovn-ipv4-bond`
ci/prow/e2e-metal-ipi-ovn-ipv4	`b427fcc`	link	true	`/test e2e-metal-ipi-ovn-ipv4`
ci/prow/e2e-agent-sno-ipv6	`b427fcc`	link	false	`/test e2e-agent-sno-ipv6`
ci/prow/e2e-metal-ipi-ovn-ipv6	`b427fcc`	link	true	`/test e2e-metal-ipi-ovn-ipv6`
ci/prow/e2e-metal-ipi-serial-ipv4-2of2	`b427fcc`	link	true	`/test e2e-metal-ipi-serial-ipv4-2of2`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 30, 2026

openshift-ci Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 30, 2026

fonta-rh force-pushed the ocpedge-2775-baremetal-deploy branch from cbd7bec to 70dc4b3 Compare June 30, 2026 13:33

fonta-rh mentioned this pull request Jun 30, 2026

Add baremetal deploy via Ansible playbook openshift-eng/two-node-toolbox#89

Merged

5 tasks

openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 1, 2026

fonta-rh force-pushed the ocpedge-2775-baremetal-deploy branch 2 times, most recently from 9e87a34 to 9a9a5d6 Compare July 6, 2026 15:23

openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 6, 2026

fonta-rh marked this pull request as ready for review July 6, 2026 15:33

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 6, 2026

openshift-ci Bot requested review from mkowalski and pawanpinjarkar July 6, 2026 15:33

openshift-ci Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 6, 2026

fonta-rh and others added 8 commits July 8, 2026 11:50

fonta-rh force-pushed the ocpedge-2775-baremetal-deploy branch from 9a9a5d6 to 3f8b16f Compare July 8, 2026 09:52

dtantsur reviewed Jul 14, 2026

View reviewed changes

This was referenced Jul 16, 2026

Split EL9/EL10-specific packages for RHEL 10 support metal3-io/metal3-dev-env#1708

Open

Rename ABI-specific baremetal vars to AGENT_BAREMETAL_* prefix openshift-eng/two-node-toolbox#97

Open

Uh oh!

Conversation

fonta-rh commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Context

Changes

Design decisions

Test plan

Uh oh!

openshift-ci Bot commented Jun 30, 2026

Uh oh!

openshift-ci Bot commented Jun 30, 2026

Uh oh!

dtantsur commented Jul 6, 2026

Uh oh!

fonta-rh commented Jul 7, 2026

Uh oh!

fonta-rh commented Jul 7, 2026

Uh oh!

fonta-rh commented Jul 7, 2026

Uh oh!

fonta-rh commented Jul 7, 2026

Uh oh!

fonta-rh commented Jul 7, 2026

Uh oh!

fonta-rh commented Jul 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fonta-rh commented Jun 30, 2026 •

edited

Loading