Infrastructure for Orka (2024 and beyond)

[UlisesGascon]

I plan to work on it during the weekend, so I can provide a good overview on the next build meeting on Tuesday.

Current tasks on MacOS infra

• Cleaned up all the macos machines that are available via SSH
• Open discussion to apply upgrade in the Firewall: Schedule Maintenance for Orka CISCO Firewall #3642
• Review Orka state: Infrastructure for Orka (2024 and beyond) #3686 (comment)
• Plan the final allocation for resources in Orka (see: Infrastructure for Orka (2024 and beyond) #3686 (comment))
  • MacOS 13
  • Nearform MacOS Machines (see: Replacement for NearForm Mac OS machines #3638)
• Provide access for @ryanaslett to the infra-mac related resources: LFIT access to secrets #3658
• Prepare and deliver and onboarding session for @ryanaslett to the Orka infrastructure
• Add new VMs for MacOS 13 Intel (@ryanaslett)

Blocked until ARM nodes are provided

• Confirm org decision regarding new ARM nodes (discussion ongoing in the mailing list)
• Add new VMs for MacOS 13 ARM
• Add new VMs for MacOS 11 ARM

[UlisesGascon]

Current Orka state

updated on April 19, 2024

SSH port	Node: macpro-4	Node: macpro-5	Node: macpro-6
8822	release-macos11-x64-1	empty	test-macos11-x64-1
8823	empty	empty	test-macos11-x64-2
8824	empty	test-macos1015-x64-2	test-macos1015-x64-1
8825	empty	empty	empty

[UlisesGascon]

Next Orka state

updated on April 22, 2024

Intel Nodes

SSH port	Node: macpro-4	Node: macpro-5	Node: macpro-6
8822	release-macos11-x64-1	test-macos13-x64-2	test-macos11-x64-1
8823	test-macos13-x64-1	release-macos13-x64-1	test-macos11-x64-2
8824	empty	test-macos1015-x64-2	test-macos1015-x64-1
8825	empty	empty	empty

ARM Nodes

We assume that ARM Nodes can handle only 2 VMs and not +4 as Intel in the past due license limitations. This needs to be confirmed with support AFAIK?

SSH port	Node: arm-1	Node: arm-2	Node: arm-3
8822	test-macos11-arm64-1	release-macos13-arm64-1	empty
8823	release-macos11-arm64-1	test-macos13-arm64-1	test-macos13-arm64-2

How Nearform machines are "relocated"?

• release-nearform-macos11.0-arm64-1 -> release-orka-macos11-arm64-1
• test-nearform-macos11.0-arm64-1 -> test-orka-macos11-arm64-1

[targos]

release-macos13-x64-2
release-macos13-arm64-2

I don't think it's necessary to have two identical release machines.

[targos]

test-nearform-macos11.0-arm64-1

Are these typos?

[UlisesGascon]

Great feedback @targos! I updated the tables

I don't think it's necessary to have two identical release machines.

We have space for redundancy, but let's remove them for now.

Are these typos?

I made a better reference for the "relocated" machines

[targos]

release-macos13-x64-2
release-macos13-arm64-2

I don't think it's necessary to have two identical release machines.

Actually, I think we should have one x64 and two arm64 machines, because there are two jobs that run on macos-arm64 during a release (osx11-release-pkg and osx11-arm64-release-tar).

[ryanaslett]

Some questions/thoughts/suggestions:

1. Requirements Question: Do we still need to support 10.15 and/or 11? from (https://github.com/nodejs/node/blob/main/BUILDING.md#supported-platforms) I see:

Node.js does not support a platform version if a vendor has expired support for it. In other words, Node.js does not support running on End-of-Life (EoL) platforms. This is true regardless of entries in the table below.

And the table lists MacOS 11>.

And that table may be outdated as it seems as though MacOS 11 was EOL as of November 2023 ?

2. ARM support in Orka:

We assume that ARM Nodes can handle only 2 VMs and not +4 as Intel in the past due license limitations. This needs to be confirmed with support AFAIK?

https://orkadocs.macstadium.com/docs/apple-arm-based-support confirms this:

IMPORTANT

You can deploy up to 2 VMs per Apple silicon-based node.

3. From what I can gather macOS infra seems to be brittle, with nodes often running into disk issues/maintenance issues.

#3592
#3685
(https://github.com/nodejs/build/issues?q=is%3Aissue+macos+is%3Aclosed+disk) etc.

My suggestion to avoid Jenkins worker decay is to lean into an ephemeral node strategy so that each build has a fresh Orka instance to run on.

We can do that with the following Jenkins plugin for Orka:
https://plugins.jenkins.io/macstadium-orka/#plugin-content-ephemeral-agents

We would first need to set up a packer build process to create our VM images so that Orka would have a baseline image to create:
https://orkadocs.macstadium.com/docs/packer

The packer process can leverage our existing ansible playbooks:
https://developer.hashicorp.com/packer/integrations/hashicorp/ansible/latest/components/provisioner/ansible.

This strategy would require that we have an Orka3.0 cluster. Rather than trying to do an upgrade of the existing cluster, I propose that we ask macstadium to allow us to provision a new cluster with the resources we need in it (enough arm/intel backing nodes for our macos11/13 testing and release), get it built/provisioned and working, and then decommission/return all the existing macstadium/orka machines.

I believe this would end up with us using roughly the same amount of resources, so should be palatable for macstadium to support this transition.

[mhdawson]

This strategy would require that we have an Orka3.0 cluster. Rather than trying to do an upgrade of the existing cluster, I propose that we ask macstadium to allow us to provision a new cluster with the resources we need in it (enough arm/intel backing nodes for our macos11/13 testing and release), get it built/provisioned and working, and then decommission/return all the existing macstadium/orka machines.

+1 from me if Macstadium will support that

[UlisesGascon]

Quick update from our last call with MacStadium:

Next week we will have a new Orka cluster (v3) that includes 2 nodes (Intel and ARM):

• Mac Studio - G1MC M1M/10/32/16/64GB/2TB/10G
• Mac mini G4E - i7/3.2Ghz/6C/64G/1T/SSD/10G

---

Pending:

Dependencies

• Add Jenkins Plugin for Orka  #3860
• Get a new Orka Cluster running Orka 3 (on going)

✅ Setup Jenkins <-> Orka

• Add documentation in the secrets repo: https://github.com/nodejs-private/secrets/pull/342
• Create a namespace orka-test for test ci
• Create a namespace orka-release for release ci
• Create a service account for test ci sa-jenkins-test
• Create a service account for release ci sa-jenkins-release
• Generate tokens for Jenkins to connect to the release namespace
• Generate tokens for Jenkins to connect to the test namespace
• Setup the Orka Plugin in Jenkins for test CI: https://ci.nodejs.org/manage/cloud/Orka%20Cluster/
• Setup the Orka Plugin in Jenkins for release CI: https://ci-release.nodejs.org/manage/cloud/Orka%20Cluster/
• Add a VPN connection between Jenkins test CI and Orka: IMPORTANT: evaluate VPN connection between Jenkins test CI and the Orka cluster  #3883
• Add a VPN connection between Jenkins release CI and Orka: (Waiting until tuesday, see: IMPORTANT: evaluate VPN connection between Jenkins test CI and the Orka cluster  #3883 (comment))
• Ensure the cloud connection in Jenkins test environment
• Ensure the cloud connection in Jenkins release environment

Current status: Completed.

✅ Create Image templates

• Define images for each MacOS Version/arch and use (Test or release) using packer
  • Team agreement on Proposal: Drop support for MacOS prior to 13 #3876
  • MacOS 13 (Ventura) test Arm (macos-13-arm-test.pkr.hcl): Add Packer support for MacOS 13 test images in Orka #3882
  • MacOS 13 (Ventura) test Intel (macos-13-intel-test.pkr.hcl): Add Packer support for MacOS 13 test images in Orka #3882
  • MacOS 13 (Ventura) release Arm (macos-13-arm-release.pkr.hcl): Add Packer support for MacOS 13 release images in Orka #3893
  • MacOS 13 (Ventura) release Intel (macos-13-intel-release.pkr.hcl): Add Packer support for MacOS 13 release images in Orka #3893
  • Update the secrets repo: https://github.com/nodejs-private/secrets/pull/348
• Add Packer to the Build project: Add initial support for Packer in Orka #3872
• Add GitHub Action to do the validation (NO auto-deployments): Add initial support for Packer in Orka #3872
• Add Packer secrets in to the secrets repo: https://github.com/nodejs-private/secrets/pull/343

Current status: Completed.

✅ Trigger Ephemeral VMs from Jenkins

• Setup Ephemeral nodes from Jenkins on demand for the test CI and probe that they can build and test node.
• Setup Ephemeral nodes from Jenkins on demand for the release CI and probe that they can build, test and sign node.

Current status: Completed

Jobs and Agents Migration

• Add the MacOS13 to the ci test existing jobs (commit-test, night builds, v8, CITGM...) to check that all the current setup can run all the pipelines
• Add the MacOS13 to the ci release existing jobs to check that all the current setup can run all the pipelines
• Configure Jenkins Plugin to be compatible with MacOS11 and MacOS10.15 -mmacosx-version-min (see: Proposal: Drop support for MacOS prior to 13 #3876) in the test ci
• Configure Jenkins Plugin to be compatible with MacOS11 and MacOS10.15 -mmacosx-version-min (see: Proposal: Drop support for MacOS prior to 13 #3876) in the release ci
• Remove labels from legacy jenkins agents and check that the jobs are working as expected in the test ci
• Remove labels from legacy jenkins agents and check that the jobs are working as expected in the release ci

Current status: @UlisesGascon working on the setup.

Clean up

• Decommission the Bare metal machines
  • Open a support ticket
  • Remove MacStadium test from inventory #3871
  • Remove MacStadium release from inventory #3868
• Decommission Orka Cluster (Intel Nodes)
  • Open a support ticket
  • Remove Orka test from inventory #3869
  • Remove Orka release from inventory #3870
• Remove MacOS references from the Ansible playbooks
  • docs: remove macos references from manual steps #3892

Other

• Add a good documentation in the Build repo about the current Orka/Jenkins setup
• Support Marketing activities related (@UlisesGascon will provide more details)
• Elaborate a separate plan for MacOS 14 (Sonoma) and MacOS 15 (Sequoia)
• Improve SSH keys management (see: Add GitHub host key to known hosts #3896 (comment))

Deadline
The idea is to try to achieve this transition in 30 days.

Important

We don't expect any downtime will doing the migration as we will have a new cluster working on isolation will the current system is in place until we are ready to transfer the operations to the new cluster and then decommission the HW.

Challenges

• Lack of support for MacOS 11 ARM in Orka: details
• Cannot deploy more than 2 VMs on an ARM host.
• Error: admission webhook "vimage.kb.io" denied the request: cannot delete image "macos13-intel-test-latest.img". The image is being used by one or more VMs: vm-ttdzh. Remove the VMs and try again
• When Jenkins create a cloud agent if this agent failed then is not removed from the cluster
• Evaluate how to use namespace
• New VMs are much slower than expected (@UlisesGascon investigating)
• Unify the HCL templates into a single one if possible (see: Orka template updates #3906 (comment))

[UlisesGascon]

Current status

I will be on PTO from the 19th to the 25th. I made some changes to the templates to add the missing dependencies (#3906).

So, @ryanaslett, in case you want to help with this during my time off:

• Check why the iojs+release-Ulises-test-orca is not passing. The current error (10:09:52 Makefile:1030: *** No xz command, cannot continue. Stop.) (details) is related to the PATH (I think), as xz is included on all the machines already since my last PR.
• Try to modify the base image for release and test machines to include and use Xcode 15.2, based on the discussion with @targos. Documentation
• Keep running and patching the other pipelines that we cloned with @mhdawson.

Probably the next errors in the CI will be related to the users; currently, we only have the admin user. Maybe we need to create a separate one like iojs in order to make the CI pipelines work.

[ryanaslett]

Check why the iojs+release-Ulises-test-orca is not passing. The current error (10:09:52 Makefile:1030: *** No xz command, cannot continue. Stop.) (details) is related to the PATH (I think), as xz is included on all the machines already since my last PR.

Started in on this.

The PATH variable is set on the existing macos machines via the script that launches the jenkins agent:
This template:
https://github.com/nodejs/build/blob/main/ansible/roles/jenkins-worker/templates/start.j2#L10
Creates a script here:
https://github.com/nodejs/build/blob/main/ansible/roles/jenkins-worker/tasks/main.yml#L179-L185
And this Template:
https://github.com/nodejs/build/blob/main/ansible/roles/jenkins-worker/templates/org.nodejs.osx.jenkins.plist
Gets put into /Library/LaunchDaemons
https://github.com/nodejs/build/blob/main/ansible/roles/jenkins-worker/vars/main.yml#L33-L37

I've added ARCH, DESTCPU, and PATH to the Environment variables to the Orka Cluster Cloud Template configurations on ci-release machine.

The osx13-x64-release-tar job worked and signed the tarball, but failed to push the release to node-www, so, need to adjust that next.

[targos]

We need this config in the image:
https://github.com/nodejs/build/blob/main/ansible/roles/release-builder/files/ssh_config

[richardlau]

We need this config in the image: https://github.com/nodejs/build/blob/main/ansible/roles/release-builder/files/ssh_config

node-www also has a ufw2 firewall and will not allow connections from ip addresses not on the allowlist.

[ryanaslett]

I've added the main orka address to the ufw2 firewall on node-www (199.7.167.98) I've confirmed that this is the address that ephemeral nodes will all appear as to node-www.

[ryanaslett]

I've requested the new nodes from MacStadium to fill out the rest of our capacity, and got a response today that they are aiming to have the nodes installed by Wed, Oct 30th.

[mhdawson]

Great to see the details and progress on this front.

One thought is that once everything is landed it would be great to do a deep dive session for other build team members who are interested in learning a bit more about now it works.

[targos]

It's been a month now. Did we get the new nodes?

[targos]

I'm sorry to insist, but I don't know what else to do to move this forward :(

[mhdawson]

@ryanaslett, @UlisesGascon any update on this?

[ryanaslett]

Hi, yes, we did get the nodes, but havent fully transitioned testing over to using them as there was still an open question about whether or not we had the right xcode and OS versions.

Apologies that I didnt see your question earlier.

I've been wrapping up some other OpenJS project stuff for the end of the year, but I can refocus on this once more to make sure its in a stable situation.

[anonrig]

Existing macOS machines (due to outdated clang/gcc versions) are blocking 4 pull-requests:

• [do not land] update ada to v3.0-pre node#56218
• src: avoid copy by using std::views::keys node#56080
• src: modernize cleanup queue to use c++20 node#56063
• src: replace splitstring with built-in node#54990

Should we start talking about lowering the support tier of macOS? The oldest PRs are from September 17 (almost 3 months ago).