feat(packaging) stop generating RPM repository on remote VM + introduces staging

[dduportal]

Related to jenkins-infra/helpdesk#4774 (comment)

Must be merged at the same time as jenkinsci/packaging#540

• Replace ALL usages of SSH in jenkinsci/packaging publication scripts by a local call. It implies changing RPM's call to createrepo (Ubuntu 18.04) by createrepo_c (Ubuntu 22.04 / 24.04)
• Fixup SuSE/RPM merge specfiles and only build a single rpm for RH/SuSE jenkinsci/packaging#430 to ensure the jenkinsci/packaging's generateSite() functions take care of all site generation in all packages
• Ensure the WAR uploadPackage() function in jenkinsci/packaging generates the latest links
• Remove skipAlreadyPublished functions in all publication scripts in jenkinsci/packaging to ensure we always override the generated packages. It's disabled for Debian and RPM nowadays.
• Ensure all rsync calls have the correct flags in all publication scripts in jenkinsci/packaging: preserving timestamps, archive mode (to ensure links and special unix file are preserved along with permissions, and include recursive folders)
• Cleanup code (simplify calls to usual commands to avoid many operations, remove dead code)

---

Important notes on the "promotePackage" stage (which is the former syncMirror but renamed to materialize its full intent):

• We copy binaries FIRST to archives.jenkins.io first to ensure the fallback is set. It is a fix to avoid published packages in get.jenkins.io which answers HTTP/404 until archives.jenkins.io or any mirror got the files (30 min max)
• The publication to pkg.origin.jenkins.io does not utilize the "staging" directory on the remote VM (/var/www/pkg.jenkins.io.staging`) anymore. Instead it deploys generated files to the remote production webserver. 2 consequences:
  • No need to call the remote script sync.sh script anymore on the remote machine. But we'll have to keep it on the VM until we have both LTS and weekly packages with these changes.
  • If the publication fails during the rsync copy (not atomic), then users may have an incomplete repository. We accept that risk assuming pkg.origin.jenkins.io will be moved to Azure soon
• The publication to pkg.origin.jenkins.io also introduces the copy to the file system for the future pkg.origin.jenkins in Azure so we can start comparing

---

Testing performed in https://release.ci.jenkins.io/job/core/job/package/job/helpdesk-4774 up to the rsync commands in dry-run + pointing to the custom branch in jenkinsci/packaging.

You can test the generated packages on https://staging.pkg.origin.jenkins.io (which serves binaries from https://staging.get.jenkins.io) with VPN enabled.

=> Will have to remove d8dcf8e before merging.

[dduportal]

Requires jenkins-infra/kubernetes-management#7273

[dduportal]

jenkins-infra/kubernetes-management#7274

[MarkEWaite]

Changes look very good to me.

I think that we will always have a .ssh directory on the agent with correct permissions, so the mkdir -p is probably not needed. However, if we're retaining it, then I think we might want to set the permissions on the directory when we create it. It is not required, just an optional item for consideration. I incorrectly thought that ssh checked the permissions of the .ssh directory, but when I modified the directory permissions on Ubuntu Linux 22.04, there was no warning or error message.

The other comment is purely stylistic and can be ignored without any issue.

[dduportal]

• jenkinsci/packaging#540 has been merged
• Feedback applied (+ a nitpick on my side)
• The "test" commit has been deleted

We can proceed to merge, before the incoming 2.539 release!