add compile command

[crhntr]

Problem

Compiling BOSH release packages in CI requires either

• an IAAS deployed director
• a task with "privileged: true"

This blocks integrating build observers and limits where we can build compiled tarballs.

We wrote a script that can build release tarballs with compiled packages by reverse engineering the commands issued to the agent from the BOSH Director. Unfortunately, this script still requires the warden stemcell container be run with privileged: true because the bosh-agent does a few things during startup that are not required for package compilation.

Both using an internal RPC interface and requiring privileged containers make using the existing BOSH stemcell images for release compilation in CI harder and more brittle than than it needs to be.

Solution

Add a sub-command to the bosh-agent that sidesteps the privileged parts of initialization and that can compile releases tarballs and that can be called directly.

$ bosh-agent compile --help
The BOSH Agent compile command creates BOSH Release tarballs with compiled packages from tarballs with source packages.

Usage:

        compile [FLAGS] [TARBALL...]

Flags:
  -output-directory string
        the directory to put the compiled release tarball (default "/tmp")

Example using the docker CLI

A Concourse task wrapping the following example should be trivial.

Given some warden stemcell jammy-stemcell-warden:1.260 as TAG,
And you have donwloaded a BOSH Release tarball at releases-input/log-cache-release-3.0.7.tgz
When you run

docker run --platform=linux/amd64 -it --rm \
           -v "${PWD}/releases-input:/releases-input" \
           -v "${PWD}/releases-output:/releases-output" \
           "${TAG}" /var/vcap/bosh/bin/bosh-agent \
           compile --output-directory=/releases-output /releases-input/log-cache-release-3.0.7.tgz

Then You will get releases-output/log-cache-release-3.0.7-ubuntu-jammy-1.260.tgz

Note the docker command does not require a "--privileged" flag.

If you want to try using this functionality locally, I added a script to replace the bosh-agent in the image and load the appropriate volumes. You can find it in a script in the repo at bin/docker-run-bosh-agent.

Limitations/Warnings

The following are not a problem when running the command on a warden linux stemcell in CI.

• The configuration at /var/vcap/bosh/warden-cpi-agent-env.json is ignored and configuration of the "blobstore" comes from the initialization function github.com/cloudfoundry/bosh-agent/releasetarball.NewCompiler. This may be surprising if someone uses the compile command on non-warden stemcells with different agent configurations.
• It uses the app.DefaultBaseDirectory as the base directory for the directories provider and may mutate files there. I am not sure if this will cause problems for a running BOSH agent.
• bosh-agent compile will fail windows with a helpful message. The code I wrote likely supports windows paths (I believe I used "filepath" and "path" appropriately); however, the part of the code that loads the stemcell name/version uses linux specific paths. See the new package "github.com/cloudfoundry/bosh-agent/stemcellmetadata" for details.

[crhntr]

An (imo non-blocking) issue with this PR:

It vendors "gopkg.in/yaml.v2", v2 which no longer supported.
v2 is imported because the BOSH CLI uses it in the manifest package.
v3 is already used by both repositories and has passed OSL.
If we don't do this, the following CVEs may show up in CVE scans: https://pkg.go.dev/search?q=gopkg.in%2Fyaml.v2&m=vuln.

I made a PR here to do the bump: cloudfoundry/bosh-cli#639

[crhntr]

I feel bad that the release tarball contains the Go package; from a deployment perspective, the number duplicate bytes across all the releases that contain the Go package is huge. I wish we had some way to annotate packages with a field like compilation_dependency: true that would signal to this compiler that the package can be omitted from a release tarball post compilation. I considered adding a flag to this command to omit the Go package but that solution seemed too brittle. Package omission should be controlled by the release author.

[jpalermo]

I feel bad that the release tarball contains the Go package; from a deployment perspective, the number duplicate bytes across all the releases that contain the Go package is huge. I wish we had some way to annotate packages with a field like compilation_dependency: true that would signal to this compiler that the package can be omitted from a release tarball post compilation. I considered adding a flag to this command to omit the Go package but that solution seemed too brittle. Package omission should be controlled by the release author.

We actually modified compilation to exclude packages not required by a job earlier this year. Unfortunately it broke everything.

The problem is that bosh uses a package fingerprint to know if the package has been modified and is different from another copy of that package. The fingerprint is based on the source files for the package (their bits, as well as their paths and permissions). This means if you change the version of the golang package you built a package with, the package fingerprint doesn't change, and bosh thinks it's the same, even though the compiled bits are built with a totally different version of golang.

So to fix it, we'd need to change the structure of release metadata to include the fingerprint data for packages a package depends on, without actually including those packages. It would likely be a largish breaking change, so we just backed away from it.

[ramonskie]

it would be a good idea to use this in our bosh director pipeline
see

• https://github.com/cloudfoundry/bosh/blob/main/ci/pipeline.yml#L254
• https://github.com/cloudfoundry/bosh/blob/main/ci/tasks/export-release.sh

[jpalermo]

I tried running it and got this:

/:/agent-dev# ./out/bosh-agent compile --output-directory=$PWD ./bosh-280.0.12.tgz
2023/12/23 01:47:27 Compiling with stemcell ubuntu-jammy/1.329
Reading BOSH Release Manifest from tarball ./bosh-280.0.12.tgz
Release bosh/280.0.12 has 17 packages
Extracting packages
failed to extract packages from tarball: open /var/vcap/data/blobs/nginx.tgz: no such file or directory

Keep in mind, this isn't the exact desired use case since I didn't have a stemcell with this agent handy. But I imported the latest jammy stemcell into docker, then ran the container locally and mounted in the bosh-agent I had built.

The error seems to imply it's looking for /var/vcap/data, which doesn't exist because that's normally created during agent bootstrapping which isn't getting run.

[crhntr]

@jpalermo I added the a MkdirAll to the command. I used 0766 permission bits. I am unsure if they should be less permissive (i found how it is done in the bootstrapping and copied that).

[crhntr]

I've been vendoring the HEAD of "github.com/cloudfoundry/bosh-cli/v7/release/manifest" from a SHA so I get the yaml.v3 changes (cloudfoundry/bosh-cli#639). Any chance there will be a tagged v7 revision of the BOSH CLI soon?

[crhntr]

I force-pushed a manual rebase, go mod tidy, and go mod download.

[selzoc]

I've been vendoring the HEAD of "github.com/cloudfoundry/bosh-cli/v7/release/manifest" from a SHA so I get the yaml.v3 changes (cloudfoundry/bosh-cli#639). Any chance there will be a tagged v7 revision of the BOSH CLI soon?

This has been released in bosh-cli 7.5.2

[jpalermo]

I was able to compile the bosh-director just fine with it:

2024/01/06 19:33:07 Compiling with stemcell ubuntu-jammy/1.329
Reading BOSH Release Manifest from tarball ./bosh-280.0.12.tgz
Release bosh/280.0.12 has 17 packages
Extracting packages
Copying package nginx to /var/vcap/data/blobs/nginx.tgz
Copying package director to /var/vcap/data/blobs/director.tgz
Copying package postgres-13 to /var/vcap/data/blobs/postgres-13.tgz
Copying package azure-storage-cli to /var/vcap/data/blobs/azure-storage-cli.tgz
Copying package davcli to /var/vcap/data/blobs/davcli.tgz
Copying package bosh-gcscli to /var/vcap/data/blobs/bosh-gcscli.tgz
Copying package postgres-15 to /var/vcap/data/blobs/postgres-15.tgz
Copying package director-ruby-3.1 to /var/vcap/data/blobs/director-ruby-3.1.tgz
Copying package health_monitor to /var/vcap/data/blobs/health_monitor.tgz
Copying package verify_multidigest to /var/vcap/data/blobs/verify_multidigest.tgz
Copying package director-ruby-3.2 to /var/vcap/data/blobs/director-ruby-3.2.tgz
Copying package postgres-10 to /var/vcap/data/blobs/postgres-10.tgz
Copying package libpq to /var/vcap/data/blobs/libpq.tgz
Copying package mysql to /var/vcap/data/blobs/mysql.tgz
Copying package powerdns to /var/vcap/data/blobs/powerdns.tgz
Copying package nats to /var/vcap/data/blobs/nats.tgz
Copying package s3cli to /var/vcap/data/blobs/s3cli.tgz
Starting packages compilation
Compiling package azure-storage-cli/7be841fab23c7dac21de007f1ef7956dda59333be3b6a7ae51884afe0dfdd49b
Finished compiling release azure-storage-cli/7be841fab23c7dac21de007f1ef7956dda59333be3b6a7ae51884afe0dfdd49b BlobstoreID=78028c3f-440b-438c-78cb-d7b0dd373f01
Compiling package bosh-gcscli/7812b42fb1b70d42d8c552a677e3752bfbc6f63c8885c5c8de040c74b69e5237
Finished compiling release bosh-gcscli/7812b42fb1b70d42d8c552a677e3752bfbc6f63c8885c5c8de040c74b69e5237 BlobstoreID=eb131610-6a63-4d59-42db-47bdc54e0578
Compiling package davcli/ba8e4c230136743bc3c6a0425026a855c02480dc8c9e04196921d909edcc8fff
Finished compiling release davcli/ba8e4c230136743bc3c6a0425026a855c02480dc8c9e04196921d909edcc8fff BlobstoreID=684dcdb2-b310-4760-57b9-18240d52c827
Compiling package libpq/b309a72768019e24e2c592f3f25ded2679e98cbb90f774c3a4d6b7745760079f
Finished compiling release libpq/b309a72768019e24e2c592f3f25ded2679e98cbb90f774c3a4d6b7745760079f BlobstoreID=3c6107e2-b2d1-4ae0-4114-ca1d0a2b1c41
Compiling package mysql/5bf574c1f12cd6b5c07bf0c266b4166584bf0503634f1281bf58637f46f2bd5f
Finished compiling release mysql/5bf574c1f12cd6b5c07bf0c266b4166584bf0503634f1281bf58637f46f2bd5f BlobstoreID=6746f492-b8cc-4868-5591-269f6d636339
Compiling package director-ruby-3.2/84ee2f9d0485530a75822fa03e7fd0c73544aa4c2f6fe24aaebebe1757195efe
Finished compiling release director-ruby-3.2/84ee2f9d0485530a75822fa03e7fd0c73544aa4c2f6fe24aaebebe1757195efe BlobstoreID=0de1b578-ace6-4906-52e2-9fa7c8103d6c
Compiling package director/1e8ac7cdd35cd2277706fb607b6b373729b9614cc949e20be10f7e22a151b190
Finished compiling release director/1e8ac7cdd35cd2277706fb607b6b373729b9614cc949e20be10f7e22a151b190 BlobstoreID=55b62c93-0571-4e2e-5b5a-f436a98c8c1e
Compiling package director-ruby-3.1/82651486268c5a872706d5e1a2458b76af7e69fe36dbfed64903c64a3af39eae
Finished compiling release director-ruby-3.1/82651486268c5a872706d5e1a2458b76af7e69fe36dbfed64903c64a3af39eae BlobstoreID=371f3897-2c2f-4251-610a-506f6e80b89e
Compiling package health_monitor/4d2d309eb06ddaf2993cc4d2253460122375786dab011591047ccf6efa932737
Finished compiling release health_monitor/4d2d309eb06ddaf2993cc4d2253460122375786dab011591047ccf6efa932737 BlobstoreID=010e49b6-262e-4a95-548f-ea2f83619fd9
Compiling package nats/a9355d553cab9913f195eaf00cbbd01ec3c2b8b692b851ce34e0b95dad7ca0b0
Finished compiling release nats/a9355d553cab9913f195eaf00cbbd01ec3c2b8b692b851ce34e0b95dad7ca0b0 BlobstoreID=4562253f-191b-4199-6250-794ad21cb917
Compiling package nginx/82a22b536cf378d354f9325dadcbcb2fa70b1ce9e37eb65a8a7a97cd35e8fc45
Finished compiling release nginx/82a22b536cf378d354f9325dadcbcb2fa70b1ce9e37eb65a8a7a97cd35e8fc45 BlobstoreID=e6f41a33-cf50-4a38-68d0-5d7091befc22
Compiling package postgres-10/e3f2ed31116e1a0c929ae6fcdde983a9d6c000c25cafde8a784fd126e06400f9
Finished compiling release postgres-10/e3f2ed31116e1a0c929ae6fcdde983a9d6c000c25cafde8a784fd126e06400f9 BlobstoreID=23d100d8-886d-4561-757a-32adc6f4d76d
Compiling package postgres-13/a3141b9f3664abe145c6fb452a54b3bbc4b772933083c2c1ef725c0a7c71824f
Finished compiling release postgres-13/a3141b9f3664abe145c6fb452a54b3bbc4b772933083c2c1ef725c0a7c71824f BlobstoreID=8da9d28a-345c-4fe4-75e8-f80d26869ce2
Compiling package postgres-15/1059ac62d543dc19011001f80f8c0bb99cc3a9ea4f8c14736e480701051ce9f0
Finished compiling release postgres-15/1059ac62d543dc19011001f80f8c0bb99cc3a9ea4f8c14736e480701051ce9f0 BlobstoreID=170a3822-e7e9-45ff-77dd-8a4b4fae9707
Compiling package powerdns/95d1ce42da1bbaa19021bcaa295ed4f0f05e82c44b40638fcc934d91d212d2af
Finished compiling release powerdns/95d1ce42da1bbaa19021bcaa295ed4f0f05e82c44b40638fcc934d91d212d2af BlobstoreID=7750d851-c5f2-4c5d-66bc-262504508fb2
Compiling package s3cli/66e805d974a48631f5c963f009b9a14d451ed5bdf88900f45fa86c9a7342a2ff
Finished compiling release s3cli/66e805d974a48631f5c963f009b9a14d451ed5bdf88900f45fa86c9a7342a2ff BlobstoreID=e9077821-2c1e-4df0-4105-4e33d6ca4bf6
Compiling package verify_multidigest/5a45c1115b9a7478682ba63804050a723d9eb6e28d91432ad36eba30cc084c59
Finished compiling release verify_multidigest/5a45c1115b9a7478682ba63804050a723d9eb6e28d91432ad36eba30cc084c59 BlobstoreID=bed88047-72f7-4007-68a8-fcd030de2bab
Finished packages compilation after 47m23.551391571s
Archiving compiled BOSH Release bosh/280.0.12 with stemcell ubuntu-jammy-1.329
Finished archiving compiled tarball /tmp/build/e55deab7/artifacts/bosh-280.0.12-ubuntu-jammy-1.329.tgz

[crhntr]

I force pushed a rebase on main fixing the conflicts in go.mod, go.sum, and vendor/modules.txt.

[rkoster]

Really nice work, only found an unused file.

[rkoster]

Thanks! @crhntr