WIP: Implement GithubActions-based CF CLI Bosh release workflow

Create devcontainer.json
Add VSCode extensions
Add readme with workflow description and plan
Add Create Release GHA workflow
update trigger branch
Run upterm only in debug mode
Switch trigger to branches
remove always
try always inside {{}}
compare to true
use full reference to the debug
use env var
use runner.debug
intentionally break step
run always and debug
run if some step is not sucess
Write workflow plan
Install act, fzf, and entr
switch to the local Dockerfile
Thsi way we can preinstall all required dev dependencies
using container image , secrets and vars and make
just a crud code porting from tasks/bump-cli-release.yml
use wget to preserve filename
updated makefile to use particular workflow file; long flags
implemented secrets feedback
Switch to devcontainers/base image with docker-outside-of-docker feature.
This works ok with act
Adds sshd to enable `gh cs ssh` for external terminal access
Add shellcheck linter and CI task for it
Externalize create-bosh-release shell script
Install shellcheck as hack postinstall step
Add useful extensions to dev container
EOD, extracted bosh release canddiate to script
Add Dockerfile
Attempt to build and push CI image
Add -ci suffix to CI image name
Build shellcheck into the CI container
Can we push with the creds from GitHub web UI
Use dev container image for GHA
Update CI dockerfile to include BOSH
Fix bosh installation directory
Build CI Dockerfile from pre-existing definition
Move old Dockerfile to new location
Try to fix Dockerfile
Try pushing to github from cloud
Try running git status instead of doing anything functional
try totally default token
Radically simplify create bosh release script
Use default ubuntu dev container
Use microsoft base dev container
print out environment
Try explicitly marking git directory safe
Do only git stuff
Add upterm
Install tmux for upterm
Fix syntax error in Dockerfile
Use CI image in bosh release task
Try installing nodejs
Use summerwind runner
Explicitly run commands as root in CI Dockerfile
Explicitly add root to sudoers file
Use GHA job output
Quiet wget, fix -latest tag
Add yamllint too why not
Install much more recent NVM to work with tmate
Add NVM installed Node to PATH
Add NVM path to PATH
Use the .profile instead of .bashrc
Install node like node does
Remember to download v8 binary
delegate git push to 3rd party action
set bash as default interpreter workflow-wide; call script as a function
temporarily add tmate to find out why v8-cli-binary is not available in the cloud
continue debugging missing directory
turn off tmate after 15m
use container root; print runner temp; run tmate only when debug logging is on
use special 41898282+github-actions[bot]@users.noreply.github.com to get fancy icon
run tmate last
troubleshooting why filesystem do not persist between steps within one job on the custom container
trigger workflow on push
checking assumption that checkout step overwrites filesystem
add pwd to the safe directory
remove current version of v8 cli from blobs, that sneaked in
strip spaces from the old blob path; script refactoring; output blobstore updates
set git safe directory on script sourcing; pass major version as parameter
agumented logs with blobs update stages
use step outputs to control execution flow
updated output syntax; to suppress warnings
bump v8 cli from 8.99.99 to 8.7.10
confirm gha logic fail scenario
add comments
Ensure workflow runs from the cloud or local act is configured

Co-authored-by: Michael Chinigo <[email protected]>
Co-authored-by: david <[email protected]>

.devcontainer/Dockerfile

@@ -0,0 +1,9 @@
+FROM ghcr.io/chinigorg/bosh-package-cf-cli-release-ci:187042013-release-workflow-latest
+
+RUN apt update && apt install --yes \
+    entr \
+    fzf \
+    htop \
+    neovim
+
+RUN curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash

.devcontainer/devcontainer.json

@@ -0,0 +1,25 @@
+{
+  "name": "Cloudfoundry CLI Bosh Release",
+  "dockerFile": "Dockerfile",
+  "features": {
+    "ghcr.io/devcontainers/features/docker-outside-of-docker:1": {},
+    "ghcr.io/devcontainers/features/github-cli:1": {},
+    "ghcr.io/devcontainers/features/sshd:1": {}
+  },
+  "customizations": {
+    "vscode": {
+      "settings": {},
+      "extensions": [
+        "DavidAnson.vscode-markdownlint",
+        "GitHub.copilot",
+        "GitHub.vscode-pull-request-github",
+        "editorconfig.editorconfig",
+        "github.vscode-github-actions",
+        "ms-vscode.makefile-tools",
+        "ms-vsliveshare.vsliveshare",
+        "timonwong.shellcheck",
+        "vmware.vscode-concourse"
+      ]
+    }
+  }
+}

.github/.secret

@@ -0,0 +1,2 @@
+AWS_ACCESS_KEY_ID:     fake-aws-access-key
+AWS_SECRET_ACCESS_KEY: fake-aws-secret-access-key

.github/.vars

@@ -0,0 +1 @@
+AWS_REGION: us-west-2

.github/workflows/README.md

@@ -0,0 +1,19 @@
+# GHA Workflows
+
+## [Create Bosh Release](create-bosh-release.yml)
+
+Why? To create a new cf cli bosh release including major cli versions.
+
+### Resources
+- [Old Concourse implementation of the release pipeline](https://ci.cli.fun/teams/main/pipelines/cf-cli-release-toolsmiths)
+    - [pipeline definition](../../ci/pipeline-toolsmiths.yml)
+
+### Plan
+
+- Acquire cf cli linux binaries for v6, v7, and v8 from s3
+- Detect latest tag under each major version
+
+- ...
+
+- Upload (where?) newly created cf cli bosh release.
+- Update Releases section on GitHub https://github.com/cloudfoundry/bosh-package-cf-cli-release/releases

.github/workflows/create-bosh-release.yml

@@ -0,0 +1,93 @@
+name: Create Release
+on:
+  workflow_dispatch:
+  push:
+
+defaults:
+  run:
+    shell: bash
+    # working-directory: ${{ }}
+
+jobs:
+  bosh_release_create_candidate:
+    runs-on: ubuntu-latest
+    container:
+      image: "ghcr.io/${{ github.actor }}/bosh-package-cf-cli-release-ci:187042013-release-workflow-latest"
+
+    steps:
+    - name: Checkout cli bosh release repo
+      uses: actions/checkout@v4
+
+    - name: Acquire latest CF CLI V8
+      run: |
+        mkdir -p v8-cli-binary
+        cd v8-cli-binary
+        wget --trust-server-names "https://packages.cloudfoundry.org/stable?release=linux64-binary&version=v8&source=bosh-package-cf-cli-release-workflow"
+        tar -xvzf cf8-cli_*_linux_x86-64.tgz --wildcards 'cf*'
+        ./cf version
+
+    - name: Create bosh release candidate
+      id: create-bosh-release-candidate-step
+      env:
+        TRACE: ${{ runner.debug }}
+        # S3 final release config to upload
+        # ACCESS_KEY_ID:
+        # SECRET_KEY:
+      run: |
+        source ./ci/scripts/create-bosh-release-candidate.sh
+        create_bosh_release_candidate 8
+
+        mkdir -p candidate-release-output
+        bosh create-release --timestamp-version --tarball=./candidate-release-output/cf-cli-dev-release.tgz
+
+    ## Store tgz with bosh release candidate
+
+    - name: Push changes
+      uses: ad-m/github-push-action@master
+      with:
+        github_token: ${{ secrets.GITHUB_TOKEN }}
+        branch: ${{ github.ref }}
+
+    ## Claim Shepherd env
+    ## Deploy cli bosh release candidate
+    ## Unclaim shepherd env
+
+
+    ## Finalize bosh release
+    ##   Get final release from assets
+    ## Push commit with final release
+    - name: Finalize bosh release
+      # if: steps.create-bosh-release-candidate-step.outputs.blobs_updated == 'yes'
+      # env:
+      #   TRACE: ${{ runner.debug }}
+      # S3 final release config to upload
+      # ACCESS_KEY_ID:
+      # SECRET_KEY:
+      run: |
+        set -o errexit -o nounset -o pipefail
+        [[ "${TRACE:-0}" == "1" ]] && set -o xtrace
+
+        # if [[ "${{ steps.create-bosh-release-candidate-step.outputs.blobs_updated }}" == "yes" ]]; then
+        
+        # echo "Bosh Blobs: initial state"
+        # bosh blobs
+
+        ## Determine next release version number
+
+        # RELEASE_VERSION=$(cat release-version/version)
+        # git config --global --add safe.directory "$(pwd)"
+        # git config user.name  "github-actions[bot]"
+        # git config user.email "41898282+github-actions[bot]@users.noreply.github.com "
+
+        # bosh create-release --final --version="${RELEASE_VERSION}" --tarball="./cf-cli-v${RELEASE_VERSION}.tgz"
+
+        # git add --all
+        # git status
+        # git commit -m "create final release ${RELEASE_VERSION}"
+
+        ls -lah ./candidate-release-output/cf-cli-dev-release.tgz
+
+    - name: Setup tmate session
+      if: ${{ always() && runner.debug }}
+      uses: mxschmitt/action-tmate@v3
+      timeout-minutes: 15

.github/workflows/ensure-ci-image.yml

@@ -0,0 +1,77 @@
+name: Ensure CI image
+
+on:
+  workflow_dispatch:
+
+env:
+  IMAGE_REGISTRY: ghcr.io
+  CI_DOCKERFILE_DIR: ./ci # Relative to project root
+  CI_DOCKERFILE_PATH: Dockerfile # Relative to CI_DOCKERFILE_DIR
+  CI_DOCKERFILE_MOST_RECENT_SHA: # Determined dynamically later on
+
+jobs:
+  calculate-latest-label:
+    runs-on: ubuntu-latest
+
+    permissions:
+      contents: read
+
+    outputs:
+      ci_dockerfile_latest_sha: ${{ steps.calculate_latest_sha.outputs.ci_dockerfile_latest_sha }}
+
+    steps:
+    - name: Checkout repo
+      uses: actions/checkout@v4
+
+    - name: Calculate label for CI image
+      id: calculate_latest_sha
+      env:
+        IMAGE_NAME: ${{ github.repository }}
+      run: |
+        dockerfile_path=${CI_DOCKERFILE_DIR}/${CI_DOCKERFILE_PATH}
+
+        [[ ! -f ${dockerfile_path} ]] && echo "Could not find Dockerfile at ${dockerfile_path}" 1>&2 && exit 1
+
+        echo "ci_dockerfile_latest_sha=$(git log --max-count 1 --pretty=format:%H "${dockerfile_path}")" >> $GITHUB_OUTPUT
+
+  build-and-push-ci-image:
+    runs-on: ubuntu-latest
+
+    permissions:
+      contents: read
+      packages: write
+
+    needs:
+    - calculate-latest-label
+
+    steps:
+    - name: Checkout repo
+      uses: actions/checkout@v4
+
+    - name: Login to GitHub container registry
+      uses: docker/login-action@v3
+      with:
+        registry: ${{ env.IMAGE_REGISTRY }}
+        username: ${{ github.actor }}
+        password: ${{ secrets.GITHUB_TOKEN }}
+
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v3
+
+    - name: Extract metadata
+      uses: docker/metadata-action@v5
+      id: meta
+      with:
+        images: ${{ env.IMAGE_REGISTRY }}/${{ github.repository }}-ci
+        tags: |
+          type=raw,value=${{ needs.calculate-latest-label.outputs.ci_dockerfile_latest_sha }}
+          type=raw,value=${{ github.ref_name }}-latest
+
+    - name: Build and push CI image
+      uses: docker/build-push-action@v5
+      with:
+        push: true
+        context: ${{ env.CI_DOCKERFILE_DIR }}
+        file: ${{ env.CI_DOCKERFILE_DIR }}/${{ env.CI_DOCKERFILE_PATH }}
+        tags: ${{ steps.meta.outputs.tags }}
+        labels: ${{ steps.meta.outputs.labels }}

.github/workflows/lint.yml

@@ -0,0 +1,15 @@
+name: Lint shell scripts
+on:
+  workflow_dispatch:
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    container: ghcr.io/chinigorg/bosh-package-cf-cli-release:187042013-release-workflow-latest
+
+    steps:
+    - name: Checkout cli bosh release repo
+      uses: actions/checkout@v4
+
+    - name: lint
+      run: find ./ci -type f -name '*.sh' | xargs -t shellcheck

Makefile

@@ -0,0 +1,32 @@
+ifndef GITHUB_USER
+	$(error GITHUB_USER is not set)
+endif
+
+create-bosh-release:
+	act \
+		--actor "${GITHUB_USER}" \
+		--secret GITHUB_TOKEN="${GITHUB_TOKEN}" \
+		--workflows .github/workflows/create-bosh-release.yml
+
+ensure-ci-image:
+	act \
+		--actor "${GITHUB_USER}" \
+		--secret GITHUB_TOKEN="${GITHUB_TOKEN}" \
+		--workflows .github/workflows/ensure-ci-image.yml
+
+lint:
+	act \
+		--actor "${GITHUB_USER}" \
+		--secret GITHUB_TOKEN="${GITHUB_TOKEN}" \
+		--workflows .github/workflows/lint.yml
+
+run:
+	ls **/* | entr -c \
+		act \
+			--actor "${GITHUB_USER}" \
+			--secret GITHUB_TOKEN="${GITHUB_TOKEN}" \
+			--workflows .github/workflows/create-bosh-release.yml \
+			--job bosh_release_create_candidate
+
+hijack-act:
+	./ci/scripts/hijack-act.sh