Skip to content

Commit a431e8a

Browse files
author
Patryk Huzarski
committed
ci: isolate Cloud Run runner pilot
1 parent 6332acb commit a431e8a

6 files changed

Lines changed: 593 additions & 13 deletions

File tree

‎.github/runner/cloud-run/.dockerignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
*
2+
!Dockerfile
3+
!start.sh

‎.github/runner/cloud-run/Dockerfile‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
ARG RUNNER_PLATFORM=linux/amd64
19+
20+
FROM --platform=${RUNNER_PLATFORM} docker.io/library/rust@sha256:39f68a3e8e3ff425f8945ffa91128e60ff930d53e17fbb5214e95824bdd46f1b AS rust-toolchain
21+
22+
RUN rustup component add clippy rustfmt --toolchain 1.97.1
23+
24+
FROM --platform=${RUNNER_PLATFORM} ghcr.io/actions/actions-runner@sha256:a1919047b038c38871d667c58cfdc7a878452711ab1212fb6036188f27a7ab16
25+
26+
USER root
27+
28+
ENV CARGO_HOME=/home/runner/.cargo
29+
ENV RUSTUP_HOME=/usr/local/rustup
30+
ENV RUSTUP_TOOLCHAIN=1.97.1-x86_64-unknown-linux-gnu
31+
ENV PATH=/usr/local/cargo/bin:${PATH}
32+
33+
COPY --from=rust-toolchain /usr/local/cargo /usr/local/cargo
34+
COPY --from=rust-toolchain /usr/local/rustup /usr/local/rustup
35+
COPY start.sh /usr/local/bin/iggy-cloud-runner-start
36+
37+
RUN chmod 0555 /usr/local/bin/iggy-cloud-runner-start \
38+
&& mkdir -p /home/runner/.cargo \
39+
&& chown runner:runner /home/runner/.cargo \
40+
&& gpasswd --delete runner sudo \
41+
&& gpasswd --delete runner docker \
42+
&& rm -f /usr/bin/docker \
43+
/usr/bin/sudo \
44+
/usr/local/lib/docker/cli-plugins/docker-buildx \
45+
&& command -v curl \
46+
&& command -v jq \
47+
&& command -v tail \
48+
&& rustc --version \
49+
&& cargo --version \
50+
&& rustfmt --version
51+
52+
USER runner
53+
54+
ENTRYPOINT ["/usr/local/bin/iggy-cloud-runner-start"]

‎.github/runner/cloud-run/README.md‎

Lines changed: 278 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,278 @@
1+
# Cloud Run runner pilot
2+
3+
This directory contains a narrow experiment for one Apache Iggy GitHub Actions
4+
job on a Cloud Run worker pool. It is not a replacement for the existing
5+
pre-merge workflows.
6+
7+
## Scope
8+
9+
The pilot runs `cargo fmt --all -- --check` and `cargo fetch --locked` on two
10+
runners for comparison:
11+
12+
- one ephemeral Cloud Run runner;
13+
- one GitHub-hosted `ubuntu-latest` runner.
14+
15+
The workflow runs only on an upstream push to
16+
`ci/self-hosted-gcp-runners`. It has read-only repository permissions and is not
17+
triggered by pull requests. Both jobs also require the repository
18+
variable `IGGY_CLOUD_RUNNER_PILOT_ENABLED` to equal `true`; an absent variable
19+
keeps it skipped while the workflow and runner-group restrictions are being
20+
reviewed. Do not make it a required check.
21+
22+
The Cloud Run image does not provide Docker, privileged mode, `sudo`, or the
23+
GitHub-hosted runner tool cache. Do not route BDD, testcontainers, Buildx,
24+
`kind`, publish, or release jobs to this runner.
25+
26+
## Security boundary
27+
28+
Apache Infra must create an organization runner group named
29+
`iggy-cloud-run-pilot` and restrict it to:
30+
31+
- the `apache/iggy` repository;
32+
- `apache/iggy/.github/workflows/cloud-run-runner-pilot.yml@refs/heads/ci/self-hosted-gcp-runners`.
33+
34+
The Cloud Run worker pool must be configured with one instance for the pilot.
35+
36+
The runner registers with the custom label `iggy-cloud-run-x64`. The workflow
37+
requires both the group and label, so a generic `self-hosted` runner cannot pick
38+
up the job accidentally.
39+
40+
The initial test uses a short-lived organization registration token issued by
41+
Apache Infra. It is stored in a dedicated Secret Manager secret, removed from
42+
the runner process environment before the job starts, and used to register an
43+
ephemeral one-job runner. Never use a classic or personal PAT here.
44+
45+
Use a dedicated GCP sandbox project. The runtime service account must have no
46+
project-level role, no VPC connector, no production access, and no service
47+
account impersonation permission. Grant it access only to the single temporary
48+
registration-token secret. Cloud Run captures container stdout and stderr
49+
without granting the workload service account a project-level logging role.
50+
51+
The worker can use its metadata identity during a job, so unsetting the token
52+
environment variable is not the complete security boundary. After the runner
53+
appears online and before pushing the pilot branch, destroy the exact one-time
54+
secret version and remove the runtime service account's direct access to the
55+
secret. Do not proceed unless Secret Manager reports the version as
56+
`DESTROYED`. This remains effective even if broader IAM accidentally grants the
57+
service account read access to the secret container.
58+
59+
## Why the first test does not autoscale
60+
61+
GitHub registration tokens expire after about one hour. A token supplied once
62+
by Infra is sufficient for a fixed, one-shot pilot, but not for a pool that
63+
creates new instances continuously.
64+
65+
Apache Beam solves continuous registration with a GitHub App and a GCP Cloud
66+
Function that exchanges an App installation token for a fresh organization
67+
runner registration token. Its newer Actions Runner Controller deployment also
68+
uses GitHub App credentials. Iggy should reuse an Infra-approved version of
69+
that pattern or an Infra-managed token broker before enabling autoscaling.
70+
71+
CREMA is intentionally excluded from this pilot. Its GitHub runner scaler has
72+
an open scale-down issue where queued work can become zero while a job is still
73+
in progress, allowing the worker pool to terminate the active runner. A fixed
74+
single instance keeps that variable out of the first experiment.
75+
76+
## Continuous registration design
77+
78+
The image also supports a renewable JIT broker mode for later scale-to-zero.
79+
Set `RUNNER_JIT_CONFIG_BROKER_URL` and
80+
`RUNNER_JIT_CONFIG_BROKER_AUDIENCE` instead of
81+
`RUNNER_REGISTRATION_TOKEN`. At startup the container obtains a Google-signed
82+
identity token from the metadata server, calls the HTTPS broker, and expects a
83+
JSON response containing a non-empty `encoded_jit_config`. It then starts the
84+
runner with `run.sh --jitconfig`.
85+
86+
The broker is a separate control-plane service. It must:
87+
88+
- accept invocations only from the dedicated worker service account;
89+
- use an ASF-installed GitHub App with organization `Self-hosted runners: write`;
90+
- call GitHub's `generate-jitconfig` endpoint with the fixed Iggy runner group,
91+
label, and `_work` directory;
92+
- ignore any client-supplied organization, runner group, labels, or repository;
93+
- return only `{ "encoded_jit_config": "..." }` and never return an App key,
94+
installation token, or registration token;
95+
- atomically lease one allowlisted queued `workflow_job` for each JIT config,
96+
limit issuance to one outstanding config per lease, and reject calls when no
97+
eligible lease exists;
98+
- rate-limit and audit issuance without logging credentials.
99+
100+
Do not grant the worker service account Secret Manager access in broker mode.
101+
Its only additional permission is invocation of that one broker service. The
102+
pilot remains trusted-push-only because job code inherits the worker identity
103+
and could otherwise invoke the broker itself.
104+
105+
Broker mode is not an authorization boundary for untrusted jobs. Rate limiting
106+
alone does not prevent job code from minting another runner. Public pull
107+
requests must remain GitHub-hosted. Supporting untrusted jobs would require a
108+
separate controller identity to pre-authorize each queued job and a worker that
109+
cannot independently mint another JIT config.
110+
111+
The broker solves runner registration, not CREMA authentication. The scaler
112+
needs a separate service account and renewable GitHub Actions read access. Do
113+
not reuse the worker identity or the broker's GitHub App private key in CREMA.
114+
115+
## Image contents
116+
117+
The image is Linux x86-64 only. Its base images are pinned by platform digest:
118+
119+
- GitHub Actions runner `2.336.0`;
120+
- Rust `1.97.1`, matching `rust-toolchain.toml`.
121+
122+
Runner auto-update and Rust channel synchronization are disabled. The workflow
123+
fails before formatting if `rust-toolchain.toml` no longer matches the image.
124+
Update both digests and `RUSTUP_TOOLCHAIN` deliberately when either version
125+
changes.
126+
127+
## One-shot test procedure
128+
129+
The commands below are an operator runbook, not an automated deployment. Pass
130+
`--project` and `--region` on every command instead of changing the active
131+
`gcloud` context.
132+
133+
First keep `IGGY_CLOUD_RUNNER_PILOT_ENABLED` absent or set it to `false`, then
134+
push this gated workflow. Both jobs must be skipped. This makes the exact
135+
workflow path and branch available for Apache Infra to review and add to the
136+
runner-group allowlist without queueing work for a runner that does not exist.
137+
138+
Set local identifiers:
139+
140+
```bash
141+
PROJECT_ID="replace-with-pmc-sandbox-project"
142+
REGION="us-central1"
143+
AR_REPOSITORY="iggy-ci-pilot"
144+
WORKER_POOL="iggy-cloud-run-pilot"
145+
RUNTIME_SERVICE_ACCOUNT="iggy-runner-runtime@${PROJECT_ID}.iam.gserviceaccount.com"
146+
TOKEN_SECRET="iggy-runner-registration-token"
147+
IMAGE_URI="${REGION}-docker.pkg.dev/${PROJECT_ID}/${AR_REPOSITORY}/runner"
148+
```
149+
150+
Build the pinned image from the repository root:
151+
152+
```bash
153+
gcloud builds submit .github/runner/cloud-run \
154+
--project "$PROJECT_ID" \
155+
--region "$REGION" \
156+
--tag "${IMAGE_URI}:pilot"
157+
158+
IMAGE_DIGEST="$(gcloud artifacts docker images describe "${IMAGE_URI}:pilot" \
159+
--project "$PROJECT_ID" \
160+
--format='value(image_summary.digest)')"
161+
test -n "$IMAGE_DIGEST"
162+
```
163+
164+
Create the dedicated secret and grant the runtime identity access only to that
165+
secret:
166+
167+
```bash
168+
gcloud secrets create "$TOKEN_SECRET" \
169+
--project "$PROJECT_ID" \
170+
--replication-policy automatic
171+
172+
gcloud secrets add-iam-policy-binding "$TOKEN_SECRET" \
173+
--project "$PROJECT_ID" \
174+
--member "serviceAccount:${RUNTIME_SERVICE_ACCOUNT}" \
175+
--role roles/secretmanager.secretAccessor
176+
```
177+
178+
Add the token supplied by Infra without placing it in shell history:
179+
180+
```bash
181+
read -rsp "Runner registration token: " RUNNER_REGISTRATION_TOKEN
182+
printf '%s' "$RUNNER_REGISTRATION_TOKEN" | gcloud secrets versions add "$TOKEN_SECRET" \
183+
--project "$PROJECT_ID" \
184+
--data-file=-
185+
unset RUNNER_REGISTRATION_TOKEN
186+
```
187+
188+
Deploy exactly one instance by immutable image digest:
189+
190+
```bash
191+
TOKEN_VERSION="replace-with-created-secret-version"
192+
gcloud run worker-pools deploy "$WORKER_POOL" \
193+
--project "$PROJECT_ID" \
194+
--region "$REGION" \
195+
--image "${IMAGE_URI}@${IMAGE_DIGEST}" \
196+
--instances 1 \
197+
--cpu 4 \
198+
--memory 8Gi \
199+
--service-account "$RUNTIME_SERVICE_ACCOUNT" \
200+
--set-secrets "RUNNER_REGISTRATION_TOKEN=${TOKEN_SECRET}:${TOKEN_VERSION}"
201+
```
202+
203+
After the runner is online, destroy the one-time token version and revoke the
204+
runtime identity's direct access before enabling or triggering the pilot:
205+
206+
```bash
207+
gcloud secrets versions destroy "$TOKEN_VERSION" \
208+
--project "$PROJECT_ID" \
209+
--secret "$TOKEN_SECRET"
210+
211+
gcloud secrets remove-iam-policy-binding "$TOKEN_SECRET" \
212+
--project "$PROJECT_ID" \
213+
--member "serviceAccount:${RUNTIME_SERVICE_ACCOUNT}" \
214+
--role roles/secretmanager.secretAccessor
215+
216+
test "$(gcloud secrets versions describe "$TOKEN_VERSION" \
217+
--project "$PROJECT_ID" \
218+
--secret "$TOKEN_SECRET" \
219+
--format 'value(state)')" = "DESTROYED"
220+
```
221+
222+
Set `IGGY_CLOUD_RUNNER_PILOT_ENABLED=true` only after Apache Infra confirms the
223+
group restriction and the command above proves the one-time credential no
224+
longer exists. Trigger the workflow with a new push to the pilot branch. Scale
225+
the pool to zero as soon as the single job finishes:
226+
227+
```bash
228+
gcloud run worker-pools update "$WORKER_POOL" \
229+
--project "$PROJECT_ID" \
230+
--region "$REGION" \
231+
--instances 0
232+
```
233+
234+
After the ephemeral runner exits, the entrypoint deliberately keeps the
235+
container idle instead of registering another runner. The pool still incurs
236+
cost until it is scaled to zero, so prompt scale-down is mandatory.
237+
238+
## Acceptance criteria
239+
240+
The first pilot passes only when all of the following are true:
241+
242+
1. Apache Infra confirms the runner group and workflow restriction.
243+
2. The Cloud Run job and GitHub-hosted baseline check the same commit SHA.
244+
3. The runner handles exactly one job and is automatically de-registered.
245+
4. No pull request or other workflow can target the runner group.
246+
5. The worker has no access to production, deployment credentials, or private
247+
networks.
248+
6. The exact registration-token secret version is `DESTROYED` before the
249+
workflow is triggered.
250+
7. Cloud Logging shows no restart, OOM, unexpected signal, or registration
251+
secret in logs.
252+
8. The pool is scaled to zero after the experiment.
253+
254+
Compare queue time, startup time, execution time, result, peak memory, and cost.
255+
Repeat at least 20 times before considering another non-Docker job.
256+
257+
## Autoscaling gate
258+
259+
Do not add CREMA or move required checks until all of these are complete:
260+
261+
- Infra approves a GitHub App or token-broker lifecycle for fresh registration
262+
or JIT configuration;
263+
- no long-lived GitHub credential is available to job code;
264+
- the scaler counts both queued and in-progress jobs and cannot terminate an
265+
active job;
266+
- at least 100 synthetic jobs complete without `lost communication`, stale
267+
registrations, or cross-workflow scheduling;
268+
- rollback to GitHub-hosted runners and scale-to-zero are exercised.
269+
270+
Relevant references:
271+
272+
- [ASF self-hosted runner policy](https://infra.apache.org/self-hosted-runners.html)
273+
- [GitHub self-hosted runner security](https://docs.github.com/en/actions/reference/security/secure-use)
274+
- [GitHub ephemeral runner guidance](https://docs.github.com/en/actions/reference/runners/self-hosted-runners)
275+
- [Cloud Run GitHub runner tutorial](https://docs.cloud.google.com/run/docs/tutorials/github-runner)
276+
- [Cloud Run runtime contract](https://docs.cloud.google.com/run/docs/container-contract)
277+
- [Apache Beam token function](https://github.com/apache/beam/blob/master/.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/generateToken/index.js)
278+
- [CREMA scale-down issue](https://github.com/GoogleCloudPlatform/cloud-run-external-metrics-autoscaling/issues/6)

0 commit comments

Comments
 (0)