Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: rootless dind #521

Merged
merged 23 commits into from
Nov 18, 2024
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
35ae7ed
fix: permissions for rootless dind
mikhail-klimko Nov 13, 2024
905c261
fix: permissions for rootless dind
mikhail-klimko Nov 13, 2024
89ce7e5
fix: rootless dind
mikhail-klimko Nov 13, 2024
2e0421c
wip: Thu Nov 14 14:00:42 +03 2024
mikhail-klimko Nov 14, 2024
6716475
wip: Thu Nov 14 19:07:51 +03 2024
mikhail-klimko Nov 14, 2024
c01aeaf
wip: Thu Nov 14 19:09:10 +03 2024
mikhail-klimko Nov 14, 2024
c19098c
wip: Thu Nov 14 19:11:35 +03 2024
mikhail-klimko Nov 14, 2024
4b3bead
wip: Thu Nov 14 19:37:25 +03 2024
mikhail-klimko Nov 14, 2024
62418b2
wip: Thu Nov 14 19:45:59 +03 2024
mikhail-klimko Nov 14, 2024
39860dc
wip: Thu Nov 14 19:53:58 +03 2024
mikhail-klimko Nov 14, 2024
f5a1b61
wip: Thu Nov 14 20:09:35 +03 2024
mikhail-klimko Nov 14, 2024
cfb7e45
wip: Thu Nov 14 20:11:10 +03 2024
mikhail-klimko Nov 14, 2024
060fa50
wip: Thu Nov 14 20:18:18 +03 2024
mikhail-klimko Nov 14, 2024
e5ecd9a
wip: Thu Nov 14 20:18:22 +03 2024
mikhail-klimko Nov 14, 2024
f5a0433
wip: Thu Nov 14 20:23:37 +03 2024
mikhail-klimko Nov 14, 2024
7a6b488
wip: Thu Nov 14 20:23:49 +03 2024
mikhail-klimko Nov 14, 2024
95d2fd9
wip: Thu Nov 14 20:28:10 +03 2024
mikhail-klimko Nov 14, 2024
c44e0bb
wip: Thu Nov 14 20:31:28 +03 2024
mikhail-klimko Nov 14, 2024
e0d61ef
wip: Fri Nov 15 10:46:12 +03 2024
mikhail-klimko Nov 15, 2024
4852e4a
helm-docs
mikhail-klimko Nov 18, 2024
7e955c4
wip: Mon Nov 18 12:02:31 +03 2024
mikhail-klimko Nov 18, 2024
468de3e
wip: Mon Nov 18 18:19:11 +03 2024
mikhail-klimko Nov 18, 2024
f4b0dfe
fix(venona-helm-chart/ci): add rootless check
mikhail-klimko Nov 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,10 @@ venona/venona
**/*.tgz
**/charts/**/charts
**/dry-run.yaml
**/values-dev.yaml
**/values-dev**.yaml

# coverage
**/cover

# debug
**/debug
**/.debug
36 changes: 36 additions & 0 deletions charts/cf-runtime/.ci/values-rootless.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
volumeProvisioner:
env:
IS_ROOTLESS: true
dind-lv-monitor:
image:
tag: 1.30.0-rootless
digest: ""
podSecurityContext:
enabled: true
runAsUser: 1000
fsGroup: 1000
volumePermissions:
enabled: true

runtime:
dindDaemon:
hosts:
- unix:///run/user/1000/docker.sock
- tcp://0.0.0.0:1300
dind:
image:
tag: 26.1.4-1.28.9-rootless
digest: ""
userVolumeMounts:
dind:
name: dind
mountPath: /home/rootless/
containerSecurityContext:
privileged: true
podSecurityContext:
enabled: true
runAsUser: 1000
fsGroup: 1000
fsGroupChangePolicy: "OnRootMismatch"
volumePermissions:
enabled: true
8 changes: 5 additions & 3 deletions charts/cf-runtime/Chart.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
apiVersion: v2
description: A Helm chart for Codefresh Runner
name: cf-runtime
version: 7.0.1
version: 7.1.0
keywords:
- codefresh
- runner
Expand All @@ -17,8 +17,10 @@ annotations:
artifacthub.io/containsSecurityUpdates: "false"
# Supported kinds: `added`, `changed`, `deprecated`, `removed`, `fixed`, `security`:
artifacthub.io/changes: |
- kind: security
description: "updating k8s-agent"
- kind: changed
description: "(rootless runtime) Update dind-volume-provisioner and dind-volume-utils"
- kind: added
description: "(rootless runtime) Add values-rootless.yaml example "
dependencies:
- name: cf-common
repository: oci://quay.io/codefresh/charts
Expand Down
43 changes: 37 additions & 6 deletions charts/cf-runtime/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## Codefresh Runner

![Version: 7.0.1](https://img.shields.io/badge/Version-7.0.1-informational?style=flat-square)
![Version: 7.1.0](https://img.shields.io/badge/Version-7.1.0-informational?style=flat-square)

Helm chart for deploying [Codefresh Runner](https://codefresh.io/docs/docs/installation/codefresh-runner/) to Kubernetes.

Expand Down Expand Up @@ -715,14 +715,45 @@ volumeProvisioner:
### Rootless DinD

DinD pod runs a `priviliged` container with **rootfull** docker.
To run the docker daemon as non-root user (**rootless** mode), change dind image tag:
To run the docker daemon as non-root user (**rootless** mode), refer to `values-rootless.yaml`:

`values.yaml`
```yaml
volumeProvisioner:
env:
IS_ROOTLESS: true
dind-lv-monitor:
image:
tag: 1.30.0-rootless
digest: ""
podSecurityContext:
enabled: true
runAsUser: 1000
fsGroup: 1000
volumePermissions:
enabled: false

runtime:
dindDaemon:
hosts:
- unix:///run/user/1000/docker.sock
- tcp://0.0.0.0:1300
dind:
image:
tag: rootless
tag: 26.1.4-1.28.9-rootless
digest: ""
userVolumeMounts:
dind:
name: dind
mountPath: /home/rootless/
containerSecurityContext:
privileged: true
podSecurityContext:
enabled: true
runAsUser: 1000
fsGroup: 1000
fsGroupChangePolicy: "OnRootMismatch"
volumePermissions:
enabled: false
```

### ARM
Expand Down Expand Up @@ -1143,7 +1174,7 @@ Go to [https://<YOUR_ONPREM_DOMAIN_HERE>/admin/runtime-environments/system](http
| runtime.accounts | list | `[]` | (for On-Premise only) Assign accounts to runtime (list of account ids) |
| runtime.agent | bool | `true` | (for On-Premise only) Enable agent |
| runtime.description | string | `""` | Runtime description |
| runtime.dind | object | `{"affinity":{},"env":{"DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE":true},"image":{"digest":"sha256:ccaf26ab24db0e00760beba79ce1810a12aef5be296f538ceab416af9ec481f7","pullPolicy":"IfNotPresent","registry":"quay.io","repository":"codefresh/dind","tag":"26.1.4-1.28.7"},"nodeSelector":{},"podAnnotations":{},"podLabels":{},"pvcs":{"dind":{"annotations":{},"name":"dind","reuseVolumeSelector":"codefresh-app,io.codefresh.accountName","reuseVolumeSortOrder":"pipeline_id","storageClassName":"{{ include \"dind-volume-provisioner.storageClassName\" . }}","volumeSize":"16Gi"}},"resources":{"limits":{"cpu":"400m","memory":"800Mi"},"requests":null},"schedulerName":"","serviceAccount":"codefresh-engine","terminationGracePeriodSeconds":30,"tolerations":[],"userAccess":true,"userVolumeMounts":{},"userVolumes":{}}` | Parameters for DinD (docker-in-docker) pod (aka "runtime" pod). |
| runtime.dind | object | `{"affinity":{},"env":{"DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE":true},"image":{"digest":"sha256:ccaf26ab24db0e00760beba79ce1810a12aef5be296f538ceab416af9ec481f7","pullPolicy":"IfNotPresent","registry":"quay.io","repository":"codefresh/dind","tag":"26.1.4-1.28.7"},"nodeSelector":{},"podAnnotations":{},"podLabels":{},"pvcs":{"dind":{"annotations":{},"name":"dind","reuseVolumeSelector":"codefresh-app,io.codefresh.accountName","reuseVolumeSortOrder":"pipeline_id","storageClassName":"{{ include \"dind-volume-provisioner.storageClassName\" . }}","volumeSize":"16Gi"}},"resources":{"limits":{"cpu":"400m","memory":"800Mi"},"requests":null},"schedulerName":"","serviceAccount":"codefresh-engine","terminationGracePeriodSeconds":30,"tolerations":[],"userAccess":true,"userVolumeMounts":{},"userVolumes":{},"volumePermissions":{"enabled":false,"image":{"digest":"sha256:2995c82e8e723d9a5c8585cb8e901d1c50e3c2759031027d3bff577449435157","registry":"docker.io","repository":"alpine","tag":3.18},"resources":{},"securityContext":{"runAsUser":0}}}` | Parameters for DinD (docker-in-docker) pod (aka "runtime" pod). |
| runtime.dind.affinity | object | `{}` | Set affinity |
| runtime.dind.env | object | `{"DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE":true}` | Set additional env vars. |
| runtime.dind.image | object | `{"digest":"sha256:ccaf26ab24db0e00760beba79ce1810a12aef5be296f538ceab416af9ec481f7","pullPolicy":"IfNotPresent","registry":"quay.io","repository":"codefresh/dind","tag":"26.1.4-1.28.7"}` | Set dind image. |
Expand Down Expand Up @@ -1234,7 +1265,7 @@ Go to [https://<YOUR_ONPREM_DOMAIN_HERE>/admin/runtime-environments/system](http
| volumeProvisioner.dind-lv-monitor | object | See below | `dind-lv-monitor` DaemonSet parameters (local volumes cleaner) |
| volumeProvisioner.enabled | bool | `true` | Enable volume-provisioner |
| volumeProvisioner.env | object | `{}` | Add additional env vars |
| volumeProvisioner.image | object | `{"digest":"sha256:c036ad717391debdf43f8da337b81b5df0e79de274d2d9af1425c675b0296dda","registry":"quay.io","repository":"codefresh/dind-volume-provisioner","tag":"1.35.0"}` | Set image |
| volumeProvisioner.image | object | `{"digest":"sha256:ede6f663c912a08b7d335b5ec5518ccc266b27c431d0854d22971005992adc5d","registry":"quay.io","repository":"codefresh/dind-volume-provisioner","tag":"1.35.2"}` | Set image |
| volumeProvisioner.nodeSelector | object | `{}` | Set node selector |
| volumeProvisioner.podAnnotations | object | `{}` | Set pod annotations |
| volumeProvisioner.podSecurityContext | object | See below | Set security context for the pod |
Expand Down
37 changes: 34 additions & 3 deletions charts/cf-runtime/README.md.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -717,14 +717,45 @@ volumeProvisioner:
### Rootless DinD

DinD pod runs a `priviliged` container with **rootfull** docker.
To run the docker daemon as non-root user (**rootless** mode), change dind image tag:
To run the docker daemon as non-root user (**rootless** mode), refer to `values-rootless.yaml`:

`values.yaml`
```yaml
volumeProvisioner:
env:
IS_ROOTLESS: true
dind-lv-monitor:
image:
tag: 1.30.0-rootless
digest: ""
podSecurityContext:
enabled: true
runAsUser: 1000
fsGroup: 1000
volumePermissions:
enabled: false

runtime:
dindDaemon:
hosts:
- unix:///run/user/1000/docker.sock
- tcp://0.0.0.0:1300
dind:
image:
tag: rootless
tag: 26.1.4-1.28.9-rootless
digest: ""
userVolumeMounts:
dind:
name: dind
mountPath: /home/rootless/
containerSecurityContext:
privileged: true
podSecurityContext:
enabled: true
runAsUser: 1000
fsGroup: 1000
fsGroupChangePolicy: "OnRootMismatch"
volumePermissions:
enabled: false
```

### ARM
Expand Down
28 changes: 28 additions & 0 deletions charts/cf-runtime/templates/runtime/runtime-env-spec-tmpl.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,34 @@ dockerDaemonScheduler:
secret:
secretName: codefresh-certs-server
{{- end }}
{{- with $dindContext.podSecurityContext }}
podSecurityContext: {{- toYaml . | nindent 4 }}
{{- end }}
{{- with $dindContext.containerSecurityContext }}
containerSecurityContext: {{- toYaml . | nindent 4 }}
{{- end }}
{{- if $dindContext.volumePermissions.enabled }}
initContainers:
- name: volume-permissions
image: {{ include (printf "%s.image.name" $cfCommonTplSemver ) (dict "image" $dindContext.volumePermissions.image "context" .) }}
imagePullPolicy: {{ $dindContext.volumePermissions.image.pullPolicy | default "Always" }}
command:
- /bin/sh
args:
- -ec
- |
chown -R {{ $dindContext.podSecurityContext.runAsUser }}:{{ $dindContext.podSecurityContext.fsGroup }} /home/rootless/
volumeMounts:
- mountPath: /home/rootless/
name: dind
{{- if eq ( toString ( $dindContext.volumePermissions.securityContext.runAsUser )) "auto" }}
securityContext: {{- omit $dindContext.volumePermissions.securityContext "runAsUser" | toYaml | nindent 6 }}
{{- else }}
securityContext: {{- $dindContext.volumePermissions.securityContext | toYaml | nindent 6 }}
{{- end }}
resources:
{{- toYaml $dindContext.volumePermissions.resources | nindent 6 }}
{{- end }}
extends: {{- toYaml .Values.runtime.runtimeExtends | nindent 2 }}
{{- if .Values.runtime.description }}
description: {{ .Values.runtime.description }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,15 @@ values:
- ../values.yaml
- ../values-private-registry.yaml
templates:
- templates/hooks/post-install/cm-update-runtime.yaml
- templates/runner/deployment.yaml
- templates/volume-provisioner/deployment.yaml
- templates/volume-provisioner/daemonset.yaml
- templates/volume-provisioner/cronjob.yaml
- templates/monitor/deployment.yaml
- templates/app-proxy/deployment.yaml
- templates/**.yaml
release:
name: cf-runtime
namespace: codefresh
revision: 1
upgrade: true
chart:
version: 1.0.0
appVersion: 1.0.0
# chart:
# version: 1.0.0
# appVersion: 1.0.0
tests:
- it: Test private registry in runtime spec
template: templates/hooks/post-install/cm-update-runtime.yaml
Expand Down
11 changes: 3 additions & 8 deletions charts/cf-runtime/tests/runner/runner_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,13 @@ suite: runner test
values:
- ../values.yaml
templates:
- templates/runner/deployment.yaml
- templates/runner/rbac.yaml
- templates/runner/secret.yaml
- templates/**.yaml
release:
name: cf-runtime
namespace: codefresh
revision: 1
upgrade: true
chart:
version: 1.0.0
appVersion: 1.0.0
tests:
- it: Test runner default metadata
Expand All @@ -24,14 +21,12 @@ tests:
of: Deployment
- isNull:
path: metadata.annotations
- equal:
- isSubset:
path: metadata.labels
value:
content:
app.kubernetes.io/instance: cf-runtime
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: cf-runtime
app.kubernetes.io/version: 1.0.0
helm.sh/chart: cf-runtime-1.0.0
codefresh.io/application: runner
- equal:
path: metadata.name
Expand Down
5 changes: 1 addition & 4 deletions charts/cf-runtime/tests/runtime/runtime_onprem_test.yaml
Original file line number Diff line number Diff line change
@@ -1,16 +1,13 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/quintush/helm-unittest/master/schema/helm-testsuite.json
suite: runtime onprem test
templates:
- templates/hooks/post-install/job-update-runtime.yaml
- templates/hooks/post-install/cm-update-runtime.yaml
- templates/runtime/secret.yaml
- templates/**.yaml
release:
name: cf-runtime
namespace: codefresh
revision: 1
upgrade: true
chart:
version: 1.0.0
appVersion: 1.0.0
tests:
- it: Test default runtime spec metadata
Expand Down
5 changes: 1 addition & 4 deletions charts/cf-runtime/tests/runtime/runtime_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,13 @@ suite: runtime test
values:
- ../values.yaml
templates:
- templates/hooks/post-install/job-update-runtime.yaml
- templates/hooks/post-install/cm-update-runtime.yaml
- templates/runtime/secret.yaml
- templates/**.yaml
release:
name: cf-runtime
namespace: codefresh
revision: 1
upgrade: true
chart:
version: 1.0.0
appVersion: 1.0.0
tests:
- it: Test default runtime spec metadata
Expand Down
10 changes: 3 additions & 7 deletions charts/cf-runtime/tests/volume-provisioner/cronjob_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,13 @@ suite: dind-volume-cleanup test
values:
- ../values.yaml
templates:
- templates/volume-provisioner/cronjob.yaml
- templates/volume-provisioner/storageclass.yaml
- templates/**.yaml
release:
name: cf-runtime
namespace: codefresh
revision: 1
upgrade: true
chart:
version: 1.0.0
appVersion: 1.0.0
tests:
- it: Test dind-volume-cleanup default metadata
Expand All @@ -25,14 +23,12 @@ tests:
of: CronJob
- isNull:
path: metadata.annotations
- equal:
- isSubset:
path: metadata.labels
value:
content:
app.kubernetes.io/instance: cf-runtime
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: cf-runtime
app.kubernetes.io/version: 1.0.0
helm.sh/chart: cf-runtime-1.0.0
codefresh.io/application: pv-cleanup
- equal:
path: metadata.name
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,13 @@ suite: dind-lv-monitor test
values:
- ../values.yaml
templates:
- templates/volume-provisioner/daemonset.yaml
- templates/**.yaml
release:
name: cf-runtime
namespace: codefresh
revision: 1
upgrade: true
chart:
version: 1.0.0
appVersion: 1.0.0
tests:
- it: Test dind-lv-monitor default metadata
Expand All @@ -24,14 +23,12 @@ tests:
of: DaemonSet
- isNull:
path: metadata.annotations
- equal:
- isSubset:
path: metadata.labels
value:
content:
app.kubernetes.io/instance: cf-runtime
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: cf-runtime
app.kubernetes.io/version: 1.0.0
helm.sh/chart: cf-runtime-1.0.0
codefresh.io/application: lv-monitor
- equal:
path: metadata.name
Expand Down
Loading