Skip to content

feat: delete statefulsets if volumeClaimTemplates changes #3917

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: delete statefulsets if volumeClaimTemplates changes #3917

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions charts/tempo-distributed/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -596,8 +596,9 @@ The memcached default args are removed and should be provided manually. The sett
| ingester.image.tag | string | `nil` | Docker image tag for the ingester image. Overrides `tempo.image.tag` |
| ingester.initContainers | list | `[]` | |
| ingester.nodeSelector | object | `{}` | Node selector for ingester pods |
| ingester.persistence | object | `{"annotations":{},"enabled":false,"inMemory":false,"labels":{},"size":"10Gi","storageClass":null}` | Persistence configuration for ingester |
| ingester.persistence | object | `{"annotations":{},"enableStatefulSetRecreationForSizeChange":false,"enabled":false,"inMemory":false,"labels":{},"size":"10Gi","storageClass":null}` | Persistence configuration for ingester |
| ingester.persistence.annotations | object | `{}` | Annotations for ingester's persist volume claim |
| ingester.persistence.enableStatefulSetRecreationForSizeChange | bool | `false` | Enable StatefulSetRecreation for changes to PVC size |
| ingester.persistence.enabled | bool | `false` | Enable creating PVCs which is required when using boltdb-shipper |
| ingester.persistence.inMemory | bool | `false` | use emptyDir with ramdisk instead of PVC. **Please note that all data in ingester will be lost on pod restart** |
| ingester.persistence.labels | object | `{}` | Labels for ingester's persist volume claim |
Expand Down Expand Up @@ -745,8 +746,9 @@ The memcached default args are removed and should be provided manually. The sett
| metricsGenerator.maxUnavailable | int | `1` | Pod Disruption Budget maxUnavailable |
| metricsGenerator.minReadySeconds | int | `10` | Minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing/terminating |
| metricsGenerator.nodeSelector | object | `{}` | Node selector for metrics-generator pods |
| metricsGenerator.persistence | object | `{"annotations":{},"enabled":false,"labels":{},"size":"10Gi","storageClass":null}` | Persistence configuration for metrics-generator |
| metricsGenerator.persistence | object | `{"annotations":{},"enableStatefulSetRecreationForSizeChange":false,"enabled":false,"labels":{},"size":"10Gi","storageClass":null}` | Persistence configuration for metrics-generator |
| metricsGenerator.persistence.annotations | object | `{}` | Annotations for metrics generator PVCs |
| metricsGenerator.persistence.enableStatefulSetRecreationForSizeChange | bool | `false` | Enable StatefulSetRecreation for changes to PVC size. This means that the StatefulSet will be deleted, recreated (with the same name) and rolled when a change to the PVC size is detected. That way the PVC can be resized without manual intervention. |
| metricsGenerator.persistence.enabled | bool | `false` | Enable creating PVCs if you have kind set to StatefulSet. This disables using local disk or memory configured in walEmptyDir |
| metricsGenerator.persistence.labels | object | `{}` | Labels for metrics generator PVCs |
| metricsGenerator.persistence.storageClass | string | `nil` | Storage class to be used. If defined, storageClassName: <storageClass>. If set to "-", storageClassName: "", which disables dynamic provisioning. If empty or set to null, no storageClassName spec is set, choosing the default provisioner (gp2 on AWS, standard on GKE, AWS, and OpenStack). |
Expand Down
124 changes: 124 additions & 0 deletions charts/tempo-distributed/templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,3 +298,127 @@ Cluster name that shows up in dashboard metrics
{{- define "tempo.clusterName" -}}
{{ (include "tempo.calculatedConfig" . | fromYaml).cluster_name | default .Release.Name }}
{{- end -}}

{{- define "tempo.statefulset.recreateOnSizeChangeHook" -}}
{{- $renderedStatefulSets := list -}}
{{- range $renderedStatefulSet := include (print .context.Template.BasePath .pathToStatefulsetTemplate) .context | splitList "---" -}}
{{- with $renderedStatefulSet | fromYaml -}}
{{- $renderedStatefulSets = append $renderedStatefulSets . -}}
{{- end }}
{{- end -}}
{{- if $renderedStatefulSets }}
{{- range $newStatefulSet := $renderedStatefulSets -}}
{{- if $newStatefulSet.spec.volumeClaimTemplates }}
{{- $currentStatefulset := lookup $newStatefulSet.apiVersion $newStatefulSet.kind $newStatefulSet.metadata.namespace $newStatefulSet.metadata.name -}}
{{- $needsRecreation := false -}}
{{- if $currentStatefulset -}}
{{- if ne (len $newStatefulSet.spec.volumeClaimTemplates) (len $currentStatefulset.spec.volumeClaimTemplates) -}}
{{- $needsRecreation = true -}}
{{- else -}}
{{- range $index, $newVolumeClaimTemplate := $newStatefulSet.spec.volumeClaimTemplates -}}
{{- $currentVolumeClaimTemplateSpec := dict -}}
{{- range $oldVolumeClaimTemplate := $currentStatefulset.spec.volumeClaimTemplates -}}
{{- if eq $oldVolumeClaimTemplate.metadata.name $newVolumeClaimTemplate.metadata.name -}}
{{- $currentVolumeClaimTemplateSpec = $oldVolumeClaimTemplate.spec -}}
{{- end -}}
{{- end }}
{{- if or (not $currentVolumeClaimTemplateSpec) (ne $newVolumeClaimTemplate.spec.resources.requests.storage $currentVolumeClaimTemplateSpec.resources.requests.storage) -}}
{{- $needsRecreation = true -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- if $needsRecreation }}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ $newStatefulSet.metadata.name }}-recreate
namespace: {{ $newStatefulSet.metadata.namespace }}
labels:
{{- $newStatefulSet.metadata.labels | toYaml | nindent 4 }}
app.kubernetes.io/component: statefulset-recreate-job
annotations:
"helm.sh/hook": pre-upgrade
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
ttlSecondsAfterFinished: 300
template:
metadata:
name: {{ $newStatefulSet.metadata.name }}-recreate
labels:
{{- $newStatefulSet.metadata.labels | toYaml | nindent 8 }}
spec:
serviceAccountName: {{ $newStatefulSet.metadata.name }}-recreate
restartPolicy: OnFailure
containers:
- name: recreate
image: {{ printf "%s/rancher/kubectl:%s" (coalesce $.context.Values.global.image.registry "docker.io") $.context.Capabilities.KubeVersion.Version }}
command:
- kubectl
- delete
- statefulset
- {{ $newStatefulSet.metadata.name }}
- --cascade=orphan
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ $newStatefulSet.metadata.name }}-recreate
namespace: {{ $newStatefulSet.metadata.namespace }}
labels:
{{- $newStatefulSet.metadata.labels | toYaml | nindent 4 }}
app.kubernetes.io/component: statefulset-recreate-job
annotations:
"helm.sh/hook": pre-upgrade
"helm.sh/hook-weight": "-10"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ $newStatefulSet.metadata.name }}-recreate
namespace: {{ $newStatefulSet.metadata.namespace }}
labels:
{{- $newStatefulSet.metadata.labels | toYaml | nindent 4 }}
app.kubernetes.io/component: statefulset-recreate-job
annotations:
"helm.sh/hook": pre-upgrade
"helm.sh/hook-weight": "-10"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
rules:
- apiGroups:
- apps
resources:
- statefulsets
resourceNames:
- {{ $newStatefulSet.metadata.name }}
verbs:
- delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ $newStatefulSet.metadata.name }}-recreate
namespace: {{ $newStatefulSet.metadata.namespace }}
labels:
{{- $newStatefulSet.metadata.labels | toYaml | nindent 4 }}
app.kubernetes.io/component: statefulset-recreate-job
annotations:
"helm.sh/hook": pre-upgrade
"helm.sh/hook-weight": "-10"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
subjects:
- kind: ServiceAccount
name: {{ $newStatefulSet.metadata.name }}-recreate
namespace: {{ $newStatefulSet.metadata.namespace }}
roleRef:
kind: Role
name: {{ $newStatefulSet.metadata.name }}-recreate
apiGroup: rbac.authorization.k8s.io
---
{{- end }}
{{ end }}
{{ end }}
{{ end }}
{{- end -}}
3 changes: 3 additions & 0 deletions charts/tempo-distributed/templates/ingester/statefulset-ingester-recreate-job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{{- if and .Values.ingester.persistence.enabled .Values.ingester.persistence.enableStatefulSetRecreationForSizeChange -}}
{{- include "tempo.statefulset.recreateOnSizeChangeHook" (dict "context" . "pathToStatefulsetTemplate" "/ingester/statefulset-ingester.yaml") -}}
{{- end -}}
3 changes: 3 additions & 0 deletions charts/tempo-distributed/templates/ingester/statefulset-ingester.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ spec:
{{- end }}
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap-tempo.yaml") . | sha256sum }}
{{- if .Values.ingester.persistence.enabled }}
storage/size: {{ .Values.ingester.persistence.size | quote }}
{{- end }}
{{- with .Values.tempo.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
Expand Down
3 changes: 3 additions & 0 deletions ...o-distributed/templates/metrics-generator/statefulset-metrics-generator-recreate-job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{{- if and .Values.metricsGenerator.persistence.enabled .Values.metricsGenerator.persistence.enableStatefulSetRecreationForSizeChange -}}
{{- include "tempo.statefulset.recreateOnSizeChangeHook" (dict "context" . "pathToStatefulsetTemplate" "/metrics-generator/statefulset-metrics-generator.yaml") -}}
{{- end -}}
3 changes: 3 additions & 0 deletions charts/tempo-distributed/templates/metrics-generator/statefulset-metrics-generator.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ spec:
{{- end }}
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap-tempo.yaml") . | sha256sum }}
{{- if .Values.metricsGenerator.persistence.enabled }}
storage/size: {{ .Values.metricsGenerator.persistence.size | quote }}
{{- end }}
{{- with .Values.tempo.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
Expand Down
6 changes: 6 additions & 0 deletions charts/tempo-distributed/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,8 @@ ingester:
persistence:
# -- Enable creating PVCs which is required when using boltdb-shipper
enabled: false
# -- Enable StatefulSetRecreation for changes to PVC size
enableStatefulSetRecreationForSizeChange: false
# -- use emptyDir with ramdisk instead of PVC. **Please note that all data in ingester will be lost on pod restart**
inMemory: false
# -- Size of persistent or memory disk
Expand Down Expand Up @@ -401,6 +403,10 @@ metricsGenerator:
persistence:
# -- Enable creating PVCs if you have kind set to StatefulSet. This disables using local disk or memory configured in walEmptyDir
enabled: false
# -- Enable StatefulSetRecreation for changes to PVC size.
# This means that the StatefulSet will be deleted, recreated (with the same name) and rolled when a change to the
# PVC size is detected. That way the PVC can be resized without manual intervention.
enableStatefulSetRecreationForSizeChange: false
size: 10Gi
# -- Storage class to be used.
# If defined, storageClassName: <storageClass>.
Expand Down