Add persistent volume to statefulset ingester failed

[Mojito2110]

Hello all,

I get an issue when I uprgade Ingester to add perstitent volume.

The first install of Mimir in my test environment, i disabled persistence for Ingester replicas. When I migrate on the latest version (2.2.0) using helm charts, I decide to add the persistence :

ingester:
  persistentVolume:
    #enabled: false
    enabled: true
    size: 2Gi
    storageClass: "t-a-g1000-t6m-e0f7-nosave"
  replicas: 3
  resources:
    limits:
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 512Mi
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target
                operator: In
                values:
                  - ingester
          topologyKey: 'kubernetes.io/hostname'

But the upgrade is unsuccessful with the message :

Error: UPGRADE FAILED: cannot patch "my-mimir-ingester" with kind StatefulSet: StatefulSet.apps "my-mimir-ingester" is invalid: spec: Forbidden: updates to statefulset spec for 
fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

What is wrong ?
Thanks in advance for your help

[56quarters]

Hello!

Kubernetes doesn't allow you to modify statefulsets in that way. You'll have to recreate them including persistence from the start.

[Mojito2110]

Ok thanks,

I delete statefulsets

kubectl delete statefulset my-mimir-ingester --cascade=false

And I relaunch the helm update, the 3 ingesters replicas are created but only one persistentvolumeclaim is created and sticky with the state 'pending'

I am not expert in Kubernetes :-( but I thought that the persistent volumes would be created dynamically

do I need reinstall completely Mimir or it's a misunterstanding about PV on Kubernetes ?

[Mojito2110]

I answer to myself,

I need to delete statefulsets and also the pods and then relaunch helm update.