Support data persistent: integrate with storageClass

[ahrtr]

The goal is to ensure the etcd's data will not get lost once the VM or POD is rebooted.

So we need to support setting VolumeClaimTemplate for the statefulset, something like below,

  volumeClaimTemplates:
  - metadata:
      name: etcd-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "csi-sc-standard"
      resources:
        requests:
          storage: 32Gi

Note that it isn't etcd-operator's responsibility to provision / manage the CSI driver in production environment.

But we need to provision / manage a CSI driver for our test environment,

• one possible solution that I can think of is openebs localpv.

• or csi-driver-host-path for single node test environment.

• or use hostPath directly, something like below,

   volumes:
   - name: my-data
     hostPath:
       path: /data/${POD_NAME}
       type: DirectoryOrCreate
  

I am open to other alternatives.

[ahrtr]

This isn't a good first issue. It needs some experience on K8s and PV/PVC/StorageClass.

[liangyuanpeng]

But we need to provision / manage a CSI driver for our test environment,

etcd-operator is using Kind for CI, and Kind including a local path provider, so i think just use local-path provider from Kind is enough?

$ k get sc
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  15m

[ahrtr]

/assign @gdasson

[k8s-ci-robot]

@ahrtr: GitHub didn't allow me to assign the following users: gdasson.

Note that only etcd-io members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

Details

In response to this:

/assign @gdasson

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ahrtr]

I think that it would be better to support both file system and block volume. We leave it to users to use what kind of storage, but we need to ensure the configuration is flexible.

If users do not configure any storage, then we just use the container's default temporary storage; it will be useful for users to do quick evaluation or experiment in their test environment.

Specifically, users should be able to configure volumes or volumeClaimTemplates.

If users configure a volume, then it's shared by all PODs. In that case, we need to create a sub directory to avoid conflict. See example below. Usually it's the use case of file system storage, i.e. NFS. The administrator needs to provision & manage the PVC/PV themselves.

      containers:
        - name: etcd
          volumeMounts:
          - name: etcd-data-shared
            mountPath: /var/lib/etcd
            subPathExpr: $(POD_NAME)            
      volumes:
      - name: etcd-data-shared
        persistentVolumeClaim:
          claimName: etcd-pvc-shared

If users configure a VolumeClaimTemplate, then a PV is dynamically created (by the CSI driver) for each POD. Usually it's the use case of block volume storage. See example below,

  volumeClaimTemplates:
  - metadata:
      name: etcd-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "csi-sc-standard"
      resources:
        requests:
          storage: 32Gi

For the volumeMounts, we don't need to use subPath, nor `subPathExpr`. 
But to keep it as simple as possible, we can keep it consistent with the volumes case above.

For now, we can define a struct StorageSpec, something like below. In future, we might need to provide more flexibility, to let users configure the volumes and VolumeClaimTemplate directly, just as how they configure a statefulsets directly.

type StorageSpec struct {
    AccessModes string // `ReadWriteOnce` (default) or `ReadWriteMany`. Note that `ReadOnlyMany` isn't allowed.
    StorageClassName string // optional, the default one will be used if not specified
    VolumeSize resource.Quantity // required.
}

and add a field Storage *StorageSpec into EtcdClusterSpec.

• If the AccessModes is ReadWriteMany, then we define the volumes.
• If the AccessModes is ReadWriteOnce (it's also the default option), then we define a VolumeClaimTemplate.
• If users do not configure StorageSpec at all, then each POD just uses the temporary storage inside the container. As mentioned above, It's not recommended in production, but will be useful for users to evaluate or experiment the etcd operator.

[gdasson]

Please assign

[ahrtr]

/assign @gdasson

[ahrtr]

@gdasson are you still working on this?

[gdasson]

@ahrtr : Yes, I am. Will update the PR with the relevant comment this weekend so that we can discuss this in our bi-weekly meeting. Thanks.

[luismacosta]

I think that it would be better to support both file system and block volume. We leave it to users to use what kind of storage, but we need to ensure the configuration is flexible.

If users do not configure any storage, then we just use the container's default temporary storage; it will be useful for users to do quick evaluation or experiment in their test environment.

Specifically, users should be able to configure volumes or volumeClaimTemplates.

If users configure a volume, then it's shared by all PODs. In that case, we need to create a sub directory to avoid conflict. See example below. Usually it's the use case of file system storage, i.e. NFS. The administrator needs to provision & manage the PVC/PV themselves.

      containers:
        - name: etcd
          volumeMounts:
          - name: etcd-data-shared
            mountPath: /var/lib/etcd
            subPathExpr: $(POD_NAME)            
      volumes:
      - name: etcd-data-shared
        persistentVolumeClaim:
          claimName: etcd-pvc-shared

If users configure a VolumeClaimTemplate, then a PV is dynamically created (by the CSI driver) for each POD. Usually it's the use case of block volume storage. See example below,

  volumeClaimTemplates:
  - metadata:
      name: etcd-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "csi-sc-standard"
      resources:
        requests:
          storage: 32Gi

For the volumeMounts, we don't need to use subPath, nor `subPathExpr`. 
But to keep it as simple as possible, we can keep it consistent with the volumes case above.

For now, we can define a struct StorageSpec, something like below. In future, we might need to provide more flexibility, to let users configure the volumes and VolumeClaimTemplate directly, just as how they configure a statefulsets directly.

type StorageSpec struct {
    AccessModes string // `ReadWriteOnce` (default) or `ReadWriteMany`. Note that `ReadOnlyMany` isn't allowed.
    StorageClassName string // optional, the default one will be used if not specified
    VolumeSize resource.Quantity // required.
}

and add a field Storage *StorageSpec into EtcdClusterSpec.

• If the AccessModes is ReadWriteMany, then we define the volumes.
• If the AccessModes is ReadWriteOnce (it's also the default option), then we define a VolumeClaimTemplate.
• If users do not configure StorageSpec at all, then each POD just uses the temporary storage inside the container. As mentioned above, It's not recommended in production, but will be useful for users to evaluate or experiment the etcd operator.

Hello,

If the AccessModes is ReadWriteMany, VolumeClaimTemplate should also be created, in case users want to define storageClassName as nfs, in order to use aws-efs-csi-driver to create the access points in Amazon EFS file system