diff --git a/.github/workflows/push.yml b/.github/workflows/push.yml index 47557f447f..5329206827 100644 --- a/.github/workflows/push.yml +++ b/.github/workflows/push.yml @@ -38,7 +38,6 @@ jobs: - name: 'use gcloud CLI' run: | - gcloud auth login gcloud info - name: Set up QEMU diff --git a/design/volume-snapshot-data-movement/volume-snapshot-data-movement.md b/design/volume-snapshot-data-movement/volume-snapshot-data-movement.md index d04a6d5b3e..f8c678dbf6 100644 --- a/design/volume-snapshot-data-movement/volume-snapshot-data-movement.md +++ b/design/volume-snapshot-data-movement/volume-snapshot-data-movement.md @@ -84,14 +84,14 @@ Below are actions from Velero and DMP: **BIA Execute** This is the existing logic in Velero. For a source PVC/PV, Velero delivers it to the corresponding BackupItemAction plugin, the plugin then takes the related actions to back it up. For example, the existing CSI plugin takes a CSI snapshot to the volume represented by the PVC and then returns additional items (i.e., VolumeSnapshot, VolumeSnapshotContent and VolumeSnapshotClass) for Velero to further backup. -To support data movement, we will use BIA V2 which supports asynchronized operation management. Here is the Execute method from BIA V2: +To support data movement, we will use BIA V2 which supports asynchronous operation management. Here is the Execute method from BIA V2: ``` Execute(item runtime.Unstructured, backup *api.Backup) (runtime.Unstructured, []velero.ResourceIdentifier, string, []velero.ResourceIdentifier, error) ``` Besides ```additionalItem``` (as the 2nd return value), Execute method will return one more resource list called ```itemToUpdate```, which means the items to be updated and persisted when the async operation completes. For details, visit [general progress monitoring design][2]. Specifically, this mechanism will be used to persist DUCR into the persisted backup data, in another words, DUCR will be returned as ```itemToUpdate``` from Execute method. DUCR contains all the information the restore requires, so during restore, DUCR will be extracted from the backup data. Additionally, in the same way, a DMP could add any other items into the persisted backup data. -Execute method also returns the ```operationID``` which uniquely identifies the asynchronized operation. This ```operationID``` is generated by plugins. The [general progress monitoring design][2] doesn't restrict the format of the ```operationID```, for Velero CSI plugin, the ```operationID``` is a combination of the backup CR UID and the source PVC (represented by the ```item``` parameter) UID. +Execute method also returns the ```operationID``` which uniquely identifies the asynchronous operation. This ```operationID``` is generated by plugins. The [general progress monitoring design][2] doesn't restrict the format of the ```operationID```, for Velero CSI plugin, the ```operationID``` is a combination of the backup CR UID and the source PVC (represented by the ```item``` parameter) UID. **Create Snapshot** The DMP creates a snapshot of the requested volume and deliver it to DM through DUCR. After that, the DMP leaves the snapshot and its related objects (e.g., VolumeSnapshot and VolumeSnapshotContent for CSI snapshot) to the DM, DM then has full control of the snapshot and its related objects, i.e., deciding whether to delete the snapshot or its related objects and when to do it. @@ -930,12 +930,20 @@ For 3, Velero leverage on DMs to decide how to save the log, but they will not g ## Installation DMs need to be configured during installation so that they can be installed. Plugin DMs may have their own configuration, for VGDM, the only requirement is to install Velero node-agent. Moreover, the DMP is also required during the installation. + +From release-1.14, the `github.com/vmware-tanzu/velero-plugin-for-csi` repository, which is the Velero CSI plugin, is merged into the `github.com/vmware-tanzu/velero` repository. +The reason to merge the CSI plugin is: +* The VolumeSnapshot data mover depends on the CSI plugin, it's reasonabe to integrate them. +* This change reduces the Velero deploying complexity. +* This makes performance tuning easier in the future. + +As a result, no need to install Velero CSI plugin anymore. + For example, to move CSI snapshot through VBDM, below is the installation script: ``` velero install \ --provider \ --image \ - --plugins velero/velero-plugin-for-csi:xxx \ --features=EnableCSI \ --use-node-agent \ ``` diff --git a/pkg/backup/actions/csi/volumesnapshot_action.go b/pkg/backup/actions/csi/volumesnapshot_action.go index 64e38c3c74..9a07ec7655 100644 --- a/pkg/backup/actions/csi/volumesnapshot_action.go +++ b/pkg/backup/actions/csi/volumesnapshot_action.go @@ -224,8 +224,8 @@ func (p *volumeSnapshotBackupItemAction) Execute( return nil, nil, "", nil, errors.WithStack(err) } - p.log.Infof("Returning from VolumeSnapshotBackupItemAction", - "with %d additionalItems to backup", len(additionalItems)) + p.log.Infof("Returning from VolumeSnapshotBackupItemAction with %d additionalItems to backup", + len(additionalItems)) for _, ai := range additionalItems { p.log.Debugf("%s: %s", ai.GroupResource.String(), ai.Name) } @@ -294,22 +294,20 @@ func (p *volumeSnapshotBackupItemAction) Progress( } if vs.Status == nil { - p.log.Debugf("VolumeSnapshot %s/%s has an empty status.", - "Skip progress update.", vs.Namespace, vs.Name) + p.log.Debugf("VolumeSnapshot %s/%s has an empty status. Skip progress update.", vs.Namespace, vs.Name) return progress, nil } if boolptr.IsSetToTrue(vs.Status.ReadyToUse) { - p.log.Debugf("VolumeSnapshot %s/%s is ReadyToUse.", - "Continue on querying corresponding VolumeSnapshotContent.", + p.log.Debugf("VolumeSnapshot %s/%s is ReadyToUse. Continue on querying corresponding VolumeSnapshotContent.", vs.Namespace, vs.Name) } else if vs.Status.Error != nil { errorMessage := "" if vs.Status.Error.Message != nil { errorMessage = *vs.Status.Error.Message } - p.log.Warnf("VolumeSnapshot has a temporary error %s.", - "Snapshot controller will retry later.", errorMessage) + p.log.Warnf("VolumeSnapshot has a temporary error %s. Snapshot controller will retry later.", + errorMessage) return progress, nil } @@ -328,8 +326,8 @@ func (p *volumeSnapshotBackupItemAction) Progress( } if vsc.Status == nil { - p.log.Debugf("VolumeSnapshotContent %s has an empty Status.", - "Skip progress update.", vsc.Name) + p.log.Debugf("VolumeSnapshotContent %s has an empty Status. Skip progress update.", + vsc.Name) return progress, nil } diff --git a/pkg/cmd/server/server.go b/pkg/cmd/server/server.go index fac74b1ddd..c98dd0f35e 100644 --- a/pkg/cmd/server/server.go +++ b/pkg/cmd/server/server.go @@ -284,7 +284,6 @@ type server struct { mgr manager.Manager credentialFileStore credentials.FileStore credentialSecretStore credentials.SecretStore - featureVerifier features.Verifier } func newServer(f client.Factory, config serverConfig, logger *logrus.Logger) (*server, error) { @@ -326,12 +325,6 @@ func newServer(f client.Factory, config serverConfig, logger *logrus.Logger) (*s return nil, err } - featureVerifier := features.NewVerifier(pluginRegistry) - - if _, err := featureVerifier.Verify(velerov1api.CSIFeatureFlag); err != nil { - logger.WithError(err).Warn("CSI feature verification failed, the feature may not be ready.") - } - // cancelFunc is not deferred here because if it was, then ctx would immediately // be canceled once this function exited, making it useless to any informers using later. // That, in turn, causes the velero server to halt when the first informer tries to use it. @@ -425,7 +418,6 @@ func newServer(f client.Factory, config serverConfig, logger *logrus.Logger) (*s mgr: mgr, credentialFileStore: credentialFileStore, credentialSecretStore: credentialSecretStore, - featureVerifier: featureVerifier, } return s, nil @@ -973,7 +965,6 @@ func (s *server) runControllers(defaultVolumeSnapshotLocations map[string]string s.kubeClient.CoreV1().RESTClient(), s.credentialFileStore, s.mgr.GetClient(), - s.featureVerifier, ) cmd.CheckError(err) diff --git a/pkg/features/mocks/PluginFinder.go b/pkg/features/mocks/PluginFinder.go deleted file mode 100644 index 7659c17720..0000000000 --- a/pkg/features/mocks/PluginFinder.go +++ /dev/null @@ -1,43 +0,0 @@ -// Code generated by mockery v2.22.1. DO NOT EDIT. - -package mocks - -import ( - common "github.com/vmware-tanzu/velero/pkg/plugin/framework/common" - - mock "github.com/stretchr/testify/mock" -) - -// PluginFinder is an autogenerated mock type for the PluginFinder type -type PluginFinder struct { - mock.Mock -} - -// Find provides a mock function with given fields: kind, name -func (_m *PluginFinder) Find(kind common.PluginKind, name string) bool { - ret := _m.Called(kind, name) - - var r0 bool - if rf, ok := ret.Get(0).(func(common.PluginKind, string) bool); ok { - r0 = rf(kind, name) - } else { - r0 = ret.Get(0).(bool) - } - - return r0 -} - -type mockConstructorTestingTNewPluginFinder interface { - mock.TestingT - Cleanup(func()) -} - -// NewPluginFinder creates a new instance of PluginFinder. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations. -func NewPluginFinder(t mockConstructorTestingTNewPluginFinder) *PluginFinder { - mock := &PluginFinder{} - mock.Mock.Test(t) - - t.Cleanup(func() { mock.AssertExpectations(t) }) - - return mock -} diff --git a/pkg/features/mocks/Verifier.go b/pkg/features/mocks/Verifier.go deleted file mode 100644 index e391345a26..0000000000 --- a/pkg/features/mocks/Verifier.go +++ /dev/null @@ -1,49 +0,0 @@ -// Code generated by mockery v2.22.1. DO NOT EDIT. - -package mocks - -import mock "github.com/stretchr/testify/mock" - -// Verifier is an autogenerated mock type for the Verifier type -type Verifier struct { - mock.Mock -} - -// Verify provides a mock function with given fields: name -func (_m *Verifier) Verify(name string) (bool, error) { - ret := _m.Called(name) - - var r0 bool - var r1 error - if rf, ok := ret.Get(0).(func(string) (bool, error)); ok { - return rf(name) - } - if rf, ok := ret.Get(0).(func(string) bool); ok { - r0 = rf(name) - } else { - r0 = ret.Get(0).(bool) - } - - if rf, ok := ret.Get(1).(func(string) error); ok { - r1 = rf(name) - } else { - r1 = ret.Error(1) - } - - return r0, r1 -} - -type mockConstructorTestingTNewVerifier interface { - mock.TestingT - Cleanup(func()) -} - -// NewVerifier creates a new instance of Verifier. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations. -func NewVerifier(t mockConstructorTestingTNewVerifier) *Verifier { - mock := &Verifier{} - mock.Mock.Test(t) - - t.Cleanup(func() { mock.AssertExpectations(t) }) - - return mock -} diff --git a/pkg/features/verify.go b/pkg/features/verify.go deleted file mode 100644 index 0474d50766..0000000000 --- a/pkg/features/verify.go +++ /dev/null @@ -1,71 +0,0 @@ -/* -Copyright the Velero contributors. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package features - -import ( - "errors" - - velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" - "github.com/vmware-tanzu/velero/pkg/plugin/framework/common" -) - -type PluginFinder interface { - Find(kind common.PluginKind, name string) bool -} - -type Verifier interface { - Verify(name string) (bool, error) -} - -type verifier struct { - finder PluginFinder -} - -func NewVerifier(finder PluginFinder) Verifier { - return &verifier{ - finder: finder, - } -} - -func (v *verifier) Verify(name string) (bool, error) { - enabled := IsEnabled(name) - - switch name { - case velerov1api.CSIFeatureFlag: - return verifyCSIFeature(v.finder, enabled) - default: - return false, nil - } -} - -func verifyCSIFeature(finder PluginFinder, enabled bool) (bool, error) { - installed := false - installed = finder.Find(common.PluginKindBackupItemActionV2, "velero.io/csi-pvc-backupper") - if installed { - installed = finder.Find(common.PluginKindRestoreItemActionV2, "velero.io/csi-pvc-restorer") - } - - if !enabled && installed { - return false, errors.New("CSI plugins are registered, but the EnableCSI feature is not enabled") - } else if enabled && !installed { - return false, errors.New("CSI feature is enabled, but CSI plugins are not registered") - } else if !enabled && !installed { - return false, nil - } else { - return true, nil - } -} diff --git a/pkg/features/verify_test.go b/pkg/features/verify_test.go deleted file mode 100644 index fee6b2b2df..0000000000 --- a/pkg/features/verify_test.go +++ /dev/null @@ -1,61 +0,0 @@ -/* -Copyright the Velero contributors. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package features - -import ( - "testing" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/mock" - - findermocks "github.com/vmware-tanzu/velero/pkg/features/mocks" -) - -func TestVerify(t *testing.T) { - NewFeatureFlagSet() - verifier := verifier{} - - finder := new(findermocks.PluginFinder) - finder.On("Find", mock.Anything, mock.Anything).Return(false) - verifier.finder = finder - ready, err := verifier.Verify("EnableCSI") - assert.Equal(t, false, ready) - assert.Nil(t, err) - - finder = new(findermocks.PluginFinder) - finder.On("Find", mock.Anything, mock.Anything).Return(true) - verifier.finder = finder - ready, err = verifier.Verify("EnableCSI") - assert.Equal(t, false, ready) - assert.EqualError(t, err, "CSI plugins are registered, but the EnableCSI feature is not enabled") - - Enable("EnableCSI") - finder = new(findermocks.PluginFinder) - finder.On("Find", mock.Anything, mock.Anything).Return(false) - verifier.finder = finder - ready, err = verifier.Verify("EnableCSI") - assert.Equal(t, false, ready) - assert.EqualError(t, err, "CSI feature is enabled, but CSI plugins are not registered") - - Enable("EnableCSI") - finder = new(findermocks.PluginFinder) - finder.On("Find", mock.Anything, mock.Anything).Return(true) - verifier.finder = finder - ready, err = verifier.Verify("EnableCSI") - assert.Equal(t, true, ready) - assert.Nil(t, err) -} diff --git a/pkg/plugin/clientmgmt/manager_test.go b/pkg/plugin/clientmgmt/manager_test.go index 3a1c39529a..1e8c149268 100644 --- a/pkg/plugin/clientmgmt/manager_test.go +++ b/pkg/plugin/clientmgmt/manager_test.go @@ -61,10 +61,6 @@ func (r *mockRegistry) Get(kind common.PluginKind, name string) (framework.Plugi return id, args.Error(1) } -func (r *mockRegistry) Find(kind common.PluginKind, name string) bool { - return false -} - func TestNewManager(t *testing.T) { logger := test.NewLogger() logLevel := logrus.InfoLevel diff --git a/pkg/plugin/clientmgmt/process/registry.go b/pkg/plugin/clientmgmt/process/registry.go index 7845f79ef3..6238c45fb0 100644 --- a/pkg/plugin/clientmgmt/process/registry.go +++ b/pkg/plugin/clientmgmt/process/registry.go @@ -37,9 +37,6 @@ type Registry interface { List(kind common.PluginKind) []framework.PluginIdentifier // Get returns the PluginIdentifier for kind and name. Get(kind common.PluginKind, name string) (framework.PluginIdentifier, error) - - // Find checks if the specified plugin exists in the registry - Find(kind common.PluginKind, name string) bool } // KindAndName is a convenience struct that combines a PluginKind and a name. @@ -128,12 +125,6 @@ func (r *registry) Get(kind common.PluginKind, name string) (framework.PluginIde return p, nil } -// Contain if the specified plugin exists in the registry -func (r *registry) Find(kind common.PluginKind, name string) bool { - _, found := r.pluginsByID[KindAndName{Kind: kind, Name: name}] - return found -} - // readPluginsDir recursively reads dir looking for plugins. func (r *registry) readPluginsDir(dir string) ([]string, error) { if _, err := r.fs.Stat(dir); err != nil { diff --git a/pkg/restore/actions/csi/volumesnapshot_action.go b/pkg/restore/actions/csi/volumesnapshot_action.go index 6f8326c226..072fe1da81 100644 --- a/pkg/restore/actions/csi/volumesnapshot_action.go +++ b/pkg/restore/actions/csi/volumesnapshot_action.go @@ -115,9 +115,7 @@ func (p *volumeSnapshotRestoreItemAction) Execute( vs.Namespace, vs.Name, velerov1api.DriverNameAnnotation) } - p.log.Debugf("Set VolumeSnapshotContent %s/%s DeletionPolicy", - "to Retain to make sure VS deletion in namespace will not", - "delete Snapshot on cloud provider.", + p.log.Debugf("Set VolumeSnapshotContent %s/%s DeletionPolicy to Retain to make sure VS deletion in namespace will not delete Snapshot on cloud provider.", newNamespace, vs.Name) vsc := snapshotv1api.VolumeSnapshotContent{ @@ -154,8 +152,8 @@ func (p *volumeSnapshotRestoreItemAction) Execute( "failed to create volumesnapshotcontents %s", vsc.GenerateName) } - p.log.Infof("Created VolumesnapshotContents %s with static", - "binding to volumesnapshot %s/%s", vsc, newNamespace, vs.Name) + p.log.Infof("Created VolumesnapshotContents %s with static binding to volumesnapshot %s/%s", + vsc, newNamespace, vs.Name) // Reset Spec to convert the VolumeSnapshot from using // the dynamic VolumeSnapshotContent to the static one. diff --git a/pkg/restore/actions/csi/volumesnapshotcontent_action.go b/pkg/restore/actions/csi/volumesnapshotcontent_action.go index bd07fb982d..c927d4dd5d 100644 --- a/pkg/restore/actions/csi/volumesnapshotcontent_action.go +++ b/pkg/restore/actions/csi/volumesnapshotcontent_action.go @@ -75,8 +75,8 @@ func (p *volumeSnapshotContentRestoreItemAction) Execute( ) } - p.log.Infof("Returning from VolumeSnapshotContentRestoreItemAction", - "with %d additionalItems", len(additionalItems)) + p.log.Infof("Returning from VolumeSnapshotContentRestoreItemAction with %d additionalItems", + len(additionalItems)) return &velero.RestoreItemActionExecuteOutput{ UpdatedItem: input.Item, AdditionalItems: additionalItems, diff --git a/pkg/restore/restore.go b/pkg/restore/restore.go index 33a8bab173..d2f383079e 100644 --- a/pkg/restore/restore.go +++ b/pkg/restore/restore.go @@ -114,7 +114,6 @@ type kubernetesRestorer struct { podGetter cache.Getter credentialFileStore credentials.FileStore kbClient crclient.Client - featureVerifier features.Verifier } // NewKubernetesRestorer creates a new kubernetesRestorer. @@ -132,7 +131,6 @@ func NewKubernetesRestorer( podGetter cache.Getter, credentialStore credentials.FileStore, kbClient crclient.Client, - featureVerifier features.Verifier, ) (Restorer, error) { return &kubernetesRestorer{ discoveryHelper: discoveryHelper, @@ -157,7 +155,6 @@ func NewKubernetesRestorer( podGetter: podGetter, credentialFileStore: credentialStore, kbClient: kbClient, - featureVerifier: featureVerifier, }, nil } @@ -322,7 +319,6 @@ func (kr *kubernetesRestorer) RestoreWithResolvers( itemOperationsList: req.GetItemOperationsList(), resourceModifiers: req.ResourceModifiers, disableInformerCache: req.DisableInformerCache, - featureVerifier: kr.featureVerifier, hookTracker: hook.NewHookTracker(), backupVolumeInfoMap: req.BackupVolumeInfoMap, restoreVolumeInfoTracker: req.RestoreVolumeInfoTracker, @@ -376,7 +372,6 @@ type restoreContext struct { itemOperationsList *[]*itemoperation.RestoreOperation resourceModifiers *resourcemodifiers.ResourceModifiers disableInformerCache bool - featureVerifier features.Verifier hookTracker *hook.HookTracker backupVolumeInfoMap map[string]volume.BackupVolumeInfo restoreVolumeInfoTracker *volume.RestoreVolumeInfoTracker @@ -1252,11 +1247,6 @@ func (ctx *restoreContext) restoreItem(obj *unstructured.Unstructured, groupReso restoreLogger.Infof("Dynamically re-provisioning persistent volume because it has a CSI VolumeSnapshot or a related snapshot DataUpload.") ctx.pvsToProvision.Insert(name) - if ready, err := ctx.featureVerifier.Verify(velerov1api.CSIFeatureFlag); !ready { - ctx.log.Errorf("Failed to verify CSI modules, ready %v, err %v", ready, err) - errs.Add(namespace, fmt.Errorf("CSI modules are not ready for restore. Check CSI feature is enabled and CSI plugin is installed")) - } - // Return early because we don't want to restore the PV itself, we // want to dynamically re-provision it. return warnings, errs, itemExists @@ -1308,11 +1298,6 @@ func (ctx *restoreContext) restoreItem(obj *unstructured.Unstructured, groupReso restoreLogger.Infof("Dynamically re-provisioning persistent volume because it has a CSI VolumeSnapshot or a related snapshot DataUpload.") ctx.pvsToProvision.Insert(name) - if ready, err := ctx.featureVerifier.Verify(velerov1api.CSIFeatureFlag); !ready { - ctx.log.Errorf("Failed to verify CSI modules, ready %v, err %v", ready, err) - errs.Add(namespace, fmt.Errorf("CSI modules are not ready for restore. Check CSI feature is enabled and CSI plugin is installed")) - } - // Return early because we don't want to restore the PV itself, we // want to dynamically re-provision it. return warnings, errs, itemExists diff --git a/pkg/restore/restore_test.go b/pkg/restore/restore_test.go index 57294013bc..19f66e9bd1 100644 --- a/pkg/restore/restore_test.go +++ b/pkg/restore/restore_test.go @@ -49,7 +49,6 @@ import ( "github.com/vmware-tanzu/velero/pkg/client" "github.com/vmware-tanzu/velero/pkg/discovery" "github.com/vmware-tanzu/velero/pkg/features" - verifiermocks "github.com/vmware-tanzu/velero/pkg/features/mocks" "github.com/vmware-tanzu/velero/pkg/itemoperation" "github.com/vmware-tanzu/velero/pkg/kuberesource" "github.com/vmware-tanzu/velero/pkg/plugin/velero" @@ -220,9 +219,6 @@ func TestRestorePVWithVolumeInfo(t *testing.T) { } features.Enable("EnableCSI") - finder := new(verifiermocks.PluginFinder) - finder.On("Find", mock.Anything, mock.Anything).Return(true) - verifier := features.NewVerifier(finder) for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { @@ -232,7 +228,6 @@ func TestRestorePVWithVolumeInfo(t *testing.T) { h.DiscoveryClient.WithAPIResource(r) } require.NoError(t, h.restorer.discoveryHelper.Refresh()) - h.restorer.featureVerifier = verifier data := &Request{ Log: h.log, @@ -3160,47 +3155,6 @@ func TestRestorePersistentVolumes(t *testing.T) { }, want: []*test.APIResource{}, }, - { - name: "when a PV has a CSI VolumeSnapshot, but CSI modules are not ready, the PV is not restored", - restore: defaultRestore().Result(), - backup: defaultBackup().Result(), - tarball: test.NewTarWriter(t). - AddItems("persistentvolumes", - builder.ForPersistentVolume("pv-1"). - ReclaimPolicy(corev1api.PersistentVolumeReclaimRetain). - ClaimRef("velero", testPVCName). - Result(), - ). - Done(), - apiResources: []*test.APIResource{ - test.PVs(), - test.PVCs(), - }, - csiVolumeSnapshots: []*snapshotv1api.VolumeSnapshot{ - { - ObjectMeta: metav1.ObjectMeta{ - Namespace: "velero", - Name: "test", - }, - Spec: snapshotv1api.VolumeSnapshotSpec{ - Source: snapshotv1api.VolumeSnapshotSource{ - PersistentVolumeClaimName: &testPVCName, - }, - }, - }, - }, - volumeSnapshotLocations: []*velerov1api.VolumeSnapshotLocation{ - builder.ForVolumeSnapshotLocation(velerov1api.DefaultNamespace, "default").Provider("provider-1").Result(), - }, - volumeSnapshotterGetter: map[string]vsv1.VolumeSnapshotter{ - "provider-1": &volumeSnapshotter{ - snapshotVolumes: map[string]string{"snapshot-1": "new-volume"}, - }, - }, - want: []*test.APIResource{}, - csiFeatureVerifierErr: "fake-feature-check-error", - wantError: true, - }, { name: "when a PV with a reclaim policy of retain has a DataUpload result CM and does not exist in-cluster, the PV is not restored", restore: defaultRestore().ObjectMeta(builder.WithUID("fakeUID")).Result(), @@ -3233,40 +3187,6 @@ func TestRestorePersistentVolumes(t *testing.T) { })).Result(), want: []*test.APIResource{}, }, - { - name: "when a PV has a DataUpload result CM, but CSI modules are not ready, the PV is not restored", - restore: defaultRestore().ObjectMeta(builder.WithUID("fakeUID")).Result(), - backup: defaultBackup().Result(), - tarball: test.NewTarWriter(t). - AddItems("persistentvolumes", - builder.ForPersistentVolume("pv-1"). - ReclaimPolicy(corev1api.PersistentVolumeReclaimRetain). - ClaimRef("velero", testPVCName). - Result(), - ). - Done(), - apiResources: []*test.APIResource{ - test.PVs(), - test.PVCs(), - test.ConfigMaps(), - }, - volumeSnapshotLocations: []*velerov1api.VolumeSnapshotLocation{ - builder.ForVolumeSnapshotLocation(velerov1api.DefaultNamespace, "default").Provider("provider-1").Result(), - }, - volumeSnapshotterGetter: map[string]vsv1.VolumeSnapshotter{ - "provider-1": &volumeSnapshotter{ - snapshotVolumes: map[string]string{"snapshot-1": "new-volume"}, - }, - }, - dataUploadResult: builder.ForConfigMap("velero", "test").ObjectMeta(builder.WithLabelsMap(map[string]string{ - velerov1api.RestoreUIDLabel: "fakeUID", - velerov1api.PVCNamespaceNameLabel: "velero.testPVC", - velerov1api.ResourceUsageLabel: string(velerov1api.VeleroResourceUsageDataUploadResult), - })).Result(), - want: []*test.APIResource{}, - csiFeatureVerifierErr: "fake-feature-check-error", - wantError: true, - }, } for _, tc := range tests { @@ -3278,14 +3198,6 @@ func TestRestorePersistentVolumes(t *testing.T) { return renamed, nil } - verifierMock := new(verifiermocks.Verifier) - if tc.csiFeatureVerifierErr != "" { - verifierMock.On("Verify", mock.Anything, mock.Anything).Return(false, errors.New(tc.csiFeatureVerifierErr)) - } else { - verifierMock.On("Verify", mock.Anything, mock.Anything).Return(true, nil) - } - h.restorer.featureVerifier = verifierMock - // set up the VolumeSnapshotLocation client and add test data to it for _, vsl := range tc.volumeSnapshotLocations { require.NoError(t, h.restorer.kbClient.Create(context.Background(), vsl)) diff --git a/site/content/docs/main/csi-snapshot-data-movement.md b/site/content/docs/main/csi-snapshot-data-movement.md index 4765eaf744..78967ccde4 100644 --- a/site/content/docs/main/csi-snapshot-data-movement.md +++ b/site/content/docs/main/csi-snapshot-data-movement.md @@ -152,14 +152,22 @@ Backup repository is created during the first execution of backup targeting to i ## Install Velero with CSI support on source cluster -On source cluster, Velero needs to manipulate CSI snapshots through the CSI volume snapshot APIs, so you must enable the `EnableCSI` feature flag and install the Velero [CSI plugin][2] on the Velero server. +On source cluster, Velero needs to manipulate CSI snapshots through the CSI volume snapshot APIs, so you must enable the `EnableCSI` feature flag on the Velero server. -Both of these can be added with the `velero install` command. +To integrate Velero with the CSI volume snapshot APIs, you must enable the `EnableCSI` feature flag. + +From release-1.14, the `github.com/vmware-tanzu/velero-plugin-for-csi` repository, which is the Velero CSI plugin, is merged into the `github.com/vmware-tanzu/velero` repository. +The reasons to merge the CSI plugin are: +* The VolumeSnapshot data mover depends on the CSI plugin, it's reasonabe to integrate them. +* This change reduces the Velero deploying complexity. +* This makes performance tuning easier in the future. + +As a result, no need to install Velero CSI plugin anymore. ```bash velero install \ --features=EnableCSI \ ---plugins=,velero/velero-plugin-for-csi:v0.6.0 \ +--plugins= \ ... ``` diff --git a/site/content/docs/main/csi.md b/site/content/docs/main/csi.md index 6fd1f9a729..42d0b653cd 100644 --- a/site/content/docs/main/csi.md +++ b/site/content/docs/main/csi.md @@ -5,7 +5,16 @@ layout: docs Integrating Container Storage Interface (CSI) snapshot support into Velero enables Velero to backup and restore CSI-backed volumes using the [Kubernetes CSI Snapshot APIs](https://kubernetes.io/docs/concepts/storage/volume-snapshots/). -By supporting CSI snapshot APIs, Velero can support any volume provider that has a CSI driver, without requiring a Velero-specific plugin to be available. This page gives an overview of how to add support for CSI snapshots to Velero through CSI plugins. For more information about specific components, see the [plugin repo](https://github.com/vmware-tanzu/velero-plugin-for-csi/). +By supporting CSI snapshot APIs, Velero can support any volume provider that has a CSI driver, without requiring a Velero-specific plugin to be available. This page gives an overview of how to add support for CSI snapshots to Velero. + +## Notice +From release-1.14, the `github.com/vmware-tanzu/velero-plugin-for-csi` repository, which is the Velero CSI plugin, is merged into the `github.com/vmware-tanzu/velero` repository. +The reasons to merge the CSI plugin are: +* The VolumeSnapshot data mover depends on the CSI plugin, it's reasonabe to integrate them. +* This change reduces the Velero deploying complexity. +* This makes performance tuning easier in the future. + +As a result, no need to install Velero CSI plugin anymore. ## Prerequisites @@ -17,27 +26,25 @@ By supporting CSI snapshot APIs, Velero can support any volume provider that has ## Installing Velero with CSI support -To integrate Velero with the CSI volume snapshot APIs, you must enable the `EnableCSI` feature flag and install the Velero [CSI plugins][2] on the Velero server. - -Both of these can be added with the `velero install` command. +To integrate Velero with the CSI volume snapshot APIs, you must enable the `EnableCSI` feature flag. ```bash velero install \ --features=EnableCSI \ ---plugins=,velero/velero-plugin-for-csi:v0.7.0 \ +--plugins= \ ... ``` To include the status of CSI objects associated with a Velero backup in `velero backup describe` output, run `velero client config set features=EnableCSI`. -See [Enabling Features][1] for more information about managing client-side feature flags. You can also view the image on [Docker Hub][3]. +See [Enabling Features][1] for more information about managing client-side feature flags. ## Implementation Choices -This section documents some of the choices made during implementation of the Velero [CSI plugins][2]: +This section documents some of the choices made during implementing the CSI snapshot. 1. VolumeSnapshots created by the Velero CSI plugins are retained only for the lifetime of the backup even if the `DeletionPolicy` on the VolumeSnapshotClass is set to `Retain`. To accomplish this, during deletion of the backup the prior to deleting the VolumeSnapshot, VolumeSnapshotContent object is patched to set its `DeletionPolicy` to `Delete`. Deleting the VolumeSnapshot object will result in cascade delete of the VolumeSnapshotContent and the snapshot in the storage provider. - 1. VolumeSnapshotContent objects created during a `velero backup` that are dangling, unbound to a VolumeSnapshot object, will be discovered, using labels, and deleted on backup deletion. - 1. The Velero CSI plugins, to backup CSI backed PVCs, will choose the VolumeSnapshotClass in the cluster based on the following logic: + 2. VolumeSnapshotContent objects created during a `velero backup` that are dangling, unbound to a VolumeSnapshot object, will be discovered, using labels, and deleted on backup deletion. + 3. The Velero CSI plugins, to backup CSI backed PVCs, will choose the VolumeSnapshotClass in the cluster based on the following logic: 1. **Default Behavior:** You can simply create a VolumeSnapshotClass for a particular driver and put a label on it to indicate that it is the default VolumeSnapshotClass for that driver. For example, if you want to create a VolumeSnapshotClass for the CSI driver `disk.csi.cloud.com` for taking snapshots of disks created with `disk.csi.cloud.com` based storage classes, you can create a VolumeSnapshotClass like this: ```yaml @@ -83,7 +90,7 @@ This section documents some of the choices made during implementation of the Vel storage: 1Gi storageClassName: disk.csi.cloud.com ``` - 1. The VolumeSnapshot objects will be removed from the cluster after the backup is uploaded to the object storage, so that the namespace that is backed up can be deleted without removing the snapshot in the storage provider if the `DeletionPolicy` is `Delete`. + 4. The VolumeSnapshot objects will be removed from the cluster after the backup is uploaded to the object storage, so that the namespace that is backed up can be deleted without removing the snapshot in the storage provider if the `DeletionPolicy` is `Delete`. ## How it Works - Overview @@ -110,10 +117,7 @@ The `DeletionPolicy` on the VolumeSnapshotContent will be the same as the `Delet When the Velero backup expires, the VolumeSnapshot objects will be deleted and the VolumeSnapshotContent objects will be updated to have a `DeletionPolicy` of `Delete`, to free space on the storage system. -For more details on how each plugin works, see the [CSI plugin repo][2]'s documentation. - **Note:** The AWS, Microsoft Azure, and Google Cloud Platform (GCP) Velero plugins version 1.4 and later are able to snapshot and restore persistent volumes provisioned by a CSI driver via the APIs of the cloud provider, without having to install Velero CSI plugins. See the [AWS](https://github.com/vmware-tanzu/velero-plugin-for-aws), [Microsoft Azure](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure), and [Google Cloud Platform (GCP)](https://github.com/vmware-tanzu/velero-plugin-for-gcp) Velero plugin repo for more information on supported CSI drivers. +From v1.14, no need to install the CSI plugin, because it is integrated into the Velero code base. [1]: customize-installation.md#enable-server-side-features -[2]: https://github.com/vmware-tanzu/velero-plugin-for-csi/ -[3]: https://hub.docker.com/repository/docker/velero/velero-plugin-for-csi diff --git a/site/content/docs/main/supported-providers.md b/site/content/docs/main/supported-providers.md index 0fc4a48aa0..6fc53f8e6b 100644 --- a/site/content/docs/main/supported-providers.md +++ b/site/content/docs/main/supported-providers.md @@ -15,7 +15,6 @@ Velero supports a variety of storage providers for different backup and snapshot | [Google Cloud Platform (GCP)](https://cloud.google.com) | [Google Cloud Storage](https://github.com/vmware-tanzu/velero-plugin-for-gcp/blob/main/backupstoragelocation.md) | [Google Compute Engine Disks](https://github.com/vmware-tanzu/velero-plugin-for-gcp/blob/main/volumesnapshotlocation.md) | [Velero plugin for GCP](https://github.com/vmware-tanzu/velero-plugin-for-gcp) | [GCP Plugin Setup](https://github.com/vmware-tanzu/velero-plugin-for-gcp#setup) | [BackupStorageLocation](https://github.com/vmware-tanzu/velero-plugin-for-gcp/blob/main/backupstoragelocation.md)
[VolumeSnapshotLocation](https://github.com/vmware-tanzu/velero-plugin-for-gcp/blob/main/volumesnapshotlocation.md) | | [Microsoft Azure](https://azure.com) | Azure Blob Storage | Azure Managed Disks | [Velero plugin for Microsoft Azure](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure) | [Azure Plugin Setup](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure#setup) | [BackupStorageLocation](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/main/backupstoragelocation.md)
[VolumeSnapshotLocation](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/main/volumesnapshotlocation.md) | | [VMware vSphere](https://www.vmware.com/ca/products/vsphere.html) | 🚫 | vSphere Volumes | [VMware vSphere](https://github.com/vmware-tanzu/velero-plugin-for-vsphere) | [vSphere Plugin Setup](https://github.com/vmware-tanzu/velero-plugin-for-vsphere#velero-plugin-for-vsphere-installation-and-configuration-details) | 🚫 | -| [Container Storage Interface (CSI)](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/)| 🚫 | CSI Volumes | [Velero plugin for CSI](https://github.com/vmware-tanzu/velero-plugin-for-csi/) | [CSI Plugin Setup](https://github.com/vmware-tanzu/velero-plugin-for-csi#kinds-of-plugins-included) | 🚫 | {{< /table >}} Contact: [#Velero Slack](https://kubernetes.slack.com/messages/velero), [GitHub Issues](https://github.com/vmware-tanzu/velero/issues) diff --git a/site/content/docs/main/tilt.md b/site/content/docs/main/tilt.md index 19ec6003bc..6d802220c9 100644 --- a/site/content/docs/main/tilt.md +++ b/site/content/docs/main/tilt.md @@ -55,7 +55,6 @@ Here is an example: "aws": "../velero-plugin-for-aws", "gcp": "../velero-plugin-for-gcp", "azure": "../velero-plugin-for-microsoft-azure", - "csi": "../velero-plugin-for-csi" }, "allowed_contexts": [ "development" diff --git a/site/content/docs/main/upgrade-to-1.13.md b/site/content/docs/main/upgrade-to-1.14.md similarity index 74% rename from site/content/docs/main/upgrade-to-1.13.md rename to site/content/docs/main/upgrade-to-1.14.md index cae4e56824..8243dd44ca 100644 --- a/site/content/docs/main/upgrade-to-1.13.md +++ b/site/content/docs/main/upgrade-to-1.14.md @@ -1,11 +1,11 @@ --- -title: "Upgrading to Velero 1.13" +title: "Upgrading to Velero 1.14" layout: docs --- ## Prerequisites -- Velero [v1.12.x][5] installed. +- Velero [v1.13.x][5] installed. If you're not yet running at least Velero v1.8, see the following: @@ -14,6 +14,7 @@ If you're not yet running at least Velero v1.8, see the following: - [Upgrading to v1.10][3] - [Upgrading to v1.11][4] - [Upgrading to v1.12][5] +- [Upgrading to v1.13][6] Before upgrading, check the [Velero compatibility matrix](https://github.com/vmware-tanzu/velero#velero-compatibility-matrix) to make sure your version of Kubernetes is supported by the new version of Velero. @@ -34,7 +35,7 @@ Before upgrading, check the [Velero compatibility matrix](https://github.com/vmw ```bash Client: - Version: v1.13.0 + Version: v1.14.0 Git commit: ``` @@ -46,21 +47,29 @@ Before upgrading, check the [Velero compatibility matrix](https://github.com/vmw **NOTE:** Since velero v1.10.0 only v1 CRD will be supported during installation, therefore, the v1.10.0 will only work on Kubernetes version >= v1.16 -3. Update the container image used by the Velero deployment, plugin and (optionally) the node agent daemon set: +3. Delete the CSI plugin. Because the Velero CSI plugin is already merged into the Velero, need to remove the existing CSI plugin InitContainer. Otherwise, the Velero server plugin would fail to start due to same plugin registered twice. +Please find more information of CSI plugin merging in this page [csi]. +If the plugin move CLI fails due to `not found`, that is caused by the Velero CSI plugin not installed before upgrade. It's safe to ignore the error. + + ``` bash + velero plugin remove velero-velero-plugin-for-csi; echo 0 + ``` + +4. Update the container image used by the Velero deployment, plugin and (optionally) the node agent daemon set: ```bash # set the container and image of the init container for plugin accordingly, # if you are using other plugin kubectl set image deployment/velero \ - velero=velero/velero:v1.13.0 \ - velero-plugin-for-aws=velero/velero-plugin-for-aws:v1.9.0 \ + velero=velero/velero:v1.14.0 \ + velero-plugin-for-aws=velero/velero-plugin-for-aws:v1.10.0 \ --namespace velero # optional, if using the node agent daemonset kubectl set image daemonset/node-agent \ - node-agent=velero/velero:v1.13.0 \ + node-agent=velero/velero:v1.14.0 \ --namespace velero ``` -4. Confirm that the deployment is up and running with the correct version by running: +5. Confirm that the deployment is up and running with the correct version by running: ```bash velero version @@ -70,23 +79,23 @@ Before upgrading, check the [Velero compatibility matrix](https://github.com/vmw ```bash Client: - Version: v1.13.0 + Version: v1.14.0 Git commit: Server: - Version: v1.13.0 + Version: v1.14.0 ``` ### Upgrade from version lower than v1.10.0 -The procedure for upgrading from a version lower than v1.10.0 is identical to the procedure above, except for step 3 as shown below. +The procedure for upgrading from a version lower than v1.10.0 is identical to the procedure above, except for step 4 as shown below. -3. Update the container image and objects fields used by the Velero deployment and, optionally, the restic daemon set: +1. Update the container image and objects fields used by the Velero deployment and, optionally, the restic daemon set: ```bash # uploader_type value could be restic or kopia kubectl get deploy -n velero -ojson \ - | sed "s#\"image\"\: \"velero\/velero\:v[0-9]*.[0-9]*.[0-9]\"#\"image\"\: \"velero\/velero\:v1.13.0\"#g" \ + | sed "s#\"image\"\: \"velero\/velero\:v[0-9]*.[0-9]*.[0-9]\"#\"image\"\: \"velero\/velero\:v1.14.0\"#g" \ | sed "s#\"server\",#\"server\",\"--uploader-type=$uploader_type\",#g" \ | sed "s#default-volumes-to-restic#default-volumes-to-fs-backup#g" \ | sed "s#default-restic-prune-frequency#default-repo-maintain-frequency#g" \ @@ -95,7 +104,7 @@ The procedure for upgrading from a version lower than v1.10.0 is identical to th # optional, if using the restic daemon set echo $(kubectl get ds -n velero restic -ojson) \ - | sed "s#\"image\"\: \"velero\/velero\:v[0-9]*.[0-9]*.[0-9]\"#\"image\"\: \"velero\/velero\:v1.13.0\"#g" \ + | sed "s#\"image\"\: \"velero\/velero\:v[0-9]*.[0-9]*.[0-9]\"#\"image\"\: \"velero\/velero\:v1.14.0\"#g" \ | sed "s#\"name\"\: \"restic\"#\"name\"\: \"node-agent\"#g" \ | sed "s#\[ \"restic\",#\[ \"node-agent\",#g" \ | kubectl apply -f - @@ -115,3 +124,4 @@ If upgrading from Velero v1.9.x or lower, there will likely remain some unused r [3]: https://velero.io/docs/v1.10/upgrade-to-1.10 [4]: https://velero.io/docs/v1.11/upgrade-to-1.11 [5]: https://velero.io/docs/v1.12/upgrade-to-1.12 +[6]: https://velero.io/docs/v1.13/upgrade-to-1.13 diff --git a/test/e2e/README.md b/test/e2e/README.md index 10a43deeb2..b1d8259997 100644 --- a/test/e2e/README.md +++ b/test/e2e/README.md @@ -272,7 +272,7 @@ As we provided several examples for E2E test execution, what if no filter is inv ### Suggested pipelines for full test Following pipelines should cover all E2E tests along with proper filters: -1. **CSI pipeline:** As we can see lots of labels in E2E test code, there're many snapshot-labeled test scripts. To cover CSI scenario, a pipeline with CSI enabled should be a good choice, otherwise, we will double all the snapshot cases for CSI scenario, it's very time-wasting. By providing `FEATURES=EnableCSI` and `PLUGINS=,velero/velero-plugin-for-csi:`, a CSI pipeline is ready for testing. +1. **CSI pipeline:** As we can see lots of labels in E2E test code, there're many snapshot-labeled test scripts. To cover CSI scenario, a pipeline with CSI enabled should be a good choice, otherwise, we will double all the snapshot cases for CSI scenario, it's very time-wasting. By providing `FEATURES=EnableCSI` and `PLUGINS=`, a CSI pipeline is ready for testing. 1. **Data mover pipeline:** Data mover scenario is the same scenario with migaration test except the restriction of migaration between different providers, so it better to separated it out from other pipelines. Please refer the example in previous. 1. **Restic/Kopia backup path pipelines:** 1. **Restic pipeline:** For the same reason of saving time, set `UPLOADER_TYPE` to `restic` for all file system backup test cases; diff --git a/tilt-resources/examples/deployment.yaml b/tilt-resources/examples/deployment.yaml index bda59ae28b..3796c4ff2c 100644 --- a/tilt-resources/examples/deployment.yaml +++ b/tilt-resources/examples/deployment.yaml @@ -84,12 +84,6 @@ spec: volumeMounts: - mountPath: /target name: plugins - - image: velero/velero-plugin-for-csi - imagePullPolicy: Always - name: velero-plugin-for-csi - volumeMounts: - - mountPath: /target - name: plugins restartPolicy: Always serviceAccountName: velero volumes: diff --git a/tilt-resources/examples/tilt-settings.json b/tilt-resources/examples/tilt-settings.json index 4e67f00565..1974a87e44 100644 --- a/tilt-resources/examples/tilt-settings.json +++ b/tilt-resources/examples/tilt-settings.json @@ -9,8 +9,7 @@ "providers": { "aws": "../velero-plugin-for-aws", "gcp": "../velero-plugin-for-gcp", - "azure": "../velero-plugin-for-microsoft-azure", - "csi": "../velero-plugin-for-csi" + "azure": "../velero-plugin-for-microsoft-azure" }, "allowed_contexts": [ "development"