restore with longhorn volumes is marked as completed before volume is actually restored

[KonstantinNosov]

I use velero to backup and restore volumes managed with longhorn.
the restore transits to completed less than in a minute:

status:
  completionTimestamp: '2024-12-24T16:49:14Z'
  hookStatus: {}
  phase: Completed
  progress:
    itemsRestored: 15
    totalItems: 15
  startTimestamp: '2024-12-24T16:48:20Z'

backup schedule spec:

spec:
  schedule: 30 0 * * *
  skipImmediately: false
  template:
    csiSnapshotTimeout: 1h0m0s
    hooks: {}
    includedNamespaces:
      - '*'
    includedResources:
      - pv
      - pvc
    metadata: {}
    ttl: 730h0m0s
  useOwnerReferencesInBackup: false

At this moment longhorn reports volumes are not ready since actual restore is yet in progress.

Volumes are restored successfully after some time, it is only about status of Restore - it is used by argocd health checks, so workloads start to be created before actual volumes restore is finished and fail with timeouts, report degraded state etc since volumes are quite big and take significant time to restore.

Not sure if it is an issue of velero, longhorn or snapshot controller, so starting discussion rather than creating an issue

[blackpiglet]

I assume you used the v1.15.0 Velero and the CSI feature is enabled.
IMO, this is how the Velero CSI plugin restore works. There is no mechanism to guarantee the data restore completion before moving to Complete phase as the backup does.

I suggest using the CSI snapshot data mover function. https://velero.io/docs/v1.15/csi-snapshot-data-movement/

[KonstantinNosov]

Thanks for answering.

TBH I'm a bit lost with how it will work with Longhorn that has it's own logic for data restore. Would not velero take data already stored by longhorn in configured backup target and copy to another location thus doubling the space used by backups?

So far I've been thinking of having hook in restore that waits for volume to be restored on longhorn side.

[blackpiglet]

Your understanding of how the CSI plugin restoration workflow is correct.
The missing part is the mechanism to wait the data restoration completion.

In contrast, The CSI backup has the logic to check whether the VolumeSnapshot and VolumeSnapshotContent status turn into ReadyToUse before setting the async operation completion.

[KonstantinNosov]

Am I getting you right - If I need restore to transit to Completed only after PV is restored and ready to be used I have two options:

1. Change VolumeSnapshotClass to have parameters.type=snap instead of bak to stop longhorn from uploading snapshot data to backup target, utilize velero data movement feature and let velero manage actual data upload and restore
2. Keep everything as is (velero creates snapshot while longhorn restores it), and implement some kind of custom check for volume status - hook in restore, job etc taht would check metadata of the volume or call longhorn API

And there is no way to instruct velero to wait for volume if restore is managed by longhorn

[blackpiglet]

Yes, you are right.

[KonstantinNosov]

I've finally got it working

• Script waiting for specific PVC to have PV restored on longhorn side
• ArgocCD app managing the whole thing with
  • Restore sync-wave=-10
  • Job executing script for each PVC in restore: sync-wave=-5
  • Workloads: sycn-wav= 0

in case someone needs it here is the script:

#!/bin/sh

set -e

VELERO_RESTORE_NAME="$1"
NAMESPACE="longhorn-system"

resource_exists() {
  kubectl get "$1" "$2" -n "$3" >/dev/null 2>&1
}

echo "Fetching PVCs restored by Velero restore: $VELERO_RESTORE_NAME"

RESTORED_PVCS=$(kubectl get pvc --all-namespaces -l velero.io/restore-name="$VELERO_RESTORE_NAME" -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name" --no-headers)

if [ -z "$RESTORED_PVCS" ]; then
  echo "No PVCs found in restore $VELERO_RESTORE_NAME."
  exit 0
fi

echo "Found restored PVCs:"
echo "$RESTORED_PVCS"




echo "$RESTORED_PVCS" | while read -r PVC_NAMESPACE PVC_NAME; do
  echo "Waiting for PVC $PVC_NAME in namespace $PVC_NAMESPACE to be created and bound to a PV and PV to be restored by longhorn..."
  while true; do
    STATUS=$(kubectl get pvc "$PVC_NAME" -n "$PVC_NAMESPACE" -o jsonpath='{.status.phase}' 2>/dev/null || echo "NotFound")
    if [ "$STATUS" = "Bound" ]; then
      echo "  PVC $PVC_NAME in namespace $PVC_NAMESPACE is bound."
      break
    fi
    echo "  PVC $PVC_NAME not yet bound. Retrying in 10 seconds..."
    sleep 10
  done

  PV_NAME=$(kubectl get pvc "$PVC_NAME" -n "$PVC_NAMESPACE" -o jsonpath='{.spec.volumeName}' 2>/dev/null)
  if [ -z "$PV_NAME" ]; then
    echo "  PVC $PVC_NAME in namespace $PVC_NAMESPACE does not have an associated PV. Skipping..."
    continue
  fi

  echo "  Checking PV $PV_NAME for Longhorn provisioner..."
  PROVISIONED_BY=$(kubectl get pv "$PV_NAME" -o jsonpath='{.metadata.annotations.pv\.kubernetes\.io/provisioned-by}' 2>/dev/null)
  if [ "$PROVISIONED_BY" = "driver.longhorn.io" ]; then
    echo "    PV $PV_NAME is provisioned by Longhorn. Waiting for Longhorn volume readiness..."

    while ! resource_exists volumes.longhorn.io "$PV_NAME" "$NAMESPACE"; do
      echo "      Waiting for Longhorn volume $PV_NAME (PVC: $PVC_NAME) to be created..."
      sleep 10
    done

    echo "    Longhorn volume $PV_NAME (PVC: $PVC_NAME) exists. Checking restore status..."

    while true; do
      RESTORE_INITIATED=$(kubectl get volumes.longhorn.io "$PV_NAME" -n "$NAMESPACE" -o jsonpath='{.status.restoreInitiated}' 2>/dev/null)
      RESTORE_REQUIRED=$(kubectl get volumes.longhorn.io "$PV_NAME" -n "$NAMESPACE" -o jsonpath='{.status.restoreRequired}' 2>/dev/null)

      if [ "$RESTORE_INITIATED" = "true" ] && [ "$RESTORE_REQUIRED" = "false" ]; then
        echo "      Longhorn volume $PV_NAME (PVC: $PVC_NAME) is ready: restoreInitiated=$RESTORE_INITIATED, restoreRequired=$RESTORE_REQUIRED"
        break
      fi
      echo "      Waiting for Longhorn volume $PV_NAME (PVC: $PVC_NAME) to complete restoration... (restoreInitiated=$RESTORE_INITIATED, restoreRequired=$RESTORE_REQUIRED)"
      sleep 10
    done
  else
    echo "    PV $PV_NAME (PVC: $PVC_NAME) is not provisioned by Longhorn. Skipping..."
  fi
done

echo "All PVCs and Longhorn volumes are ready."