Velero backup terminating after 4h, no matter what

[jcmickey33]

UPDATE: sseago's answer is correct. For anyone doing a helm installation, as I was, it's under the configuration section, field name is fsBackupTimeout

# values.yaml
configuration:
  fsBackupTimeout: 8h # or whatever you want to make it

---

Let me start off with the following disclaimer: I'm working in environments that are either airgapped and/or have certain restrictions on what information can be disseminated. So if additional information about my situation is desired, I'll do my best to get it but I may not be able to.

BLUF: velero backups are terminating before all information in the clusters have been backed up. I need to determine the best way allow them to at least run LONGER, if not run until they complete. Using the --csi-snapshot-timeout and --item-operation-timeout flags has not worked

Background: my team and I are running multi-cluster Kubernetes environments and using Velero for our cluster backups. In one of our environments, one of the clusters is significantly larger than the other. (~120 nodes vs ~40 nodes). Three times a week, we'll restore one namespace from the previous night's backup into a dummy namespace as a sanity check. We noticed last week that one of the alphabetically-later namespaces (starts with an "S") didn't backup correctly and began investigating. From looking at the description and logs for the backup (velero backup describe <backup_name> --details and velero backup logs <backup_name>) we noticed pod volumes didn't seem to all be being backed up among other issues.

We noticed that backups from both of the clusters, despite their difference in size, were all terminating after 4 hours (give or take a few minutes).

Smaller cluster, screenshot from yesterday's nightly backup:

Larger cluster, screenshot from yesterday's nightly backup:

This made sense, given that the ItemOperationTimeout was set to the default of 4 hours. So in an attempt to fix that, we tried running backups varying the values entered for --csi-snapshot-timeout and ---item-operation-timeout flags. Most of the variance has been with the --item-operation-timeout flag, but neither has seemed to make any difference in run time. Below is a screenshot from a backup I attempted to run the other day with both flags set to 8h. It still finished after about 4 hours and a restore of the same namespace mentioned earlier with essentially the same deficiencies/failures.

Larger cluster, 8 hour timeouts:

Google searches have been mostly fruitless and no one's been able to determine any other flag adjustments that might help. Feel free to ask any clarification questions you may have. As mentioned, I'll give you as much information as I can. Any help you've got would be much appreciated!

[sseago]

Are you using csi snapshots or filesystem backup for your volume data? If you're using filesystem backup, that has a different parameter -- --fs-backup-timeout also defaults to 4 hours. It sounds like you may still have some PodVolumeBackups in progress at 4 hours, so those will time out unless you increase that. If you're not using filesystem backup, then nothing should be timing out at 4 hours with the other timeouts set to 8h.

[jcmickey33]

We've tried using both snapshots and filesystem (though FS backup with the node-agent DS is what we've been leaning towards). Didn't see that flag in the velero help message, so hadn't tried it up til now. It didn't work, though.

I'll go start digging on that, but is that flag something to be used in the velero backup create command or is that something to be added to the daemonset manifest/helm chart installation?

[kaovilai]

This flag is part of velero deployment server command, and can be added as part of velero install as well as helm chart.

[sseago]

fs-backup-timeout is a server arg. It's configured for the velero pod, not per individual backup.

[jcmickey33]

You're my hero! Sure enough, I looked through the original values file for the helm chart and found the fsBackupTimeout field, ran the helm upgrade and I've got one running. I'll come back and officially mark the answer tomorrow (trust but verify, and all), but THANK YOU!

[kaovilai]

Glad to hear it's resolved!