Add sync-dir function for Cloud Files

[filler]

Hello --

First off, rack rules. Thanks for that. 😸

It would be helpful if there was a sync-dir function for Cloud Files. Much in the spirit of upload-dir, the idea is rack would only upload files to a named container which dont exist in the container but exist in the source of the sync-dir function. Also objects in the container which are not part of the source directory would be pruned.

There are many times when operators land files in a backup target directory on a source host, perform some life-cycle management (eg prune files older than n days) then want to shunt the remaining source files up to Cloud Files for safe-keeping (examples include db backups, log escrowing, etc).

Currently rack users have to empty a container and then call upload-dir in order to get a lifecycle-managed directory on the source into to the container. Thats rather inefficient for source bandwidth and the Files API.

It would be nice to optionally pass in an --prune option at sync-dir runtime to let the operator choose whether to remove files in the container which are not part of the source directory. Sometimes operators want to keep all the things in Files, other times operators do not and simply want a 1:1 relationship between files in the source directory and the container.

Duplicity is a tool which lots of operators use in order to incrementally sync directories into Cloud Files. Its an operational pain whereas rack isnt. 😄

rclone has sync behavior like this by default. Given a fileset both on local disk and in a container, removing some of the files in the set on local disk and calling rclone sync removes them from the container they were previously synced in. It looks like it doesnt offer the operator the option to just shunt new local files on disk to the container and keeping the files in the container which were not on disk.

[jrperritt]

Thank you for filing this; it's something that was on my to-do list and got pushed back. I think I'll be able to work on it next week.

[filler]

That would be so 🔥. Thanks @jrperritt!

[mbejda]

Any update on this ?

[jrperritt]

Currently there are no plans to implement this.