Support for protocols other than SSH

[Compizfox]

I'm currently using Duplicity for my off-site backups which works quite well apart from one thing: it lacks incremental forever backups.

Borg does have this feature.

However, my backup backend only supports WebDAV. In Duplicity I can directly use a WebDAV backend but Borg only seems to support SSH. I could mount the WebDAV server locally, but davfs2 isn't available on FreeBSD (the platform of my fileserver). There is an analog called wdfs, but it hasn't been updated since 2007. Besides, I've always been told that mounting (off-site) backups locally isn't a good idea because that makes it more likely that your backups can be overwritten in some way.

So, if I want to use Borg instead of duplicity, I'd have two options, none of which are ideal:

• Mounting the WebDAV server locally using wdfs
• Running Borg on a separate VM running Linux, mounting the WebDAV server locally using davfs2, using NFS to access the FreeBSD fileserver

So I was curious whether support for other protocols, like WebDAV, is a planned feature.

[ThomasWaldmann]

Well, we have quite some plans with borg, but a own webdav backend is not a priority for the core devs for sure (I heard that webdav is a rather shitty protocol, btw.).

That said, you could use any kind of tools that either mounts webdav as a FS (as you tried - assuming that the fs emulation is good enough) or first do a "local" backup and then additionally push that to your webdav server using separate tools.

There might be some other backends some day, but likely that will first require bigger internal changes (see also the already existing ticket about S3 support).

[verygreen]

The thing with webdav vs ssh is that with SSH you can run borg copy on the server, and then the two talk to each other using ssh as the data pipe. This is not possible with webdav, no matter how you cut it webdav = full file access without involving borg at all which is universally bad for backups.

I imagine whoever holds your backups is not super keen on letting you run random commands on their hosts and that's why you get webdav (or others might get ftp and the like) - i.e. full filesystem access protocol.

[RonnyPfannschmidt]

webdav is not something thats implementable with a reasonable cost due to sloppy spec and many many sloppy implementations

[Compizfox]

Alright, thanks all for your answers.

I have Borg running in combination with WDFS (an ages old FUSE FS for WebDAV) now which seems to work fine. Only the performance is much lower than what I had with Duplicity; I now get 20 Mb/s max on a 100 Mb/s connection, Duplicity used to max out that speed. I don't know if WDFS or Borg is the bottleneck of course, although I suspect the former.

On another machine (Debian) I'm trying Borg with davfs2 now, but here I'm running into some problems. When I try to initialise a Borg repository on a davfs2 mount, Borg crashes (see attachment for log).

borglog.txt

The thing with webdav vs ssh is that with SSH you can run borg copy on the server, and then the two talk to each other using ssh as the data pipe. This is not possible with webdav, no matter how you cut it webdav = full file access without involving borg at all which is universally bad for backups.

I don't see that as "universally bad" at all. Backup software like Borg and Duplicity don't require any software on the backend which is actually a big pro, because that means that I can use it with any backend I want (theoretically). Cloud providers don't let you run code on their backend.

So IMO it is actually a very good thing that Borg doesn't require software on the backend but still can do incremental forever backups. rdiff-backup for example can also do incremental forever but it requires software on the backend (and can't do encryption, but that's an unrelated issue) so I simpy can't use it.

webdav is not something thats implementable with a reasonable cost due to sloppy spec and many many sloppy implementations

So how does Duplicity do this? Duplicity supports a whole shitload of storage backends (which is great), including WebDAV. This means you don't have to use some FUSE filesystem, which is a good thing.

[verygreen]

I don't see that as "universally bad" at all. Backup software like Borg and Duplicity don't require any software on the backend which is actually a big pro, because that means that I can use it with any backend I want (theoretically). Cloud providers don't let you run code on their backend.

Well, I think there's a misunderstanding here on your part. It's like saying that "borg can write directly to a local fs, so it does not require any software on the backend to be effective (And to support ext4/zfs/whatever).

So IMO it is actually a very good thing that Borg doesn't require software on the backend but still can do incremental forever backups. rdiff-backup for example can also do incremental forever but it requires software on the backend (and can't do encryption, but that's an unrelated issue) so I simpy can't use it.

Borg absolutely does require software on the backend (the other borg instance), otherwise it's just reduced to a local filesystem access be it provided by a local kernel driver (native fs, fuse) or a layer in the app itself (ld_preloaded library, all accesses going through a translation layer like what I suspect duplicity does).
The difference here is: When you have borg (or rsync-backup or bacula or some such) backend running on the node, you can talk only via the backup protocol to it. It does not allow you to touch random files and perform random actions, you are confined to the protocol supported by this particular backup software and that's it. No file deletion or modifying random backup data. borg in append-only mode does not allow you to remove anything in the remote repo like that even if you ask to prune stuff.

When it is said that borg supports ssh out of the box, what is meant is "via ssh we can start another borg instance on the other end and then communicate with it via ssh pipe, in fact we recommend to use ssh force command to ensure only borg could be started for a particular key so nobody has unrestricted access to the repo, only via borg protocol".

[RonnyPfannschmidt]

a first step towards dumb storage might be a storage format that requires less logic
i think its order of magnitude easier to just do atomic uploads of richer segments, than it is to run the current locked segments in place

[enkore]

Proper cloud storage support will require a completely different storage design, since the environment is completely different. Repository+LoggedIO produce strong guarantees on consistency and integrity (somewhat dependent on file system and hardware), which isn't really possible with anything that doesn't give similar guarantees to a POSIX-y file system. Which is why there have been various failures observed with things like DAV or FTP FUSE FSes.

I think this has been discussed in the S3 thread already, but I recap anyway. A possible design could revolve around hash-packs (i.e. blobs of chunks identified by a hash-over-ordered-IDs) to avoid inevitable failures when trying to do segment-counting in the cloud. Those need indexing and versioned objects (=the manifest) need another storage strategy. Compaction needs different strategies. Concurrent accesses needs different strategies. Locking is per se not possible in the storage. And things like append-only are not possible etc.

It would just be an entirely different Repository implementation that will be much more complex. There is really nothing stopping that from happening (just saying).

[RonnyPfannschmidt]

there is 2 ways to go about that

a) direct chunk upload - grat for s3, moot for dav/sshfs
b) rich segments that dont need locking in the store (tricky to get right, needs exclusive create and uploads and a segment dependency graph and graph merges)

[enkore]

a) direct chunk upload - grat for s3, moot for dav/sshfs

S3 PUTs cost money as well. Iirc about 0.5 cents per 1000. Storing just a couple million [small] chunks will cost much more in PUTs than actual storage costs.

E: 100 GB, 10 million chunks = 3 USD/mo for storage, but the upload costed 50 USD. A hard drive is [much] cheaper in that scenario.

[Compizfox]

Borg absolutely does require software on the backend (the other borg instance), otherwise it's just reduced to a local filesystem access be it provided by a local kernel driver (native fs, fuse) or a layer in the app itself (ld_preloaded library, all accesses going through a translation layer like what I suspect duplicity does).

Maybe I'm misunderstanding you but Borg doesn't require Borg to be installed on the backend, right? Hence why I can use FUSE for WebDAV support. It does support it though. I quote from the README:

If Borg is installed on the remote host, big performance gains can be achieved compared to using a network filesystem (sshfs, nfs, ...).

 

The difference here is: When you have borg (or rsync-backup or bacula or some such) backend running on the node, you can talk only via the backup protocol to it. It does not allow you to touch random files and perform random actions, you are confined to the protocol supported by this particular backup software and that's it. No file deletion or modifying random backup data. borg in append-only mode does not allow you to remove anything in the remote repo like that even if you ask to prune stuff.

I understand why this is better from a security point of view but for most off-site backups this is simply not possible.

[RonnyPfannschmidt]

borg needs to be on the server in case of ssh, since a rpc protocol is used, not a file locking protocol

[verygreen]

Maybe I'm misunderstanding you but Borg doesn't require Borg to be installed on the backend, right?

It's required in case you are using ssh://.... repos. It's not required if you use sshfs through fuse.

[Compizfox]

Thanks, I understand now. I missed the part about sshfs. I thought it was also possible to use ssh:// repos without having Borg installed on the backend, but that is not the case.

[DavidCWGA]

rclone supports most of the traditional cloud services including Amazon Cloud Drive. They are also open source. Perhaps some "borrowing" of code could work here?

https://github.com/ncw/rclone