macOS WebDAV

[maxexcloo]

Hi,

First of all, awesome project! This is neat and I like how much more efficient it seems than before!

I'm trying to mount the created webdav share on macOS and not having a lot of luck - I'm not sure what exactly the problem is as it seems to fail silently with just this error:

zurg logs don't seem to be showing much either, do you have any tips?

Appreciate the help!

[yowmamasita]

rclone mount zurg: /Users/xyz/zurgmountpoint --dir-cache-time 10s --allow-other -v

I'm also on a mac and I use this, does this work for you?

[maxexcloo]

I'm trying to bypass using macfuse or an rclone intermediary, my theory goes that this would improve stability (as I've had quite a few system hangs).

zurg (inside docker) > rclone (native with macfuse) > mac dir: this seems to work but has some stability/hangup problems
zurg (inside docker) > rwebdav mac dir: this doesn't connect, the issue above
zurg (inside docker) > rclone serve webdav (inside docker) > native mac webdav mount: this seems to work but wondering about the overhead of running so many layers
zurg (inside docker) > rclone (native with nfsmount) > mac dir: this hung the mac up everytime I tried, wouldn't recommend this

Are you running zurg natively or inside Docker? Which Docker solution are you running (if you are?) How many files/folders and how's performance?

[maxexcloo]

Ideally I'd like to mount like this:

zurg (inside docker or even without docker) > native mac webdav mount

[yowmamasita]

I realized this is not fully supported yet, but I will!

[maxexcloo]

It seems to be okay using rclone as a proxy between them - but I imagine it would be cleaner without?

[yowmamasita]

@maxexcloo yes exactly. Always good to have a simpler setup.

[neaumusic]

I use rclone from brew install rclone and rclone nfsmount instead. Definitely had issues with macfuse, and haven't tried installing directly from rclone binary downloads. I feel like the rclone docker-volume-plugin is what's missing to expose the rclone volume to the host machine, but the instructions seem awful / only for linux

[yowmamasita]

I will have to move the milestone to v0.9.4 sorry just too complex for me to do right now given that I'm prioritizing renames

[skearney414]

Trying to get this working too, it’s not an authentication issue is it ?

[yowmamasita]

It's not, it's just the bare-bones implementation of webdav protocol that I did for performance that is causing this

[skearney414]

Yeah perfect thanks for coming back, was able to download rclone and run it locally instead. Easier to have mount local instead of docker so you can use native plex app with hardware transcoding

[skearney414]

FYi @yowmamasita Have been looking at this and seems all you need is a head response, was able to pull this from logs outer Method not allowed: HEAD /dav/ map[Accept-Encoding:[gzip,deflate] Authorization:[Basic YW5vbnltb3VzOmN5YmVyZHVja0BleGFtcGxlLm5ldA==] Connection:[Keep-Alive] User-Agent:[Mountain Duck/4.15.3.21820 (Mac OS X/14.2.1) (x86_64)]] once a head response is given should work :) might know already but said I would send on

[JurajNyiri]

This would be super helpful if we had it, especially because of often times rclone crashing on macos rendering the full scans impossible (everything gets deleted in plex because the mount dissapears) - see https://forum.rclone.org/t/rclone-keeps-disconnecting-using-zurg/43953 .

[JurajNyiri]

Here is a little bit more further research I have done to try and resolve the rclone crashes.

Docker compose

• Docker compose never worked for me, it basically kills the whole docker with 500 error

Docker Rclone volume plugin

I tried to create a new volume using https://rclone.org/docker/, needed to do some special steps due to how docker works on mac OS:

sudo mkdir -p /var/lib/docker-plugins/rclone/config                                          
sudo mkdir -p /var/lib/docker-plugins/rclone/cache 

docker plugin install rclone/docker-volume-rclone:arm64 \                                    
    --alias rclone --grant-all-permissions args="-v" \
    config="/private/var/lib/docker-plugins/rclone/config" \
    cache="/private/var/lib/docker-plugins/rclone/cache"

sudo chown -R $(whoami):staff /var/lib/docker-plugins/rclone 

docker volume create zurg -d rclone -o type=http -o http-url=http://192.168.100.19:9999/http/

Now this works and I can use the volume inside containers, however there is no way that I know of on how to use the volume on host system as volume propagation is not supported on macos.

I could move my plex into the docker but then I would need to solve the hw acceleration etc.

nfsmount with fuse-t

I installed https://www.fuse-t.org/ and tried using rclone nfsmount. It seemed to work in filesystem but plex never starts a movie...

[JurajNyiri]

I have tried to move my zurg/rclone setup to synology NAS - successfully. However, when scanning through Plex, mac seems to disconnect the network drive resulting in scan failure and all media becoming unavailable. I cannot move Plex because of lack of hw for transcoding.

[JurajNyiri]

For documenting purposes, here is how one can use docker rclone on mac to expose webdav to host os:

docker run -p 8080:8080 -v /CHANGE_ME/rclone.conf:/config/rclone.conf --rm rclone/rclone --config /config/rclone.conf --addr 0.0.0.0:8080 serve webdav zurg:

[JurajNyiri]

After adding the vfs-cache, plex scan seems to be working well for a few minutes now.

Running rclone natively, without docker.

sudo rclone --config /REPLACE_ME/rclone.conf --dir-cache-time=20s --vfs-cache-mode full --cache-dir="/REPLACE_ME/Zurg-Tmp" --addr 0.0.0.0:8080 serve webdav zurg:

"zurg" is the HTTP endpoint in rclone.

I created a new library that is now being fully added for all my movies. I will keep it running during the night. Fingers crossed.

On testing with rsync, I am getting near gigabit speeds and seem to be rock stable so far.

Internet connection maxed during scan, finally!

[JurajNyiri]

I think I am in a pretty good state now. I changed my mount for the webdav native mount with following script in cron, solving the plex scan issues:

WEBDAV_SERVER="127.0.0.1:8080"
mntpoint="/REPLACE_ME/"

curl -s --head $WEBDAV_SERVER > /dev/null
if [ $? -eq 0 ]; then
    echo "WebDAV server at $WEBDAV_SERVER is available."
else
    echo "WebDAV server at $WEBDAV_SERVER is not available."
    /usr/sbin/diskutil unmount force "${mntpoint}/Zurg"
    /Users/jurajnyiri/rclone \
        --config /REPLACE_ME_2/rclone.conf \
        --dir-cache-time=20s \
        --vfs-cache-mode full \
        --cache-dir="/REPLACE_ME/Zurg-Tmp" \
        --vfs-cache-max-size 50G \
        --addr 0.0.0.0:8080 \
        serve webdav zurg: &
    sleep 10
    /sbin/mount_webdav http://127.0.0.1:8080/ "${mntpoint}/Zurg"
fi

Then I found that when I run rdtclient it tends to kill the native webdav mount, possibly due to docker layer on mac mini. However, that was not the case when I was using the rclone mount.
Therefore I created a bit of a Frankenstein monster.

Rclone mount script that runs on cron every minute in addition to above:

media="/REPLACE_ME_4"
mntpoint="/REPLACE_ME"
mkdir -p "$mntpoint/ZurgDirect"

mkdir -p "${media}/tmp/rclone_cache_Zurg"

result1=$( ls -la "${mntpoint}/ZurgDirect" 2>&1 | grep "fts_read: Input/output error" )
result2=$( ls -la "${mntpoint}/ZurgDirect" 2>&1 | grep "fts_read: Permission denied" )
result3=$( ls -la "${mntpoint}/ZurgDirect" | grep "all" )
if [ -n "$result1" ] || [ -n "$result2" ] || [ -z "$result3" ]; then
    echo "Mounting ${mntpoint}/ZurgDirect..."
    echo "Mounting ${mntpoint}/ZurgDirect..." > "$media"rclone-cron.log
    /usr/sbin/diskutil unmount force "${mntpoint}/ZurgDirect"
    rm -rf "${media}/tmp/rclone_cache_Zurg"
    /Users/jurajnyiri/rclone mount \
        --config=/REPLACE_ME_2/rclone.conf \
        --log-level=INFO \
        --log-file="$media"rclone-mount.log \
        --gid=1000 \
        --uid=1000 \
        --allow-other \
        --cache-dir="${media}/tmp/rclone_cache_Zurg" \
        --dir-cache-time=20s \
        --vfs-cache-max-size 50G \
        --vfs-cache-mode full \
	zurg: "${mntpoint}/ZurgDirect" &
fi

And finally the docker containers;

Note: I am using rogerfar/rdtclient:latest not the pukabyte since that does not work on Mac Mini M1.

docker run -d --pull=always \
    -v /REPLACE_ME_3/rdtClient5:/data \
    -v /REPLACE_ME/ZurgDirect:/REPLACE_ME/Zurg \
    -v /REPLACE_ME_4/symlinks:/data/downloads \
	--log-driver json-file \
	--log-opt max-size=10m \
	-p 6500:6500 \
	--name rdtclient \
    --restart unless-stopped \
	rogerfar/rdtclient:latest

docker run -d \
  --name=radarr \
  -e TZ=Etc/UTC \
  -p 7878:7878 \
  -v /REPLACE_ME_3/RadarrDocker2:/config \
  -v "/REPLACE_ME/ZurgDirect":"/REPLACE_ME/Zurg" \
  -v /REPLACE_ME_4/symlinks:/data/downloads \
  -v "/REPLACE_ME_4/movies":/movies \
  --restart unless-stopped \
  lscr.io/linuxserver/radarr:latest

docker run -d \
    --name=sonarr \
    -e TZ=Etc/UTC \
    -p 8989:8989 \
    -v /REPLACE_ME_3/SonarrDockerZurg:/config \
    -v "/REPLACE_ME/ZurgDirect":"/REPLACE_ME/Zurg" \
    -v /REPLACE_ME_4/symlinks:/data/downloads \
    -v "/REPLACE_ME_4/shows":/shows \
    --restart unless-stopped \
    lscr.io/linuxserver/sonarr:latest

[maxexcloo]

Did anything happen with this?

[akhanalcs]

@JurajNyiri Thank you for those scripts. I downloaded zurg and rclone binaries and ran them locally.

This is where I put them:

home-media-folder
└── non-docker-apps
    ├── rclone
    │    ├── rclone <------ binary
    │    └── rclone.conf
    └── zurg
        ├── zurg <------ binary
        └── config.yml

I ran Zurg first:

Guys-MacBook-Pro:zurg guy$ ./zurg

And then rclone:

Guys-MacBook-Pro:zurg guy$ ./rclone serve webdav zurg: --addr 0.0.0.0:8080 --dir-cache-time 20s --vfs-cache-mode full --vfs-cache-max-size 10G --cache-dir="/Users/fastesthorse/ZurgTmp" 

And then I connected to http://localhost:8080 from Finder -> Go -> Connect to a server. And I can see my RD files at: /Volumes/localhost. It looks good so far.
But when I play something from it, the rclone logs show a bunch of errors (even though the files are playing fine). What's causing these errors? And is there a way to stop them? @yowmamasita

Guys-MacBook-Pro:rclone guy$ ./rclone serve webdav zurg: --dir-cache-time 20s --vfs-cache-mode full --vfs-cache-max-size 50G --cache-dir="/Users/ashishkhanal/ZurgTmp" --addr 0.0.0.0:8080
2024/10/01 19:03:06 NOTICE: webdav root '': WebDav Server started on [http://[::]:8080/]
2024/10/01 19:04:32 ERROR : ._.DS_Store: Failed to copy: 404 page not found: 404 Not Found
2024/10/01 19:04:32 ERROR : ._.DS_Store: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: 404 page not found: 404 Not Found
2024/10/01 19:04:32 ERROR : .DS_Store: Failed to copy: 404 page not found: 404 Not Found
2024/10/01 19:04:32 ERROR : .DS_Store: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: 404 page not found: 404 Not Found
2024/10/01 19:04:43 ERROR : ._.DS_Store: Failed to copy: 404 page not found: 404 Not Found
2024/10/01 19:04:43 ERROR : ._.DS_Store: vfs cache: failed to upload try #2, will retry in 20s: vfs cache: failed to transfer file from cache to remote: 404 page not found: 404 Not Found
2024/10/01 19:04:47 ERROR : .DS_Store: Failed to copy: 404 page not found: 404 Not Found
2024/10/01 19:04:47 ERROR : .DS_Store: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: 404 page not found: 404 Not Found
2024/10/01 19:05:33 ERROR : rdfiles/Some.Movie.Name.2016.2160p.UHD.BluRay.X265/._Some.Movie.Name.2016.2160p.UHD.BluRay.X265.mkv: Failed to copy: Method not allowed: 405 Method Not Allowed
2024/10/01 19:05:33 ERROR : rdfiles/Some.Movie.Name.2016.2160p.UHD.BluRay.X265/.Some.Movie.Name.2016.2160p.UHD.BluRay.X265.mkv: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: Method not allowed: 405 Method Not Allowed

It keeps on going on and on. Maybe this is impacting performance?

I'm on M1 Max Sonoma 14.6.1. Rclone version: v1.68.1. Zurg version: v0.9.3-final.

[akhanalcs]

@JurajNyiri Is there a way I can reach out to you? Like Discord or Reddit or somewhere I can chat with you?

I'm unable to setup Zurg, Rclone and RDTClient on a M1 Max MBP.
Whenever I try mount my rclone mount as a volume to my RDTClient and Radarr containers, my Radarr and RDTClient containers refuse to start up.

I'd greatly appreciate your help.