Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wip] fix: moraine filesystem mountpoints #101

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

[wip] fix: moraine filesystem mountpoints #101

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b41e43a
fix(shell:aliases|nu): workaround to support exa aliases option
montchr Sep 6, 2023
7ccc444
docs(moraine): filesystem layout update
montchr Sep 4, 2023
3844528
fix(moraine): implement datastore filesystem layout
montchr Sep 4, 2023
1b82083
feat(moraine|fs|rclone): rclone mount silo with unix mount
montchr Sep 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 20 additions & 17 deletions machines/moraine/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Layout

Loosely based on TRaSH's advice [link](https://trash-guides.info/Hardlinks/How-to-setup-for/
Loosely based on TRaSH's advice [link](https://trash-guides.info/Hardlinks/How-to-setup-for/)

```shell-session
sudo chown -R $USER:$USER /data
Expand All @@ -11,22 +11,25 @@ sudo chmod -R a=,a+rX,u+w,g+w /data

```txt
.
├── mnt # │ │ │ │ │ │
│ └── silo # │ │ │ │ │ [5] rclone mount of storage box
└── srv # │ │ │ │ │ │
└── data # │ │ │ │ │ │
├── media # │ │ │ │ │ │
│ ├── incoming # │ │ │ │ [4] queued for starr processing
│ └── library # │ │ │ │ │ [5] symlink: /mnt/silo/data (tbd: unionfs)
│ ├── movies # │ │ │ │ │ [5] target for radarr
│ ├── music # │ │ │ │ │ [5] target for lidarr
│ ├── other # │ │ │ │ │ [5] tbd (archival, oddities)
│ └── tv # │ │ │ │ │ [5] target for sonarr
└── torrents # │ │ │ │ │ │
├── complete # │ │ │ [3] │ │
├── incoming # │ [1] │ │ │ │
├── metadata # │ │ [2] meta-info files for active torrents
└── watch # [0] torrent meta-info file intake
├── mnt
│ └── silo # │ │ │ │ │ [5] │
└── srv # │ │ │ │ │ │ │
└── data # │ │ │ │ │ │ │
├── media # │ │ │ │ │ │ │
│ ├── incoming # │ │ │ [3] │ │ │
│ ├── library # │ │ │ │ │ [5] │
│ │ ├── movies # │ │ │ │ │ [5] │
│ │ ├── music # │ │ │ │ │ [5] │
│ │ ├── other # │ │ │ │ │ [5] │
│ │ └── tv # │ │ │ │ │ [5] │
│ └── outgoing # │ │ │ │ [4] │ │
└── torrents # │ │ │ │ │ │ │
├── complete # │ │ [2] │ │ │ │
├── incoming # │ [1] │ │ │ │ │
├── metadata # │ [1] │ │ │ │ │
└── watch # [0] │ │ │ │ │ │
├── load # [0] │ │ │ │ │ │
└── start # [0] │ │ │ │ │ │
```

### `/mnt`
Expand Down
41 changes: 27 additions & 14 deletions machines/moraine/bin/provision.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,7 @@ export NEW_HOSTNAME=moraine
export FSOPTS="defaults,x-mount.mkdir,noatime"
export BTRFSOPTS="${FSOPTS},compress=zstd"

# Preserve the Cloudbox/Saltbox `/mnt/local/` structure for easier reference.
# Mounting a non-root fs under `/mnt` also seems like best practice.
export LOCAL_PREFIX="/mnt/mnt/local"
export LOCAL_PREFIX="/mnt/srv/"

# boot/root
export NVME01="/dev/nvme0n1"
Expand Down Expand Up @@ -220,20 +218,35 @@ btrfs device scan
mkdir -p "${LOCAL_PREFIX}"
mount -t btrfs LABEL=local "${LOCAL_PREFIX}"
btrfs subvolume create "${LOCAL_PREFIX}/@backups"
btrfs subvolume create "${LOCAL_PREFIX}/@completed"
btrfs subvolume create "${LOCAL_PREFIX}/@torrents"
btrfs subvolume create "${LOCAL_PREFIX}/@media"
btrfs subvolume create "${LOCAL_PREFIX}/@bt-completed"
btrfs subvolume create "${LOCAL_PREFIX}/@bt-incoming"
btrfs subvolume create "${LOCAL_PREFIX}/@bt-metadata"
btrfs subvolume create "${LOCAL_PREFIX}/@bt-watch"
btrfs subvolume create "${LOCAL_PREFIX}/@media-incoming"
btrfs subvolume create "${LOCAL_PREFIX}/@media-outgoing"
btrfs subvolume list -a "${LOCAL_PREFIX}"
umount "${LOCAL_PREFIX}"

mount -t btrfs -o "subvol=@backups,${FSOPTS}" LABEL="local" \
"${LOCAL_PREFIX}/backups"
mount -t btrfs -o "subvol=@completed,${FSOPTS}" LABEL="local" \
"${LOCAL_PREFIX}/downloads/completed"
mount -t btrfs -o "subvol=@torrents,${FSOPTS}" LABEL="local" \
"${LOCAL_PREFIX}/downloads/torrents"
mount -t btrfs -o "subvol=@media,${FSOPTS}" LABEL="local" \
"${LOCAL_PREFIX}/Media"
mount -t btrfs -o "subvol=@backups,${BTRFSOPTS}" LABEL="local" \
"${LOCAL_PREFIX}/backups"

mount -t btrfs -o "subvol=@bt-completed,${BTRFSOPTS}" LABEL="local" \
"${LOCAL_PREFIX}/data/torrents/completed"

mount -t btrfs -o "subvol=@bt-incoming,${BTRFSOPTS}" LABEL="local" \
"${LOCAL_PREFIX}/data/torrents/incoming"

mount -t btrfs -o "subvol=@bt-metadata,${BTRFSOPTS}" LABEL="local" \
"${LOCAL_PREFIX}/data/torrents/metadata"

mount -t btrfs -o "subvol=@bt-watch,${BTRFSOPTS}" LABEL="local" \
"${LOCAL_PREFIX}/data/torrents/watch"

mount -t btrfs -o "subvol=@media-incoming,${BTRFSOPTS}" LABEL="local" \
"${LOCAL_PREFIX}/media/incoming"

mount -t btrfs -o "subvol=@media-outgoing,${BTRFSOPTS}" LABEL="local" \
"${LOCAL_PREFIX}/media/outgoing"

# Repeat mkdir since `/mnt` has since been umounted.
mkdir -p /mnt/boot
Expand Down
34 changes: 26 additions & 8 deletions machines/moraine/filesystems.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,31 +45,49 @@ in {
fileSystems."/var/lib/postgres" = {
device = "/dev/disk-by-label/nixos";
fsType = "btrfs";
options = commonOpts ++ ["subvol=@postgres" "ssd" "nofail"];
options = commonOpts ++ ["subvol=@postgres" "ssd"];
};

fileSystems."/mnt/local/backups" = {
fileSystems."/srv/backups" = {
device = "/dev/disk/by-label/local";
fsType = "btrfs";
options = commonOpts ++ ["subvol=@backups" "nofail"];
};

fileSystems."/mnt/local/downloads/completed" = {
fileSystems."/srv/data/torrents/completed" = {
device = "/dev/disk/by-label/local";
fsType = "btrfs";
options = commonOpts ++ ["subvol=@completed" "nofail"];
options = commonOpts ++ ["subvol=@bt-completed"];
};

fileSystems."/mnt/local/downloads/torrents" = {
fileSystems."/srv/data/torrents/incoming" = {
device = "/dev/disk/by-label/local";
fsType = "btrfs";
options = commonOpts ++ ["subvol=@torrents" "nofail"];
options = commonOpts ++ ["subvol=@bt-incoming"];
};

fileSystems."/mnt/local/Media" = {
fileSystems."/srv/data/torrents/metadata" = {
device = "/dev/disk/by-label/local";
fsType = "btrfs";
options = commonOpts ++ ["subvol=@media" "nofail"];
options = commonOpts ++ ["subvol=@bt-metadata"];
};

fileSystems."/srv/data/torrents/watch" = {
device = "/dev/disk/by-label/local";
fsType = "btrfs";
options = commonOpts ++ ["subvol=@bt-watch"];
};

fileSystems."/srv/media/incoming" = {
device = "/dev/disk/by-label/local";
fsType = "btrfs";
options = commonOpts ++ ["subvol=@media-incoming"];
};

fileSystems."/srv/media/outgoing" = {
device = "/dev/disk/by-label/local";
fsType = "btrfs";
options = commonOpts ++ ["subvol=@media-outgoing"];
};

fileSystems."/boot" = {
Expand Down
221 changes: 221 additions & 0 deletions machines/moraine/rclone-mount.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,221 @@
# FIXME: set up rclone remote declaratively (somehow)
# TODO: sshfs or samba/cifs?
### Sources:
# <https://github.com/nzbr/nixos/blob/0721f599a073e7bb5b3acf8a38bf415418188cbf/module/service/borgbackup.nix#L76C1-L105C11>
# <https://arne.me/writing/plex-on-nixos/>
# <https://rclone.org/commands/rclone_mount/#rclone-as-unix-mount-helper>
# <https://github.com/rclone/rclone/wiki/rclone-mount-helper-script>
# <https://github.com/rclone/rclone/wiki/Systemd-rclone-mount>
# <https://rclone.org/sftp/#hetzner-storage-box>
# <https://rclone.org/smb/>
# <https://docs.hetzner.com/robot/storage-box/access/access-overview>
# <https://docs.saltbox.dev/apps/hetzner_nfs/?h=hetzner>
# <https://docs.saltbox.dev/advanced/feeder/>
# <https://www.reddit.com/r/linuxquestions/comments/9gs0bm/samba_vs_nfs_vs_sshfs/>
# TODO: why not use `rclone union` remote type? saltbox does not seem to use this (why?) <https://rclone.org/union/>
# TODO: <https://github.com/saltyorg/Saltbox/blob/master/roles/remote/defaults/main.yml>
# TODO: <https://trash-guides.info/Hardlinks/Examples/>
# TODO: <https://github.com/saltyorg/Saltbox/blob/2a2d4ae8ea269526f5f5b5853876c106a9d22242/roles/unionfs/defaults/main.yml#L50-L55>
# TODO: <https://github.com/saltyorg/Saltbox/blob/2a2d4ae8ea269526f5f5b5853876c106a9d22242/roles/common/tasks/btrfs.yml#L30-L32>
#
# NOTE(rclone): Option --rc does not work together with --daemon or in mount.rclone mode. We are aware of this limitation and work on fixing it. It is tracked by tickets:
# https://github.com/rclone/rclone/issues/2618
# https://github.com/rclone/rclone/issues/5664
{pkgs, ...}: let
rcloneRemote = "silo";
runDir = "/srv/data/run";
mediaDir = "srv/data/media/";
# FIXME: should be unionfs
mountPath = "${mediaDir}/library";

# TODO: btrfs subvolume for cache?
# -- yea actually... what happens when it's NOT a subvol?
# i bet it falls back to the nvme `/` root vol, not the hdd vol...
cachePath = "${mediaDir}/cache";
in {
systemd.tmpfiles.rules = [
# FIXME: adjust to needs
"d ${cachePath} 0755 root root -"
];

system.fsPackages = [
# <https://rclone.org/commands/rclone_mount/#rclone-as-unix-mount-helper>
(pkgs.runCommand "mount.rclone" {} ''
mkdir -p $out/bin
ln -s ${pkgs.rclone}/bin/rclone $out/bin/mount.rclone
'')
];

################################
# Feeder
################################

# TODO:
# feeder_mount_start_command: |-
# /usr/bin/rclone mount \
# --user-agent='{{ user_agent }}' \
# --config={{ rclone_config_path }} \
# --allow-other \
# --copy-links \
# --dir-cache-time=1m \
# --max-read-ahead=200M \
# --umask=002 \
# --syslog \
# -v \
# feeder:/mnt/local /mnt/feeder

# NOTE: vfs mount
# rclone_vfs_mount_start_command: |-
# /usr/bin/rclone mount \
# --user-agent='{{ user_agent }}' \
# --config={{ rclone_config_path }} \
# --allow-other \
# --rc \
# --rc-no-auth \
# --rc-addr=localhost:5572 \
# --drive-skip-gdocs \
# --vfs-read-chunk-size=64M \
# --vfs-read-chunk-size-limit=2048M \
# --buffer-size=64M \
# --poll-interval=15s \
# --dir-cache-time=1000h \
# --timeout=10m \
# --drive-chunk-size=64M \
# --drive-pacer-min-sleep=10ms \
# --drive-pacer-burst=1000 \
# --umask=002 \
# --syslog \
# {{ '--bind=' + ansible_default_ipv4.address + ' ' if mounts.ipv4_only else '' }}-v \
# {{ rclone.remote }}: /mnt/remote

# NOTE(rclone): > The umount operation can fail, for example when the
# > mountpoint is busy. When that happens, it is the user's
# > responsibility to stop the mount manually."
# rclone_vfs_mount_stop_command: /bin/fusermount -uz /mnt/remote

# rclone_vfs_mount_start_post_command: |-
# /usr/bin/rclone rc vfs/refresh recursive=true --url http://localhost:5572 _async=true

# NOTE: vfs cache
fileSystems."/mnt/silo" = {
fsType = "rclone";
# via <https://github.com/saltyorg/Saltbox/blob/master/roles/remote/defaults/main.yml>
options = [
"config=${rcloneConf}"
"allow-other"
"async-read=true"
"dir-cache-time=1000h"
"buffer-size=32M"
"poll-interval=15s"
"rc"
"rc-no-auth"
"rc-addr=localhost:5572"
"use-mmap"
"vfs-read-ahead=128M"
"vfs-read-chunk-size=32M"
"vfs-read-chunk-size-limit=2G"
# FIXME:"vfs-cache-max-age=???????????????"
"vfs-cache-mode=full"
"vfs-cache-poll-interval=30s"
# FIXME: "vfs-cache-max-size=???????????"
"timeout=10m"
# FIXME: verify umask
"umask=002"
"cache-dir=${vfsCacheDir}"
# FIXME: maybe? "syslog"
# FIXME: what? "bind=${ipv4} ..... if mounts.ipv4only else '' -v"
# TODO: remove probably: "user-agent=????????";
"${rcloneSiloRemote}: "
# FIXME: prob unnecessary "mountpoint"
];
# FIXME: translate above:
# /usr/bin/rclone mount \
# --user-agent='{{ user_agent }}' \
# --config={{ rclone_config_path }} \
# --allow-other \
# --async-read=true \
# --dir-cache-time=1000h \
# --buffer-size=32M \
# --poll-interval=15s \
# --rc \
# --rc-no-auth \
# --rc-addr=localhost:5572 \
# --use-mmap \
# --vfs-read-ahead=128M \
# --vfs-read-chunk-size=32M \
# --vfs-read-chunk-size-limit=2G \
# --vfs-cache-max-age={{ rclone_vfs_cache_max_age }} \
# --vfs-cache-mode=full \
# --vfs-cache-poll-interval=30s \
# --vfs-cache-max-size={{ rclone_vfs_cache_max_size }} \
# --timeout=10m \
# --drive-skip-gdocs \
# --drive-pacer-min-sleep=10ms \
# --drive-pacer-burst=1000 \
# --umask=002 \
# {{ '--cache-dir=' + rclone_vfs_cache_dir + ' ' if (rclone_vfs_cache_dir | length > 0) else "" }}--syslog \
# {{ '--bind=' + ansible_default_ipv4.address + ' ' if mounts.ipv4_only else "" }}-v \
# {{ rclone.remote }}: /mnt/remote
};

# TODO:
# rclone_vfs_cache_mount_stop_command: /bin/fusermount -uz /mnt/remote

# TODO:
# rclone_vfs_cache_mount_start_post_command: |-
# /usr/bin/rclone rc vfs/refresh recursive=true --url http://localhost:5572 _async=true

# TODO:
# rclone_vfs_refresh_interval: 86400
# rclone_vfs_refresh_command: |-
# /usr/bin/rclone rc vfs/refresh recursive=true --url http://localhost:5572 _async=true

#############################################################################
# TODO: <https://github.com/saltyorg/Saltbox/blob/2a2d4ae8ea269526f5f5b5853876c106a9d22242/roles/unionfs/defaults/main.yml#L50-L55>

# local_mount_branch: "{{ '/mnt/feeder' if (feeder_remote_is_active is defined and feeder_remote_is_active) else '/mnt/local' }}"

# mount_type: "mergerfs"

# mergerfs_mount_branches: "{{ local_mount_branch }}=RW:/mnt/remote=NC"

# mergerfs_mount_service_after: "network-online.target"

# mergerfs_mount_start_command: |-
# /usr/bin/mergerfs \
# -o category.create=ff,async_read=true,cache.files=partial \
# -o dropcacheonclose=true,minfreespace=0,fsname=mergerfs \
# -o xattr=nosys,statfs=base,statfs_ignore=nc,umask=002,noatime \
# {{ mergerfs_mount_branches }} /mnt/unionfs

# mergerfs_mount_stop_command: /bin/fusermount -u /mnt/unionfs

###################################################################

# FIXME: set mode/group
# <https://trash-guides.info/Hardlinks/How-to-setup-for/Native/#permissions>
fileSystems.${mountPath} = {
# NOTE: Ends with `:`.
# <https://rclone.org/commands/rclone_mount/#rclone-as-unix-mount-helper>
device = "${rcloneRemote}:";
fsType = "rclone";
options = [
"user"
"noauto"
"_netdev"
"rw"
"allow_other"
"args2env"
"vfs-cache-mode=writes"
"vfs-cache-max-size=1G"
"config=/root/.config/rclone/rclone.conf"
"cache-dir=${cachePath}"
"log-file=${runDir}/rclone.log"
"x-systemd.automount"
"x-systemd.mount-timeout=5"
"x-systemd.idle-timeout=30"
];
};

environment.systemPackages = [pkgs.rclone];
}
Loading