Home/group directory in JuiceFS

[netrc]

Does anyone have an example/best-practices for having a home directory stored in JuiceFS?

[netrc]

If nothing has been documented, I'd be happy to work on this with other Juicers as a side project.....

[netrc]

FWIW - I've been using Juicefs home dir (AWS S3 bucket) for about a year with no noticeable issues. Also, have stored many binaries in a sw/ directory (docker, flyio, java, linux tools, lua, node, tailscale, tcl, vim, etc), and using those binaries for day to day development. Paths are managed with modulefiles.

In a similar way, could be nice to have juicefs make their binaries, docs, tools, available from an S3 bucket we could mount.

[Eris7090]

I've been wanted to do this for years but never could decide on what filesystem to use. I just found juicefs. How has it been?

[netrc]

long reply....

I love it; it's become my household (and more) standard file system. The setup is pretty simple - just get a juicefs cloud account, juicefs script, S3 bucket, and attach as a Linux FUSE filesystem. Done.

With this, I have access to exact same cloud filesystem from all my Linux and Windows devices. My linux devices include at home, at a cloud provider, I've also attached from e.g. google cloud shell, my fly.io dev deployments, and even WSL. Note this means that I don't "deploy" code to google cloud shell or fly.io, I just have to configure a run script to run whatever software I want.

I have a spare Linux notebook server at home running Samba which is how my family general Windows machines can access all their files, our shared files, from one well known mount. Again performance is fine. Media files play with no problem.

I have three buckets I make available from a single top-level directory, e.g. /jfs : /jfs/home, /jfs/sw, and /jfs/media -- home directories, installed software, media photos, music, movies.

Some use cases:

• From the web, my downloads (on PC or linux) go to /jfs/home/ric/Downloads/web - and that directory of downloads is visible across my household. Easy to download something on PC and then run other scripts/tools on it from Linux.
• multiple software tools with multiple versions are in /jfs/sw/, e.g. /jfs/sw/node. I can easily switch between any versions needed trivially. I use https://github.com/envmodules/modules as a standard way to manage environment. So any tool in /jfs/sw is available to all my systems. No matter where you go, there you are.

Notes:

1. community juicefs has more features than juicefs.com cloud including Windows client. The big difference is that juicefs.com (so far) provides the central metadata database for you (and for which you pay); in community version, you need to roll your own DB cluster.

As far as it goes, my S3 buckets are in AWS, which suffices. Performance is fine, and the durability is unmatched (eleven 9s if fully configured). For my household stuff, durability is more important than just uptime, though I don't recall any problem ever reading/writing Juicefs due to AWS. I could conceivably get better performance with a more local (NYC area) cloud S3 provider but haven't needed to check this out.

As far as paying juicefs.com for their metadata database, I'd rather pay the nominal monthly fee so as to reduce the number of services I have to admin at home. Setting up a redis or postgres cluster just isn't worth it - and I'd still need to back that up somewhere as well.

2. While juicefs performance is fine. Accessing most tools, binaries, files through /jfs mount is more than fast enough. That said, there are some tools that provoke bad performance. Any interpreter that insists on accessing thousands of imported files will have slow startup times due to thousands of metadata access across the internet. So, something like the 'aws' CLI or node code that imports node_modules stored in /jfs takes several seconds to start. For misc work, it's still nice to not have to deploy code to containers or tar/untar everywhere I go, but it not perfect. As far as I can see, Juicefs file caching performance is fine but that doesn't extend to metadata. I'd like an option to make a mount purely RO, so local metadata caching work be just about as fast as local, but can't find out how to optimize that.

Note that the /jfs/sw filesystem is designed (manually) to be be append only - in other words, add new software, add new versions of software, effectively never update software, so local RO caching in theory should work. (With so many versions of software, that's what envmodules/modules helps with)

And, finally, there is some software with hardcoded configurations that just doesn't fit in to my /jfs. Sometimes I have to install local; mostly I just drop that software and find something else.

3. Each juicefs mount takes a decent chunk of system memory. I would have liked to have had more S3 buckets for some finer-grained access control, but have to limit number of JFS mounts to keep resources down. For example, at fly.io deployments I just mount the one /jfs/sw bucket to keep under resource limits. Would like to have everything mounted, but would have to pay for a bigger VM.

So, working great for me. Don't even think about it much. Saves a lot of time when the files are just where you left them.

(All that said, I've been using distributed file systems for 40 years and just expect it to work. Have installed enterprise DFS several times. Can't live without it. And interesting to see so many tools these days try to internalize file copying and caching - from npx copying scripts into your ~/.npm dir, etc, etc.)