Does Spacedrive require all operations to go through it, to keep its database up to date?

[Dieterbe]

Hello!
I've read the docs on the website and the readme, but have not quite been able to figure this out.
(I haven't ran the app yet, as it's still so early)

When using spacedrive, do you need to do all your file operations (moves, deletes, etc) through it?
I'm a heavy user of shell scripts and commands to manage my files. Does this mean spacedrive needs to do periodic rescans ? Or is there a clever trick like maybe a fuse or LD_PRELOAD layer that can see all i/o calls and update the database accordingly?

[utkubakir]

We have location watchers that rescan the location when you navigate into it. In the future, we plan to run scan jobs periodically too.

I would love to know more about the clever trick by the way, we can and will always improve how we do things :)

[fogodev]

Also, we plan to have our core always running on each OS, starting up alongside the user's system if they want it. With this, we will manage to always keep our database up to date, as we already have the mentioned watcher which received file system events for each location.

[Dieterbe]

I would love to know more about the clever trick by the way, we can and will always improve how we do things :)

i was thinking the user could run their own applications and shells with an LD_PRELOAD or, access the filesystem through a custom fuse filesystem (controlled by spacedrive), which would allow you to intercept all systems calls relating to the filesystem, and therefore update spacedrive's metadata/database in real time any time a change happens. This is somewhat clunky but could guarantee atomic updates (you can have the fs operations only complete when spacedrive is also updated) and IIRC doesn't require root (except maybe to do initial fuse setup, i suppose most linux distros don't have it on by default), it's also specific to linux.

The solution mentioned by @fogodev to use filesystem watchers, makes sense. I had a quick look at the repository and saw inotify is used. I've used this in the past but quickly ran into watcher limits, which requires root to adjust (and in my experience, no limit is ever high enough, especially if you have many small-ish files), but IIRC it's async (which may not be an issue here) and doesn't support recursion properly. But i think there might be a more modern replacement by now on linux.

[fogodev]

We're using the https://docs.rs/notify crate to abstract away the specific implementation regarding each OS, we haven't received a report complaining about watcher limits yet, we set only a single a watcher per location. But can't say for sure how the notify crate interacts with inotify on Linux.

[Dieterbe]

https://docs.rs/notify/latest/notify/#watching-large-directories says this:

When watching a very large amount of files, notify may fail to receive all events. For example the linux backend is documented to not be a 100% reliable source. See also issue notify-rs/notify#412.

😨

[fogodev]

I'm aware, I was just noting that we haven't received users reporting from our watcher failing yet haha. But as Utku mentioned, we do shallow re-scans every time the user navigates through the app. I have some ideas to further improve the performance of our directory walker, as faster scanning huge directories would enable us to run re-scans periodically too.

The idea of custom fuse filesystem to intercept syscalls seems interesting, but I would like to avoid a very OS specific big feature for now. But we don't discard this idea in the future to support very big and dynamic environment, as a single spacedrive server where multiple users keep interacting with many files at the same time. The watcher for example already requires more OS specific code than I like lol

[Dieterbe]

Okay, thanks to both of you for the interesting discussion.

Sounds like spacedrive is eventually consistent wrt to the filesystem (as in, it will catch up, but with a possible delay). I was wondering if this can lead to any issues, but sounds like it's designed to make it a non issue. (but like for example, what happens if in spacedrive i view a directory and delete it, but it didn't show a file yet that another process has just finished adding to the directory)