Desktop: File system mirroring

[personalizedrefrigerator]

Summary

This pull request adds support for syncing a folder and its subfolders with the file system. This has a few differences from existing file system sync:

• Markdown files in the folder are editable by a user. Changes to the files are synced back to Joplin.
• Directory structure is preserved.
• The folder is watched. Joplin applies the changes to local notes without needing a full sync.
• It's possible to sync a single folder, rather than all folders.

Development: New files/classes

New classes

At present, file system mirroring involves several different objects:

1. FolderMirroringService
2. FolderMirror
3. LinkTracker
4. ItemTree

FolderMirror

A FolderMirror is responsible for syncing one notebook and all of its subnotebooks with a filesystem directory. It is designed to do two things:

1. An initial full sync.
2. Watch for database and file system changes and update the database/file system in response.

Internally, it uses two ItemTrees to keep track of the state. One represents the database state and the other the file system state. This is done because paths can be different (particularly when starting a new fullSync) in the file system and database.

ItemTree

ItemTrees map from paths to folder and note entities and from IDs to paths. Internally, all paths are POSIX paths relative to the root output folder. For example, out/a.md.

An ItemTree has various methods for manipulating the tree. For example, move and deleteItemAtPath. Each such method takes a listeners parameter that the caller uses to determine what effect (if any) the method should have.

For example, different listeners can make a call update the database/file system, or just the tree:

// Deletes from the tree representing the local
// database. Does nothing on complete.
await localTree.deleteAtPath(localPath, noOpActionListeners);

// Deletes from the remote tree (represents the file system).
// The modifyRemote argument contains listeners that cause the changes
// to be written to the file system, after the tree is modified.
await remoteTree.deleteAtPath(remotePath, modifyRemote);

LinkTracker

Each FolderMirror has a LinkTracker. LinkTrackers maintain a graph of links between notes. After a full sync, the local and remote graphs should be identical. As such, a link tracker is created only for the remote tree.

Note: File system notes use paths for links (e.g. [link](../target.md)) instead of IDs (e.g. [link](:/0123456789abcdef0123456789abcdef)).

When a note is renamed, the LinkTracker determines what other notes need to be updated with the new path. It also handles conversion between ID links and path links.

FolderMirroringService

The FolderMirroringService is responsible for creating and managing FolderMirror objects. Each FolderMirror is responsible for mirroring a notebook and its subnotebooks to a single directory in the file system.

The service ensures that multiple FolderMirrors aren't created for the same directory and, if a FolderMirror for a (folder, destination) already exists, re-uses that mirror for future calls.

The FolderMirroringService is also responsible for forwarding database events to individual FolderMirror objects.

Screen recording

file-mirror-demo.mp4

To-do

• Include resources in the mirrored folder.
• Don't use separate metadata files.
  • Rather than relying on .metadata.yml files for resources and folders, try to guess the ID of folders and resources based on the notes that they contain or that reference them.
  • For example, notes within a folder all have an id property. If these notes already exist within Joplin, then the parent_id of the majority of these notes ID of their parent folder in Joplin can provide the ID of the remote folder.
  • This will require some refactoring.
• Allow changing tags associated with notes in the mirrored folder.
• UI
  • Allow mirroring all folders.
  • Preserve which folders are mirrored when restarting Joplin?
• Bug fixes
  • (Existing Joplin bug) Note does not update when changed in DB.
  • Bugs related to same-name items.
  • Updating a resource's metadata file might not update the resource's metadata in Joplin until the resource is modified.
  • Moving an item in the file system doesn't update its links to other items.
  • Deleting an item remotely when Joplin isn't running will cause the item to be re-created remotely when Joplin syncs with the output directory again.
  • Windows-specific bugs
    • Joplin MD exporter - data loss due to note titles #10525 will probably apply to this.
    • ...
  • ...
• Move some changes to other pull requests.
  • Changes to AsyncActionQueue
  • Frontmatter-related refactoring

[personalizedrefrigerator]

On one hand, seeing Joplin work like Obsidian by treating notes like files while being a superior note app is awesome.

On the other hand, I need my notes encrypted. So... any plans to make it work with Cryptomator on mobile? On the desktop that would not be an issue.

I'm linking to the relevant discussion on Lemmy: https://sopuli.xyz/post/12850169

For now, this pull request focuses on desktop and CLI. On Android (when it's implemented :) ), selecting a directory to sync with will probably be similar to the existing file system sync logic. As such, if the current file system sync works with Cryptomator, then this should, too.