Bad behaviour if the disk is full

[llemeurfr]

From a user:

I borrowed books from libraries which were in Thorium. I would like to know if it is normal that everything disappeared when the computer is overcharged which is my case. I am able to download new stuff from libraries on Thorium but I lost what was there before. Is there a way to have everything back? If I manage to not overcharge my computer would this solve the problem?

[panaC]

If the disk is full or in read only, that seems like a normal behavior to me. We write a json file to the filesystem which contains the db, if the database and his backup file cannot be writen a new fresh database state is loaded. See getPath from electron.

[llemeurfr]

It is NOT normal behaviour for an application to loose its database.

[danielweck]

The "disk full" scenario is difficult to handle predictably due to the complexity of filesystem I/O in Thorium: the application code (NodeJS Javascript) reads/writes from the storage medium at regular points in time (this includes the central configuration database and bookshelf information, now all in JSON files instead of database formats like SQLite) ... but there is also lots of disk activity outside of our direct control in the Electron and Chromium subsystems (data cache).

Worst case scenario, if Thorium's own "database" is corrupted there is always the possibility of recovering all the publications from the publications subfolder inside the EDRLab.ThoriumReader folder:

• Windows: %APPDATA%
• Linux: $XDG_CONFIG_HOME or ~/.config
• Mac: ~/Library/Application Support

Technical reference (because Thorium is an Electron app):
https://www.electronjs.org/docs/latest/api/app#appgetpathname

[danielweck]

I converted this to a "discussion" as this is currently not an actionable issue (i.e. we cannot effectively fix a bug for a non-reproducible problem).
However there is a worthy discussion to be had about data loss/corruption in edge case scenarios for example "disk full". I personally experienced this with compiler software in Linux virtual machines a few times, the behaviour is "undefined" as there is no clear strategy on how to handle this catastrophic event across the entire system (in the case of Thorium, like I mentioned above there are multiple I/O pathways into the filesystem).

[llemeurfr]

A solution could be to stop accepting imports if the free volume on the disk that is used for Thorium storage is lower than an ad-hoc minimum (e.g. 200 Mb).

[danielweck]

My thoughts: this would not be a solution, because the disk can be filled excessively by an external factor (on my Linux virtual machines where I experienced this problem due to non-expandable / fixed-size VM volume, the storage gobblers were running Docker containers with growing sparse files, NPM and APT caches or tmp folders during updates, ... and of course my own compiler process which was creating a myriad of temporary files, eating RAM swap too).

In addition to "importing" publications, Thorium would need to monitor "every" filesystem write from its own code (and possibly its own Electron and Chromium subsystems too) in order to skip an attempt to write to a file handle when nearing the zero storage point. In fact because of concurrency and race conditions, even a negative check for "disk nearly full" could result in failure. Behaviour will vary from Windows/NTFS to Linux/ext4 to Mac/APFS so this would be difficult to test anyway.

I think a much more reliable approach would be to write atomically and in case of failure provide a guarantee that the old persistent state can be restored (redundancy, preserve a snapshot of the previous dataset on disk).

Thorium already writes to disk atomically (no streaming of large dataset, the "database" is just a JSON file) via the NodeJS APIs if I remember correctly.

I wonder what exactly happened on the user's computer. Were they trying to import multiple publications or just a single one? The problem could have just as well happened AFTER import during an operation that attempted to persist the current application state from memory to disk.