disregarding "db/tx.17a6f6aeb948b94b", it is zero length

[ettom]

Occasionally, if a writer is cancelled, a 0-length file is left behind. This can be reproduced by running

for i in $(seq 0 100000); do echo "test $i $i" | timeout 0.01s sonnerie -d db add --format "u"; done

The timeout will probably need adjusting based on the speed of the filesystem, but on my machine this very quickly produces lots of empty transaction files.

I suppose the behaviour is to be expected and sonnerie can't really prevent this from happening (?), but from an UX perspective, perhaps compacting could also clean up these empty files? Maybe an option flag to do it automatically?

[njaard]

The reason there are empty files is that a filename needs to be reserved before the committed transaction file can atomically replace it. That prevents another writer getting the same filename which you then replace, deleting the other transaction.

An alternative to creating the "reservation files" is to use the option RENAME_NOREPLACE of the system call renameat2. The current strategy can be used in situations where RENAME_NOREPLACE isn't available.

[ettom]

What do you recommend as a solution for now? Is it safe to delete 0-length files before compacting?

[njaard]

Empty files can be deleted. Empty main files cannot be deleted because they are what sonnerie uses to identify a database.

[njaard]

@ettom I have committed the change that removes those eprintlns from the library and into the CLI.

I will leave this ticket open to track that those empty files are created in the first place. That would be an API-breaking change because commit_to assumes it's allowed to overwrite existing files, but in order to avoid creating empty files, it would be forbidden to do that. However, compactions replace one of the files they compact, either main or the last transaction file they replaced .

[ettom]

Thanks, sounds great! Would you mind creating a release?

[njaard]

released!