Avoid SpoolReader/SpoolWriter races with synchronized (atomic-like) updates to spool files

[zmousm]

munin-asyncd by default queries munin-node in turn for each plugin and writes to spool files. With --fork this happens in parallel. On the other hand munin-async iterates through spool files and scans them for new data.

Nothing stops spoolreader from reading from a file while (or before) spoolwriter writes to it in a particular round of updates. After such a spoolfetch, the master will have missed any updates written to spool files after spoolreader visited them. If it did fetch data that had already been written in that round, the master will also have bumped the spoolfetch timestamp and therefore it will skip over such data in subsequent spoolfetch, which means data loss.

This race condition is particularly evident when spoolfetch and spoolwriter coincide. For example if munin-update runs on the master every 5 minutes and munin-asyncd wakes up at the same time, the nodes visited by munin-update in the first 30-60 seconds are most likely to exhibit this problem. munin-asyncd --fork does help, but again if some munin plugins take a long time to run the respective services will be susceptible to data loss.

The proposed patch addresses the problem by emulating atomic-like updates to spool files. SpoolWriter writes updates to copies of spool files and moves (renames) them over in one go at the end. In fork mode, this involves pipe IPC where munin-asyncd reads files to be committed from its' children processes stdout.

[steveschnepp]

while I do agree with the issue, I'm not really at ease with heavy file copying.

It would be much more convenient to add some kind of locking via flock().

I also thought about enhancing the protocol inside the spoolfetch files to be able for the reader to detect a half-wrote dataset.

[kimheino]

Commit a48026a tries to fix this (or similar case?).

Well... I doesn't fix this, it is just a workaround to hide this problem.