Ensure rsync is completed before data processing

[clemgoyens]

@corizzi
Data are transferred from the rugged PC to the server. On the server we have the automatic scheduler running... We often have an error mentioning that the "metadata.txt" file is missing. However, when looking at the raw data, the metadata.txt file is there (and processor runs fine when done manually). So we think that the error arises because the rsync is not yet completed. If this is the case, do you know a way to check if the data has been transferred completely from the rugged PC to the server with rsync? Or any idea how we could avoid processing when data transfer from a single sequence is not yet completed? Thanks!

[jkuusk]

A simple workaround that first came to my mind is syncing some lock file (e.g. the same metadata.txt) separately after syncing the rest. I.e. exclude the metadata.txt from the main sync and add another rsync call to the end of the script that syncs only the metadata.txt. If metadata.txt exists on the server then (unless the first rsync failed) the rest of the data should be also fully synced to the server.

[shunt16]

Opened issue for this on hypernets_tools - can't progress here until that is closed

[pdevis]

The log files are synced after the data, so if any of the log files are there, we can start processing the data. Should be sufficient for us right? I have closed the issue on hypernets_tools.