tar_mirage: feature proposal for a set_streaming

[hannesm]

In our flagship project, opam-mirror, it'd be great to support the following use-case:

• we download a file to a temporary swap file system (i.e. we write the downloaded chunks to disk)
• we verify the validity of the checksum
• if the checksum is ok, we want to move that data over to the tar file system
• now, the big issue is about the API and failure safety: at any moment in time, there may be some power failure -- so we'd like the semantics of set -- i.e. first write the data, then as last elements the tar header -- so that at (nearly every) moment the tar is consistent, and the data that is written is correct / valid

Now, the question is what the API should be - maybe val set_streaming : t -> key -> size:int -> (bytes -> offset:int -> length:int -> (unit, error) Lwt.t) -> (unit, error) Lwt.t

Or should we rely on Lwt_stream.t? Maybe val set_stream : t -> key -> string Lwt_stream.t -> (unit, error) Lwt.t is a neat way to go?

It is a bit unclear to me what to do about ownership of buffers/data, and also who is in charge of the size of chunks?

A "size" argument is not really necessary -- we'll just keep the write_lock, so there's no other simultaneous writer -- also once the stream is finished, we know by heart how much data we wrote ;).

Another question is whether a similar read interface is useful and worth to implement (i.e. val get_stream : t -> key -> (string Lwt_stream.t, error) Lwt.t)? Again, who should control the chunk size, and what should be done with intermediate read errors?

I guess some more research / looking at other streaming APIs may be beneficial. If someone has nice pointers, please feel free to suggest them.