python/go grpc interface for async uploads

[andreasjansson]

Removes duplicate logic in Python, reusing the Go implementation via a grpc API.

Apologies for this massive PR, I couldn't see a way of splitting it up since it touches everything.

Closes #317
Closes #344

[bfirsh]

This is changing behavior I think isn't it? I think this got bumped because we wanted to come up with some sensible behavior for localhost, IIRC. #203

[bfirsh]

The problem in the previous review still applies, I think. The main problem, off the top of my head, is that experiments run on your laptop will have different hosts when you have different local IPs, which is very odd behavior. The "HOST" column will suddenly appear when you connect to a different network or get a new DHCP lease.

Maybe we should make RFC 1918 addresses the blank string until we come up with a better solution?

[andreasjansson]

I think this function always will return local addresses, so maybe the solution is to just blank it out for now. It's a shame since it's useful information when you're running on multiple hosts, but I can't think of a good solution.

[bfirsh]

Broadly looks good! I realize there is a lot of unfinished stuff in here so I won't review in any detail.

A few high-level thoughts:

• Have you thought about how the user interface might work? Maybe there is some of it in here but I can't see it obviously. I can think of things like displaying errors, what is printed when the training process is finished and things are still uploading, etc
• This is also currently unresolved, but have you put any more thought into partial writes? This might become more of an issue if things are done in the background. Checking out a partially written checkpoint might be particularly destructive (you could lose your current work, and the checked out stuff is corrupted!)
• We need to make sure there's a bit of developer documentation, otherwise it is going to very hard for people to add, e.g., a new bit of metadata to an experiment.

[andreasjansson]

Since the data hasn't yet been uploaded before saving the checkpoint, it doesn't have much use to the user. But I see your point about it being weird the way it looks now.

What I as a user want from the logs is to know that something successfully finished, not that it started. I want to trust that Replicate has uploaded my data and that the checkpoint is consistent.

I agree that it reads a little strange how messages show up out of order, but since uploads happen asynchronously I'm expecting that. In a sense it's nice, because it tells me that Replicate isn't blocking my training loop. Putting the step number in the Replicate log message would make that abundantly clear.

[bfirsh]

What I as a user want from the logs is to know that something successfully finished, not that it started. I want to trust that Replicate has uploaded my data and that the checkpoint is consistent.

On the other hand, you could argue that by Replicate not printing anything when you create a checkpoint, it looks broken because it doesn't print anything.

The broader point is that this is a change in behavior, and I don't think we should change behavior. The old behavior didn't print a message on success either.

[andreasjansson]

The broader point is that this is a change in behavior, and I don't think we should change behavior. The old behavior didn't print a message on success either.

There is a change in the logic though, in that checkpoints are now uploaded in the background. I think that should be reflected in the log output.

[bfirsh]

TODO, discussed on zoom: Copy to temp directory before uploading (and block)

[bfirsh]

I have created a bunch of issues for things mentioned in this PR. They are mentioned in the reference messages above.

[bfirsh]

TODO, discussed on zoom: Copy to temp directory before uploading (and block)

for future reference, this was fixed in https://github.com/replicate/replicate/pull/464