RESUMABLE: Clarify client recovery behavior for 409 Conflict responses

[danielresnick]

This issue is a spin-off from the discussion in #3193 regarding chunk granularity, as suggested by @guoye-zhang.

The core issue is that the current specification is ambiguous about whether clients need to handle a 409 Conflict response. This ambiguity can lead to upload failures in recoverable scenarios, undermining the specs goals.

Problem

A fully spec complaint client can receive a 409 Conflict response due to factors outside the client's direct control e.g. a network interruption causing the client and server to become desynchronized regarding the current Upload-Offset (client thinks a PATCH request has failed, but it actually succeeded).

The Concurrency section of the spec discusses this, and RECOMMENDS an approach to avoid ever serving clients 409 Conflict responses (latest request wins; previous requests are cancelled), however this is a non-binding recommendation and is acknowledged to be onerous / potentially impractical for servers to implement. A more realistic approach for large-scale servers to meet the requirements of the Concurrency section would be to use a strongly consistent database to maintain session state (current offset, etc) along with optimistic concurrency control to ensure versioning.

Even a proactive client can't avoid this race condition. Consider a scenario where a client's PATCH request times out locally, but is still in-flight to the server. To recover, the client sends a HEAD request to get the current offset. The server might respond to this HEAD request before it has processed the original, in-flight PATCH. By the time the client acts on that HEAD response and sends a new PATCH, the original one may have finally completed, advancing the server's offset and causing the new PATCH to be correctly rejected with a 409 Conflict.

As confirmed by @guoye-zhang, major client implementations like Apple's NSURLSession today treat all 4xx responses as fatal errors and do not attempt to retry / recover.

If a client gives up on a recoverable 409 Conflict, the protocol fails to deliver on its promise of reliable uploads over unreliable connections.

Proposed Solution

The spec should be updated to unambiguously state that a client MUST (or at least SHOULD) treat a 409 Conflict response as a recoverable signal.

The recovery mechanism would be via Upload-Offset in the 409 Conflict response and/or the Offset Retrieval mechanism.

Mandating this recovery logic (or at least strongly recommending it) would ensure that it is baked into foundational libraries, providing robust, out-of-the-box reliability for application developers who may not be aware of this specific failure case.

Keen for feedback from others as I’m not sure to what extent this has been discussed previously.

[guoye-zhang]

Today, the draft intends that 409 to be used for the cases where the server is certain that the client is non-compliant. E.g. resumed from a different offset from the preceding offset retrieval. It puts the burden on the server to ensure that a conflict can't occur when talking to a spec-compliant client.

If the server is uncertain that the client is compliant and simply wants it to try resuming an again, maybe a different status code can be used?

[smatsson]

We have discussed status codes and conflicts before and while I agree that the current spec idea with closing/ignoring older connections is ideal I do agree with @danielresnick here that there are some cases where some 4xx responses should be retriable, 409 being one and possibly even 412 Precondition failed in some cases. These are kind of oddballs not fitting into the whole "It's the client's fault" and I think they should be excluded from the list of status codes that should cause the client to abandon the upload. The only alternative as I see it is for the server to return a 5xx error on conflict, which is doable but will make it harder to debug what is going on. Just my 2 cents.

[danielresnick]

Today, the draft intends that 409 to be used for the cases where the server is certain that the client is non-compliant. E.g. resumed from a different offset from the preceding offset retrieval. It puts the burden on the server to ensure that a conflict can't occur when talking to a spec-compliant client.

Interesting. FWIW I didn't read the spec this way, so maybe the wording needs tweaking if this is the intention?

My core point though is that putting the entire burden on the server to prevent conflicts is unworkable for any large-scale service. The required statefulness (e.g., via exclusive locks or request cancellation) makes the system partition intolerant. During a network partition, such a system would be forced to sacrifice availability to maintain its state guarantees, which is a difficult trade-off for a global / large-scale upload service. As an example, AFAICT, resumeable upload services from both Google and Cloudflare don't offer such guarantees.

I guess it'd be possible to use a different status code but I'm not sure what it'd buy us over 409 Conflict which seems like the intended code for this sort of "state desynchronization" situation. Either way I think we need a mechanism for clients to handle this scenario gracefully and I agree with @smatsson that 5xx is less desirable.

[guoye-zhang]

Yeah, the draft may not be super clear on this. It treats state desynchronization as a server bug or a client bug, so does not allow the client to retry.

We could definitely retry 409, but let's think about the scenarios of desynchronization so we can design a retry strategy that works. The issue described in "Concurrency" is especially problematic since for example, the client switches from a high latency cellular network to a fast wifi network and decides to resume the upload over wifi, if the server is not able to discard the in-progress transfer, offset retrieving can keep returning invalid offsets. Currently URLSession performs 3 retries, and if none of them makes any forward progress, it considers the resumable upload failed.

[danielresnick]

Thanks @guoye-zhang! Apologies for the slow reply here - lots been going on.

In your example I agree the server should abort the in-flight (cellular) request as soon as it's aware of the new (wifi) one in order to meet the requirements of the spec around avoiding data corruption, etc.

I've drafted up a PR to (hopefully) help drive alignment on a path forward here. I've kept out specific retry strategies (backoff, num retries, etc) as I don't see a logical place for it in the spec at the moment, but curious if you think it's worth including.