List of a prefix 503s when a single object key contains an empty path segment (//) #116

Summary

When a ListObjectsV2 page returned by the backend contains a single object whose key has an empty path segment (a //), multistore fails the entire list request with 503 ServiceUnavailable instead of listing the other (valid) keys. One malformed object key takes out the whole prefix.

object_store's strict Path parser rejects empty segments, and multistore propagates that as a fatal backend error for the page rather than skipping the offending key.

Observed error

Proxy log:

[WARN] multistore::proxy: request failed {
  error=backend error: Encountered object with invalid path:
         Path "raw//" contained empty path segment,
  status=503, s3_code=ServiceUnavailable
}

Client just sees the generic:

An error occurred (ServiceUnavailable) when calling the ListObjectsV2 operation
(reached max retries: 2): Service unavailable

Reproduction

An object genuinely exists in S3 with a // in its key (double slash after raw/):

s3://.../raw//b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DELAYED-2045.001/atm/proc/tseries/day_1/....nc

aws s3 ls --no-sign-request against the raw bucket lists it fine (S3 keys are opaque byte strings and // is legal). Listing the same prefix through multistore 503s.

Only prefixes that contain such a key fail; sibling prefixes list normally.

Impact

A single stray upload with a double slash (or a zero-byte folder-marker object ending in //) makes an entire prefix unlistable through the gateway, with an error that gives the operator no hint about which object or why. S3 itself accepts and serves these keys, so the gateway is stricter than the backend it fronts.

Suggested fix

On list, don't let one unparseable key fail the page. Options:

• Skip keys that fail strict Path::parse, ideally logging the raw key at warn so operators can find and fix it; or
• Parse list keys leniently (Path::from / from_url_path, which tolerate empty segments) instead of strict parse.

At minimum, surface the offending raw key in the error so it's diagnosable without correlating backend logs.

Environment

• multistore 0.6.3 (features: azure), via multistore-cf-workers on Cloudflare Workers
• Backend: S3