Stop acknowledging upload progress after a failed storage flush

perfloop-oss/distribution · Avoidable Rejected Work

https://perfloop.ai/t/oss/case_kzxdr0dspc

Verdict

VERIFIED · settled 2026-11-11 · merged as distribution/distribution#5069

What happened: The check fails before this change and passes with it.

Hypothesis

A client can follow a successful chunk response and start a continuation that the registry rejects, even though Close already reported a storage failure. Response construction ignores that error and signs an offset based on accepted bytes rather than bytes the next request can resume. The failed-flush likelihood and customer impact are unmeasured; this does not claim loss of committed blobs or an observed production incident.

The documented chunked-upload contract says 202 accepts the chunk and Range reports current progress for the next chunk (docs/content/spec/api.md:617-662 and 4256-4275). blobWriter.Close forwards fileWriter.Close errors after a successful checkpoint. The filesystem writer counts buffered bytes in Size; its Close can fail in Flush before those bytes reach the file and then closes the descriptor. blobUploadResponse ignores Upload.Close's result at blobupload.go:353 and uses Size to produce the signed continuation state. A later ResumeBlobUpload compares that advertised offset with the newly opened writer's actual size and rejects a mismatch before copying the next request body.

The native route check on prop_w4jcx7fssw failed at f49f48736337e8fc9f7816a044819b0f7967d557: `go test -count=1 -v -run=^TestPerfloopPatchAcknowledgedProgress$ ./registry/handlers` (checkProperty call_kMUpqMTFKHhSSNQRDvahGE1B). With a 4096-byte filesystem write buffer, the one-byte fault arm printed closeFaults=1, ackStatus=202, ackOffset=1, storedOffset=0 and continuationStatus=416/RANGE_INVALID. The 4095-byte arm failed similarly. All three fault-free controls and the zero-byte fault control passed through continuation, Commit and read-back. The minimal nonempty deterministic replay is already in the stored test: POST, a one-byte PATCH with one response-time flush failure, then a PATCH following its returned location and range.

The fault seam models a Flush that writes zero pending bytes and releases its descriptor. It uses the driver writer's Cancel only to discard a wholly buffered, previously empty data file while preserving accepted Size; it does not invoke blob-upload Cancel. For the subsequent append-Resume size check, the missing empty file is equivalent to the empty file left by a zero-byte failed Flush. Payloads are bounded below the buffer size, requests are sequential, and hash-state writes and later operations are not faulted. This establishes a counterexample under that explicit driver-fault model, not a native disk-full experiment or an unbounded failure proof.

This is distinct from https://github.com/distribution/distribution/pull/5064: that patch releases the driver writer when saving a hash state fails, but still returns errors that the response caller ignores. Here checkpointing succeeds and the data flush fails. https://github.com/distribution/distribution/pull/5063 changes only Close after Commit. The admitted duplicate-Close repair hyp_f2njg0fq7y cannot fix an ignored first-close failure, and the buffered-size repair hyp_hhag229t1f concerns the in-memory backend rather than filesystem response gating. Focused prior searches for blobUploadResponse and upload Close-error handling found no matching repair. Open work touching the caller and Close was reviewed through the final page; the merged-path-history gap remains the shallow checkout and HTTP-403 history reads. CONTRIBUTING.md uses GitHub PR review.

The smallest useful Case check is the saved one-byte fault replay, paired with a direct filesystem Flush-error seam to validate the fault model. Confirm that the request either reports failure before acknowledging unusable progress or returns a continuation that actually resumes, while preserving fault-free PATCH/Commit/read-back and resource cleanup. Propagating the existing Close error may be a small source change, but it changes failed-upload responses; compatibility, documentation and the treatment of nonessential hash-checkpoint failures remain design questions. The test layout and fault-injection method are provisional, not additional frozen requirements. No production-only failure-rate gate or performance threshold is requested.

Change to test: Do not advertise chunk progress that the storage writer failed to make resumable; report the failure before 202 or recover a truthful continuation offset. A handler-side Close-error check is provisional, and the Case should account for the different consequences of digest-checkpoint errors and data-flush errors.

Where it lives

perfloop-oss/distribution

Evidence

The check fails before this change and passes with it: `Upload response construction propagates a BlobWriter.Close error before advertising continuation headers.`

Checks: 3 of 3 passed. Verification: no defect found.

Timeline