Avoid full-size buffers on cold opens of small blobs
perfloop-oss/distribution · Allocation In Hot Loop
https://perfloop.ai/t/oss/case_p6h3vmsws2
Verdict
VERIFIED · settled 2026-11-10 · merged as distribution/distribution#5070
What happened: The paired measurements met the required improvement.
Hypothesis
A cold, directly streamed small-blob download allocates a 4 MiB reader buffer regardless of its known length. Size-aware buffering could reduce cold-open allocation and the capacity held by each active small stream. The production size mix, concurrency, prevalence and end-to-end effect remain unmeasured.
The assigned GetBlob path reaches blobServer.ServeBlob, which passes desc.Size to newFileReader after the redirect branch falls through. fileReader.reader allocates through bufio.NewReaderSize(fr.rc, fileReaderBufferSize) when a driver open succeeds and fr.brd is nil. The constant is 4,194,304 bytes. Subsequent seeks reuse that buffer with Reset; this is not another 4 MiB allocation per seek. Failed opens, missing-path retries, redirects and requests completed without a Read do not pay this successful-open buffer cost.
Measurement receipt call_DrtItElf7s4p23geECUBzfgw, output:1, compares the base with two temporary policies using the repository's Go tests and benchmarks. The command was go test -count=1 -run='^(TestPerfloopColdOpenProbe|TestSimpleRead|TestFileReaderSeek|TestFileReaderNonExistentFile)$' -bench='^BenchmarkPerfloop(ColdOpenProbe|BlobStreamProbe)$' -benchtime=3x -benchmem ./registry/storage. For its constructed 4 KiB direct HTTP response from an in-process driver, three measured iterations printed 4,199,658 B/op on the base and 9,170 B/op with capacity capped by total blob size. The cold-open probe also exercised a first read and three seek/read cycles at six sizes from 1 byte through 4 MiB plus one byte. These are local mechanism samples, not a production workload distribution or a stable latency estimate.
The broader comparator removed wrapper buffering altogether. It passed those focused controls, but the constructed 16 MiB HTTP response used 512 driver Read calls instead of four with either fixed or capped buffering. Those are stream-method calls, not remote request counts. A conservative size cap leaves large-stream coalescing intact rather than bundling a driver-dependent buffering removal.
Prior hyp_ycez521tc2 pools fixed 4 MiB readers between streams. It does not reduce a cold buffer's capacity or the capacity checked out for concurrent small streams. This proposal concerns that remaining sizing delta, not another pool. The inspected implementation of https://github.com/distribution/distribution/pull/5052 bypasses fileReader for eligible filesystem downloads; it leaves non-filesystem and wrapped-reader fallback buffering unchanged. Exclude that sendfile fast path and opt-in range passthrough from the benefiting operation.
Customer feedback is applied with its original scope. user:4 closed hyp_kzzsk0j32b because redundant GetBlob Stat removal saved only with no descriptor cache, whereas shipped configurations enable one; no Stat removal is proposed here. The same actor closed hyp_zgfbf47kq7 because PR #5052 already owns sendfile, which is excluded here. Their November 6 GTM closures of upload hypotheses hyp_6z8h7v9dgk and hyp_wzgesn443a lower repository priority, but do not measure or refute this sizing mechanism.
CONTRIBUTING.md names GitHub pull requests as the code-review venue and support Slack separately. All open-PR pages were followed, including drafts; the relevant sendfile implementation was read. Recent merged owner history is an evidence gap: this checkout is shallow and bounded requests for the two files' host commit histories were refused with HTTP 403. No maintainer acceptance of this sizing change is assumed.
The smallest useful Case check is a cold direct-download allocation comparison across small-blob sizes, with a larger-stream control and no assumed minimum percentage gain. Confirm the remaining capacity reduction against fixed-size reuse, and preserve full/range bytes, seek offsets, EOF, missing-path retries and close behavior. The local test layout and sampled sizes above are provisional proof methods, not extra frozen requirements. Backend throughput and real traffic frequency remain separate unknowns.
Change to test: Make first-open buffering respect the blob's known size, subject to the buffering library's minimum, while retaining the 4 MiB ceiling and reuse across seeks. Keep this internal, without a new driver or configuration contract; the exact sizing implementation is provisional.
Where it lives
perfloop-oss/distribution · registry/handlers/blob.go
Evidence
Cold file reader · 9 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
B/op |
4195868 |
4943 |
−99.9% (−4190925) |
−4191308 to −4190863 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
ns/op |
687084 |
1943 |
−99.7% (−685345) |
−927405 to −348475 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
14 |
12 |
−14.3% (−2) |
−3 to −2 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Small blob download · 11 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
B/op |
4200063 |
5570 |
−99.9% (−4194493) |
−4194585 to −4194366 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
ns/op |
801030 |
14092 |
−98.3% (−787074) |
−994667 to −425556 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
64 |
61 |
−4.7% (−3) |
−3 to −3 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Large blob download · 121 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
B/op |
37753381 |
37753388 |
+9 |
−49 to +108 |
loss ≤ 0.05 × |baseline median| |
PASSED |
ns/op |
12004541 |
11982030 |
−112745 |
−516350 to +549694 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
74 |
74 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Checks: 5 of 5 passed. Verification: no defect found.