Avoid copying serialized entries on HTTP cache hits
perfloop-oss/skipper · Avoidable Copy
https://perfloop.ai/t/oss/case_wa7jcm67v7
Verdict
VERIFIED · settled 2026-11-01 · merged as zalando/skipper#4396
What happened: The paired measurements met the required improvement.
Hypothesis
A resident HTTP cache entry is copied in full before it is decoded on every L1 read. This adds allocation and memory traffic proportional to the serialized response size, even for HEAD and conditional responses that return no body. The removable work is source-grounded; whole-handler savings and production exposure remain unmeasured. Confidence is medium.
Proxy.ServeHTTP calls Proxy.do, which runs cacheFilter.Request through applyFiltersToRequest. A normal warm read reaches LRUStorage.Get directly, or through L2Storage.Get's L1-first branch. LRUStorage.Get calls ShardedByteLRU.Get and then json.Unmarshal. On a byte-store hit, lruShard.get allocates len(item.data) bytes and copies them while holding the shard mutex. A miss does neither. A later decode error or hard-expiry rejection does not undo the copy. Vary metadata can add a separate small lookup; it is not another full-response copy in every request.
The broader option of retaining decoded Entry objects would remove decoding too, but would abandon the explicit raw-byte, pointer-light GC design. A narrower ownership-preserving route exists: LRUStorage.Set obtains private bytes from json.Marshal, replacement swaps the slice instead of modifying its contents, and deletion only drops the cache reference. An internal read-only snapshot can therefore remain alive while decoding outside the lock. This must not simply remove the defensive copy from the exported byte API, or expose shared mutable Payload or Header values through Storage.Get.
Current implementation comparisons with https://github.com/zalando/skipper/pull/4395 and https://github.com/zalando/skipper/pull/4369 show occupancy and lookup-metric changes, but retain the copying L1 read. https://github.com/zalando/skipper/pull/4394 changes dispatch, freshness and retention rather than serialized-byte ownership. Those actively owned fixes are not part of this proposal.
The smallest useful Case check is an allocation profile of complete ServeHTTP warm L1 hits, followed by a minimal internal-snapshot comparison at varied body sizes. Confirm or reject the claim using allocated bytes per completed request and copy-frame attribution, keeping decoding, routing, tracing and response-writing costs in the denominator. Preserve GET, HEAD, conditional 304 and Vary results; verify returned-value isolation and overwrite, delete, eviction and expiry behavior with concurrent readers under the race detector. The assigned earlier controlled 4 KiB observation nominates this frame, but is not a full-handler or production baseline.
Change to test: Let LRUStorage decode an internally owned, immutable byte snapshot without first cloning the serialized entry. Keep public ShardedByteLRU.Get results independently mutable, preserve byte-based eviction and the pointer-light stored representation, and release the shard mutex before decoding.
Where it lives
perfloop-oss/skipper · proxy/proxy.go
Evidence
warm L1 HTTP cache hit (4 KiB) · 9 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
B/op |
30498 |
24337 |
−20.2% (−6158) |
−6172 to −6155 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
ns/op |
39701 |
36592 |
−3117 |
−4002 to +384 |
loss ≤ 0.05 × |baseline median| |
PASSED |
warm L1 HTTP cache hit (64 KiB) · 11 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
B/op |
237772 |
147447 |
−38% (−90312) |
−90382 to −90287 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
ns/op |
256746 |
219130 |
−15.2% (−38957) |
−84923 to −21412 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Checks: 5 of 5 passed. Verification: no defect found.