Avoid copying serialized entries on HTTP cache hits

perfloop-oss/skipper · Avoidable Copy

https://perfloop.ai/t/oss/case_wa7jcm67v7

Verdict

VERIFIED · settled 2026-11-01 · merged as zalando/skipper#4396

What happened: The paired measurements met the required improvement.

Hypothesis

A resident HTTP cache entry is copied in full before it is decoded on every L1 read. This adds allocation and memory traffic proportional to the serialized response size, even for HEAD and conditional responses that return no body. The removable work is source-grounded; whole-handler savings and production exposure remain unmeasured. Confidence is medium.

Proxy.ServeHTTP calls Proxy.do, which runs cacheFilter.Request through applyFiltersToRequest. A normal warm read reaches LRUStorage.Get directly, or through L2Storage.Get's L1-first branch. LRUStorage.Get calls ShardedByteLRU.Get and then json.Unmarshal. On a byte-store hit, lruShard.get allocates len(item.data) bytes and copies them while holding the shard mutex. A miss does neither. A later decode error or hard-expiry rejection does not undo the copy. Vary metadata can add a separate small lookup; it is not another full-response copy in every request.

The broader option of retaining decoded Entry objects would remove decoding too, but would abandon the explicit raw-byte, pointer-light GC design. A narrower ownership-preserving route exists: LRUStorage.Set obtains private bytes from json.Marshal, replacement swaps the slice instead of modifying its contents, and deletion only drops the cache reference. An internal read-only snapshot can therefore remain alive while decoding outside the lock. This must not simply remove the defensive copy from the exported byte API, or expose shared mutable Payload or Header values through Storage.Get.

Current implementation comparisons with https://github.com/zalando/skipper/pull/4395 and https://github.com/zalando/skipper/pull/4369 show occupancy and lookup-metric changes, but retain the copying L1 read. https://github.com/zalando/skipper/pull/4394 changes dispatch, freshness and retention rather than serialized-byte ownership. Those actively owned fixes are not part of this proposal.

The smallest useful Case check is an allocation profile of complete ServeHTTP warm L1 hits, followed by a minimal internal-snapshot comparison at varied body sizes. Confirm or reject the claim using allocated bytes per completed request and copy-frame attribution, keeping decoding, routing, tracing and response-writing costs in the denominator. Preserve GET, HEAD, conditional 304 and Vary results; verify returned-value isolation and overwrite, delete, eviction and expiry behavior with concurrent readers under the race detector. The assigned earlier controlled 4 KiB observation nominates this frame, but is not a full-handler or production baseline.

Change to test: Let LRUStorage decode an internally owned, immutable byte snapshot without first cloning the serialized entry. Keep public ShardedByteLRU.Get results independently mutable, preserve byte-based eviction and the pointer-light stored representation, and release the shard mutex before decoding.

Where it lives

perfloop-oss/skipper · proxy/proxy.go

Evidence

warm L1 HTTP cache hit (4 KiB) · 9 sample pairs

metric baseline candidate paired median change confidence range required result
B/op 30498 24337 −20.2% (−6158) −6172 to −6155 gain > 0; gain > 0.05 × |baseline median|; arm medians separated PASSED
ns/op 39701 36592 −3117 −4002 to +384 loss ≤ 0.05 × |baseline median| PASSED

warm L1 HTTP cache hit (64 KiB) · 11 sample pairs

metric baseline candidate paired median change confidence range required result
B/op 237772 147447 −38% (−90312) −90382 to −90287 gain > 0; gain > 0.05 × |baseline median|; arm medians separated PASSED
ns/op 256746 219130 −15.2% (−38957) −84923 to −21412 loss ≤ 0.05 × |baseline median| PASSED

Checks: 5 of 5 passed. Verification: no defect found.

Timeline