Reuse reencode value buffer across columns

perfloop/parquet-go · Allocation In Hot Loop

https://perfloop.ai/t/oss/case_s4m829jd2j

Verdict

VERIFIED · settled 2026-10-28 · merged as parquet-go/parquet-go#632

What happened: The paired measurements met the required improvement.

Hypothesis

copyColumnValues constructs make([]Value, reencodeValueBufferSize) at line 197 on every invocation, while Writer.writeRowGroupByColumn calls it once per source column in its serial loop. Thus the removable allocation delta is N-1 buffers for a row group with N columns, at the real cadence of every re-encoded row group; ReadValues still operates in the same 1024-value batches. I ran `go test -run '^$' -gcflags='-m=2' .` and filtered the diagnostics for this call site. The compiler reported that `make([]Value, 1024)` at writer_reencode.go:197 escapes to the heap, including the flow through ReadValues/WriteRowValues. That confirms the allocation shape, not a measured CPU or GC impact. The proposed sequential reuse is a hypothesis that must preserve any value-lifetime requirements. A case session should benchmark Writer.WriteRowGroup through the column re-encode route with repeated multi-column row groups, compare correctness plus `-benchmem` and allocation/CPU profiles, and require allocations/op to fall by roughly N-1 per row group before attributing any latency or GC improvement.

Change to test: Allocate one reencodeValueBufferSize []Value scratch buffer in Writer.writeRowGroupByColumn before its serial column loop, pass that buffer into a buffer-taking copyColumnValues helper, and reuse it only after each synchronous column copy returns. Preserve the existing ReadValues batch size, error behavior, and reader.Close lifetime; do not share the scratch slice across concurrent Writer calls.

Where it lives

perfloop/parquet-go · writer.go

Evidence

steady-state 209-column one-row L3 row-group re-encode · 9 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 1224716 517314 −58% (−709987) −752604 to −673677 loss ≤ min(0, 0.05 × |baseline median|) PASSED
B/op 5934925 260066 −95.6% (−5674883) −5675425 to −5674643 gain > 5000000; gain > 0.05 × |baseline median|; arm medians separated PASSED
allocs/op 3360 3145 −6.4% (−215) −217 to −215 gain > 200; gain > 0.05 × |baseline median|; arm medians separated PASSED

four 1k-row 24-column file-backed re-encode groups, uncompressed output · 11 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 2555428 2188995 −15.1% (−386549) −499643 to −256971 loss ≤ 0.05 × |baseline median| PASSED
B/op 3070707 479365 −84.4% (−2590504) −2603021 to −2577872 gain > 0; gain > 0.05 × |baseline median|; arm medians separated PASSED
allocs/op 2895 2782 −3.9% (−113) −114 to −112 loss ≤ 0.05 × |baseline median| PASSED

four 25k-row 24-column file-backed re-encode groups, uncompressed output · 12 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 39349907 38959370 −660859 −3147029 to +303789 loss ≤ 0.05 × |baseline median| PASSED
B/op 3145937 578513 −81.6% (−2567774) −2634679 to −2519042 gain > 0; gain > 0.05 × |baseline median|; arm medians separated PASSED
allocs/op 2893 2791 −3.6% (−103.5) −106 to −97 loss ≤ 0.05 × |baseline median| PASSED

existing 24-column L3 benchmark: four 25k-row groups, Snappy input and Zstd output · 13 sample pairs

metric baseline candidate paired median change confidence range required result
allocs/op 2946 2843 −3.5% (−102) −132 to −80 loss ≤ 0.05 × |baseline median| PASSED

four 100k-row 24-column file-backed re-encode groups, uncompressed output · 13 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 156870750 153795422 −2589362 −9546003 to +1206745 loss ≤ min(0.05, 0.03) × |baseline median| PASSED

four 1M-row 24-column file-backed re-encode groups, uncompressed output · 14 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 1618599724 1608079798 −7053713 −94517897 to +21519890 loss ≤ min(0.05, 0.03) × |baseline median| PASSED

Checks: 3 of 3 passed. Verification: no defect found.

Timeline