Reuse reencode value buffer across columns
perfloop/parquet-go · Allocation In Hot Loop
https://perfloop.ai/t/oss/case_s4m829jd2j
Verdict
VERIFIED · settled 2026-10-28 · merged as parquet-go/parquet-go#632
What happened: The paired measurements met the required improvement.
Hypothesis
copyColumnValues constructs make([]Value, reencodeValueBufferSize) at line 197 on every invocation, while Writer.writeRowGroupByColumn calls it once per source column in its serial loop. Thus the removable allocation delta is N-1 buffers for a row group with N columns, at the real cadence of every re-encoded row group; ReadValues still operates in the same 1024-value batches. I ran `go test -run '^$' -gcflags='-m=2' .` and filtered the diagnostics for this call site. The compiler reported that `make([]Value, 1024)` at writer_reencode.go:197 escapes to the heap, including the flow through ReadValues/WriteRowValues. That confirms the allocation shape, not a measured CPU or GC impact. The proposed sequential reuse is a hypothesis that must preserve any value-lifetime requirements. A case session should benchmark Writer.WriteRowGroup through the column re-encode route with repeated multi-column row groups, compare correctness plus `-benchmem` and allocation/CPU profiles, and require allocations/op to fall by roughly N-1 per row group before attributing any latency or GC improvement.
Change to test: Allocate one reencodeValueBufferSize []Value scratch buffer in Writer.writeRowGroupByColumn before its serial column loop, pass that buffer into a buffer-taking copyColumnValues helper, and reuse it only after each synchronous column copy returns. Preserve the existing ReadValues batch size, error behavior, and reader.Close lifetime; do not share the scratch slice across concurrent Writer calls.
Where it lives
perfloop/parquet-go · writer.go
Evidence
steady-state 209-column one-row L3 row-group re-encode · 9 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
1224716 |
517314 |
−58% (−709987) |
−752604 to −673677 |
loss ≤ min(0, 0.05 × |baseline median|) |
PASSED |
B/op |
5934925 |
260066 |
−95.6% (−5674883) |
−5675425 to −5674643 |
gain > 5000000; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
allocs/op |
3360 |
3145 |
−6.4% (−215) |
−217 to −215 |
gain > 200; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
four 1k-row 24-column file-backed re-encode groups, uncompressed output · 11 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
2555428 |
2188995 |
−15.1% (−386549) |
−499643 to −256971 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
3070707 |
479365 |
−84.4% (−2590504) |
−2603021 to −2577872 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
allocs/op |
2895 |
2782 |
−3.9% (−113) |
−114 to −112 |
loss ≤ 0.05 × |baseline median| |
PASSED |
four 25k-row 24-column file-backed re-encode groups, uncompressed output · 12 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
39349907 |
38959370 |
−660859 |
−3147029 to +303789 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
3145937 |
578513 |
−81.6% (−2567774) |
−2634679 to −2519042 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
allocs/op |
2893 |
2791 |
−3.6% (−103.5) |
−106 to −97 |
loss ≤ 0.05 × |baseline median| |
PASSED |
existing 24-column L3 benchmark: four 25k-row groups, Snappy input and Zstd output · 13 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
allocs/op |
2946 |
2843 |
−3.5% (−102) |
−132 to −80 |
loss ≤ 0.05 × |baseline median| |
PASSED |
four 100k-row 24-column file-backed re-encode groups, uncompressed output · 13 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
156870750 |
153795422 |
−2589362 |
−9546003 to +1206745 |
loss ≤ min(0.05, 0.03) × |baseline median| |
PASSED |
four 1M-row 24-column file-backed re-encode groups, uncompressed output · 14 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
1618599724 |
1608079798 |
−7053713 |
−94517897 to +21519890 |
loss ≤ min(0.05, 0.03) × |baseline median| |
PASSED |
Checks: 3 of 3 passed. Verification: no defect found.