Prevent reconstruction from reading stale pooled columns after errors

perfloop-oss/parquet-go · Uncatalogued correctness

https://perfloop.ai/t/oss/case_gjj3ndc5q2

Verdict

VERIFIED · settled 2026-11-04 · merged as parquet-go/parquet-go#640

What happened: The check fails before this change and passes with it.

Hypothesis

Row reconstruction may reuse column slices left in a shared scratch buffer by an earlier failing map or repeated-field call. A later row with a missing column can then receive another call's data or appear to contain a required value. The submitter reports a regression from their PR #581; the reproductions, affected input shapes, incidence, and repair's allocation impact remain to be verified.

At the accepted revision eaeafbd572ef36d90eb98ad6ffa93d31a6ef80d3, valuesSliceBuffer.reserve returns v.values[:n] without clearing on its reuse branch (schema.go:409-420). Release only puts the buffer back, and acquisition resets its length without clearing its backing slots (schema.go:422-437; internal/memory/pool.go:11-25). Schema.Reconstruct calls reserve at the submitted locality, schema.go:392, and assigns columns only through the row's Range callbacks before invoking reconstruction (schema.go:388-402). The repeated and map reconstruction closures populate pooled slots, then release on error before their post-success slot advancement (row.go:669-717 and 767-841). A missing leaf would otherwise produce the explicit no-values error in reconstructFuncOfLeaf (row.go:932-941). The anchor is the submitted reserve definition, reached directly from the public Schema.Reconstruct operation, not either row.go helper.

The submitter reports an independent rerun in which TestVerifyTruncatedRowIsolation passes 6/6 subtests on base c215a4ab66 and fails 6/6 on merge 3e52054150. They report the same split for TestVerifyPooledErrorIsolation, persistence on upstream master 90bee42, and all six subtests passing after adding clear(columns) immediately after b.reserve in Schema.Reconstruct. Their reported consequences include Z:[]string{"rejected-private-data"} appearing in a later result and a missing required leaf incorrectly succeeding. These are attributed, unconfirmed reports, not results obtained in this run. The named tests and the specific wide-map reconstruction benchmark were not found in the accepted checkout.

The smallest useful next check is to recover or provisionally recreate the named sequential isolation tests and check the reported base/merge split and the accepted revision, observing both returned values and errors. The requested repository regression test must perform a failing map or repeated-field Reconstruct followed by a Reconstruct whose row lacks a column, assert that no values carry over, and cover the reported missing-required-leaf false success. Test layout and pool-reuse control are provisional implementation choices, not additional frozen requirements. The performance duty remains to retain #581's allocation gain and benchmark its wide-map reconstruction workload before and after the repair. The existing AddressBook BenchmarkReconstruct (row_test.go:690-719) is not evidence for that wide-map allocation result; the original fixture must be recovered or faithfully recreated. No elapsed-time gain, production exposure, or hardware-specific improvement is asserted.

Upstream https://github.com/parquet-go/parquet-go/pull/581 is merged at 3e520541506eeb62cde54891bf1b33b4b32d8856, based on c215a4ab66f89035709106b5bf7e7991ffcc9cff. Its available patch confirms replacing make with the shared pooled buffer in both reconstruction closures. Current draft https://github.com/parquet-go/parquet-go/pull/580 at head ff492dd48e5905a27b14a4d2913cdef9910e5c7d already adds the identical clear(columns) line in Schema.Reconstruct for its partial-column list-reading pipeline. This is direct implementation overlap, not upstream-clear or a merged repair. The reviewed maintainer comment asks for formatting and does not establish acceptance of this separate error-isolation outcome. The remaining contribution is the original map/repeated error-isolation regression, shared-reset coverage consideration, and allocation-retention check without requiring the draft's larger list fast path. Both open-PR pages were read; only the relevant schema.go and row.go patches of #580 were inspected, so its complete regression coverage remains unknown. https://github.com/parquet-go/parquet-go/pull/634 changes promoted-pointer field handling rather than pooled columns. The relevant #581/#580 bodies identify no linked GitHub issue; broader issue evidence remains unexamined.

Change to test: Clear reserved column slots before use, for example with clear(columns) after b.reserve in Schema.Reconstruct; consider clearing in valuesSliceBuffer.reserve or on release to cover the shared map and repeated paths too. Add the failing-map/repeated-then-missing-column regression test for no carryover or missing-required-leaf false success, and retain #581's allocation gain by benchmarking wide-map reconstruction before and after.

Where it lives

perfloop-oss/parquet-go · schema.go

Evidence

The check fails before this change and passes with it: `After JSON assignment errors in map, repeated-group, or list reconstruction, Schema.Reconstruct using pooled column scratch storage and a fresh destination must not read prior-call values for a missing final string column: a repeated field stays empty and a required leaf returns the no-values error.`

Wide map reconstruction · 60 sample pairs

metric baseline candidate paired median change confidence range required result
allocs/op 326 326 0 0 to 0 loss ≤ min(0, 0.05 × |baseline median|) PASSED
B/op 11938 11938 0 0 to 0 loss ≤ 0.05 × |baseline median| PASSED
ns/op 28836 28863 +151 −847 to +1411 loss ≤ 0.05 × |baseline median| PASSED

Checks: 3 of 3 passed. Verification: no defect found.

Timeline