Replace nested map slice partitioning with a schema-compiled level-cursor decoder
perfloop/parquet-go · INEFFICIENT ALGORITHM
https://perfloop.ai/t/oss/case_q8p970nh3n
Verdict
VERIFIED · settled 2026-07-23 · merged as parquet-go/parquet-go#581
What happened: Validated: pooling the temporary descendant-span tables removes the measured allocation component without changing map/list level semantics. In the co-measured 32-entry map with two repeated descendants, allocations fell from 347 to 282 allocs/op (18.7%, p=0.0000108), which reaches the estimated 65-allocation span-table ceiling for this workload. B/op held at 20,409 to 18,655; ns/op held with no significant change (52,814 to 52,480). The focused reconstruction test, concurrent race check, and repository Go suite all passed.
Hypothesis
The source shows that Schema.Reconstruct first reindexes the flat, column-contiguous Row into a pooled [][]Value table (schema.go:388-407), after which the selected map path allocates another [][]Value, scans its first column to count entries, and, for every entry, re-expands, scans, and narrows every descendant column before recursing (row.go:764-816). The group closure then keeps passing capped subranges to its children (row.go:869-918) until the leaf selects column[0] and dispatches through Type.AssignValue to byteArrayType.AssignValue and Value.byteArray (row.go:926-934 and type_byte_array.go:57-80). A level-cursor plan changes that representation across the path: structural boundaries become persistent cursor state rather than repeatedly manufactured descendant spans, while the necessary map/slice construction and safe byte or string ownership conversion remain the architectural floor. An end-to-end Schema.Reconstruct sweep over map cardinality, descendant-leaf width, and nesting depth, with allocation counts, CPU profiles, and counters for level inspections/span partitions, would falsify this case if the present path does not show a material width-by-depth structural component or if the cursor plan does not remove that component after payload-copy and output-construction work are accounted for.
Change to test: Compile a reconstruction plan once per Schema that holds structural-node level rules, field setters, and per-leaf cursor metadata. Have Schema.Reconstruct initialize one cursor table over the flat Row and execute that plan: map and repeated nodes use governing repetition/definition-level cursors to emit entries or elements directly, groups address children through the compiled plan, and leaves consume their cursor value before retaining the existing Type.AssignValue semantics. This replaces recursive [][]Value subrange creation, cap-based re-expansion, and descendant-column repartitioning across the whole structural walk; retain the existing reflective closure path only as a compatibility fallback for shapes the plan cannot represent initially.
Where it lives
perfloop/parquet-go · row.go