Preserve null placement when sorting descending optional buffer keys

perfloop-oss/parquet-go · Uncatalogued correctness

https://perfloop.ai/t/oss/case_vvr5q3jr7w

Verdict

VERIFIED · settled 2026-11-17 · merged as parquet-go/parquet-go#683

What happened: The check fails before this change and passes with it.

Hypothesis

Descending buffer sorting reverses null placement as well as value order. A two-row public-API check leaves the null last even when NullsFirst is requested. Sorted rows can therefore disagree with the buffer's declared sorting columns. Downstream impact and occurrence in applications are unmeasured.

At revision 90bee42a64b670cb55f054a8b26e72381b825611, GenericBuffer.Less delegates to Buffer.Less. Buffer.configure selects nullsGoFirst or nullsGoLast at buffer.go:243-270, then wraps the optional comparator in reversedColumnBuffer at lines 274-278. The adapter swaps the complete comparison's operands. optionalColumnBuffer.Less at column_buffer_optional.go:144-155 applies null placement inside that reversal. SortingColumn.NullsFirst and NullsFirst explicitly promise nulls at the beginning in row_group.go:89-103.

The provisional native checker in hunt_null_placement_test.go searched inputs by increasing length, targeting all lengths 0..3 over {null, 0, min-int64, -1, 1, max-int64} under four direction/placement combinations. `go test -run '^TestHuntNullPlacementSearch$' -count=1 -v .` stopped at case 35 of the planned 1,036 cases: desc=true, nullsFirst=true, input=[0 null], output=[0 null]. It did not complete that search. The reduced deterministic test, `go test -run '^TestHuntDescendingNullPlacementReplay$' -count=1 -v .`, failed with `got=[0 null] want=[null 0]`. It uses exactly one Write, sort.Sort, and Read on GenericBuffer with a *int64 key. Both runs exited 1 because the ordering assertion failed, not because of a fixture or build failure.

The independent proof target is null placement independent of direction for optional keys, with descending non-null values and ascending, equal-key, and all-null controls. The checker layout and finite alphabet are provisional proof methods, not an additional frozen domain requirement. CONTRIBUTING.md:50 requires bug fixes to have tests. The current upstream PR index, https://github.com/parquet-go/parquet-go/pulls (observed 2026-11-17), showed no matching comparator change, and concrete prior-repair searches returned no match. The checkout is shallow and bounded upstream file-history requests were refused, so recent merged-history coverage remains a gap; no alternate review venue is named in CONTRIBUTING.md.

Change to test: Make descending optional-key sorting preserve the requested null placement while reversing non-null values. Cover the shared GenericBuffer and Buffer behavior with native regression tests, retaining ascending order and equal/all-null behavior.

Where it lives

perfloop-oss/parquet-go · buffer.go

Evidence

The check fails before this change and passes with it: `For optional int64 keys written through Buffer.WriteRows or GenericBuffer.Write, sorting places nulls according to NullsFirst independently of ascending or descending value order; equal-key, all-null, and empty inputs retain their values.`

The check fails before this change and passes with it: `For optional int64 keys in Buffer and GenericBuffer, two successive sort/read cycles with NullsFirst return [null,1,0] for Descending on [null,0,1] and [null,0,1] for Ascending on [null,1,0], without panicking.`

Checks: 6 of 6 passed. Verification: no defect found.

Timeline