Keep snapshot rows readable as table columns change

perfloop/influxdb · UNCATALOGUED MECHANISM

https://perfloop.ai/t/oss/case_vrqbebjh09

Verdict

VERIFIED · settled 2026-09-30

What happened: The assertion is violated on the comparison and satisfied with this change.

Hypothesis

A table scan can fail while an older batch is being persisted if the table gains a column. The buffer asks the frozen batch for every column in the newer schema and treats a missing one as an error. This can interrupt queries for otherwise valid rows; how often these events overlap has not been measured.

`TableBuffer::snapshot` freezes batches using the table definition at snapshot time (`table_buffer.rs:229-274,844-892`). The buffer releases its write guard before background parquet jobs finish and retains each snapshotting batch until its file is registered (`queryable_buffer.rs:236-303,356-406,428-445`). Writes can commit new fields or tags to the catalog (`validator.rs:115-123,353-372`), and a new SQL query gets its database schema from that catalog (`influxdb3_query_executor/src/lib.rs:159-170,469-500`). When a scan's time filter includes the old snapshot and its table definition contains a newly added column, `partitioned_record_batches` returns `FieldNotFound` for that batch (`table_buffer.rs:155-171`); `table_chunks_locked` propagates this as a query error. In contrast, the live-chunk `materialize` path fills absent columns with typed nulls (`table_buffer.rs:796-812`). This claim does not apply if the snapshot has already handed off to parquet or the filter prunes it.

The required property is that `QueryTable::scan` over a still-buffered snapshot and a newer table definition returns the old rows, with NULL for columns absent when those rows were frozen, while preserving old values and live-versus-snapshot ordering. A reproducible check is to gate a parquet persist job as in `queryable_buffer/tests.rs:665-743`, commit a new column while the earlier snapshot is queryable, and run a SQL scan covering both old and new timestamps before and after releasing the gate. The result should contain the same older rows, NULL in their new column, and no missing-field error; a scan that succeeds unchanged while the older snapshot is still present would refute the claimed failure.

Change to test: Align each still-buffered snapshot batch to the current table schema when serving a query: reuse its existing arrays and supply type-correct NULL arrays for columns added after the snapshot. Preserve the frozen batch used for persistence and the atomic handoff to its parquet file.

Where it lives

perfloop/influxdb · influxdb3_write/src/write_buffer/mod.rs

Evidence

The assertion is violated on the comparison and satisfied with this change: `With a scan covering a still-buffered snapshot and later live writes after an integer field and a tag are added, QueryTable::scan returns the same logical rows before and after parquet handoff, preserves old values, chooses the live overwrite, and supplies typed NULLs for both new columns on the untouched old row.`

Checks: 4 of 4 passed. Verification: no defect found.

Timeline