Keep last-N CSV cells aligned with their field names

perfloop/victorialogs · IDENTITY COLLISION

https://perfloop.ai/t/oss/case_ezevym2jap

Verdict

VERIFIED · settled 2026-09-29

What happened: The assertion is violated on the comparison and satisfied with this change.

Hypothesis

A local last-N CSV query can place a timestamp under the message header. The optimization changes row-column order while the HTTP response retains the query's header order. How often callers request this field order is unmeasured.

At revision 6cdc3d6544cd8dc46891646d7ce9d865ce1a20dd, `ProcessQueryRequest` builds the CSV header from `GetFixedFields` and writes row values by column position with `appendCSVRow`. For single-node local storage, `format=csv`, and `query=* | sort by (_time desc) limit 5 | fields _msg, _time` without an HTTP `limit` argument, the last-N optimization is eligible. The trailing `fields` pipe leaves the header as `[_msg, _time]`. `getLogRowsFromDataBlock` forces `_time` ahead of `_msg`, and `runOptimizedLastNResultsQuery` uses that order to build the output block. On a successful nonempty result, the first CSV cell is therefore a timestamp under the `_msg` header. The two requested field layouts become the same time-first row layout, even though CSV positions identify different fields.

The focused `TestLastNCSVOrderProfile` check (receipt `call_Ahj12fMVkhIo4ANAz9m9Jgsx`) passed its assertions that the parsed query is eligible with limit 5, its fixed header is `[_msg, _time]`, and a ten-row two-field block is reconstructed as `[_time, _msg]`. Its 200,000 repeated in-memory conversions sampled 330 ms total CPU; this synthetic profile is not an HTTP-query cost or request-frequency measurement. It did not execute the HTTP endpoint, so end-to-end output remains the proof target. Confidence is medium.

A Case can ingest two rows with distinguishable messages and timestamps on a single local-storage node, request that exact query with `format=csv`, and verify each cell against its header and the ingested row. Preserve newest-first row order and the limit; use a pre-sort `fields _msg, _time` query and a time-first projection as ordering controls. A correctly aligned HTTP response under the stated conditions would falsify the proposed defect.

Change to test: Carry the original query's final fixed-field order into local last-N result assembly and emit columns in that order, keeping `_time` separately as the sort key. Preserve descending row order, offset and limit, and time-first output when the original query requires it.

Where it lives

perfloop/victorialogs · app/vlselect/logsql/logsql.go

Evidence

The assertion is violated on the comparison and satisfied with this change: `For a successful single-node local-storage POST to /select/logsql/query with format=csv, no HTTP limit, and query '* | sort by (_time desc) limit 5 | fields _msg, _time', seven rows with unique _msg and _time values yield a _msg,_time header and the five newest rows in descending _time order, with each cell equal to its named field.`

Checks: 6 of 6 passed. Verification: no defect found.

Timeline