Failed SSTable reads must not count as successful partition scans

perfloop/cass · IGNORED FAILURE

https://perfloop.ai/t/oss/case_7mshe0tkx6

Verdict

VERIFIED · settled 2026-10-01 · pull request opened as mweiden/cass#122

What happened: The assertion is violated on the comparison and satisfied with this change.

Hypothesis

A partial-key SQL read can appear to succeed while omitting persisted rows if a table file cannot be read. The scan drops that failure and returns the rows it managed to collect. How often storage reads fail is unknown.

At revision 6af3f8f, the documented `CassService::query` operation `SELECT * FROM orders WHERE customer_id = 'nike'` has a composite primary key. When the schema is present and only the partition key is specified, `SqlEngine::exec_select_schema` calls `Database::scan_ns` rather than `get_ns`. In `scan_ns`, `if let Ok(raw) = self.storage.get(&table.path).await` skips an SSTable on error, then returns a normal `Vec` of rows, possibly exposing an older version from another table. `Cluster::run_read_with_quorum` counts an `Ok` result as a success, including this incomplete scan.

The required property is that a replica's partial-key SELECT under an SSTable read error must not present an incomplete scan as a successful result or quorum vote. This does not imply the entire client request must fail if enough other replicas return complete scans. A Case can create and flush a composite-key table on a local-storage server, keep its schema readable, inject a failed read for a data SSTable, then issue the documented gRPC Query with read consistency ONE. An empty or stale successful response confirms the risk; after the change, that replica must report a read error. Repeat with sufficient intact replicas to check that quorum succeeds without treating the failed scan as an acknowledgement. The current frequency of this fault and its effect in any deployed cluster are unmeasured.

Change to test: Propagate SSTable read failures through partition scanning and schema-aware SELECT so incomplete scans are not successful replica votes. Preserve successful scans' ordering, tombstones, and newer-value precedence, and allow the coordinator to use other complete replica results when the configured quorum is met.

Where it lives

perfloop/cass · src/main.rs

Evidence

The assertion is violated on the comparison and satisfied with this change: `For an RF=1/ONE gRPC partial-key SELECT with readable schema, a failed persisted SSTable read must make the replica return an error, not successful empty or partial rows.`

The assertion is violated on the comparison and satisfied with this change: `For RF=3 QUORUM partial-key SELECT, only complete scans count toward two reads: two SSTable scan errors fail, while one error plus two complete replicas returns the complete newest rows.`

The assertion is violated on the comparison and satisfied with this change: `For an RF=3 QUORUM gRPC partial-key SELECT with a readable schema and no matching rows, one failed SSTable scan plus two complete empty scans must return empty rows, not an I/O error.`

The assertion is violated on the comparison and satisfied with this change: `For an RF=1 gRPC SHOW TABLES query with a persisted orders registry, a failed read of the only replica's newer SSTable must return a storage error, not a successful empty table list.`

Checks: 13 of 13 passed. Verification: no defect found.

Timeline