Filter deleted and expired block headers before loading their payloads

perfloop/victoriametrics · LATE FILTERING

https://perfloop.ai/t/oss/case_dgmfa60mz8

Verdict

VERIFIED · settled 2026-09-26

What happened: The paired measurements met the required improvement.

Hypothesis

`mergeBlockStreamsInternal` discards a block when `dmis.Has(b.bh.TSID.MetricID)` or when `b.bh.MaxTimestamp` precedes its retention deadline. Those tests happen after `blockStreamMerger.NextBlock` calls `blockStreamReader.NextBlock`, whose `readBlock` has already parsed the header and populated timestamp and value payload buffers. The discarded block only adds its header row count to `localRowsDeleted`; it never reaches `WriteExternalBlock`.

Header-first filtering leaves the retained-block merge path unchanged. Its floor remains index-header reads plus payload reads, deduplication, and writes for live blocks. The removable term is the compressed payload read and buffer materialization for blocks that the existing header predicates discard. The skip path must advance both payload offsets correctly, including adjacent blocks that reuse a timestamp payload.

A later case should run file-backed historical forced merges with controlled fractions of deleted metric IDs and fully expired blocks, compare filestream read bytes and calls, CPU, and elapsed merge time, and verify output headers, rows, `rowsDeleted`, and forced-stop behavior against baseline. If discarded-block share does not reduce payload-read counters or merge cost, that measurement falsifies the move.

Change to test: Split `blockStreamReader.NextBlock` into header advancement and lazy payload loading. Let `blockStreamMerger` reject a header when its MetricID is in `dmis` or its `MaxTimestamp` is below `getRetentionDeadline` before materializing `timestampsData` and `valuesData`; add an offset-aware skip path for file and in-memory readers that preserves shared-timestamp references, header validation, heap order, `rowsDeleted` accounting, and per-block cancellation checks.

Where it lives

perfloop/victoriametrics · lib/storage/table.go

Evidence

File-backed historical forced merge through partition.ForceMergeAllParts: one pre-dedup part with 64 series and 16,777,216 rows total (16 deleted-ID series, 16 fully expired series, 32 retained series; 262,144 rows each), with a final 2-minute dedup interval. Each benchmark iteration performs one merge. · 10 sample pairs

metric baseline candidate paired median change confidence range required result
wall time (ns/op) 102645061 99362385 −3.2% (−3244432) −7475128 to +38709 ≤ 5132253 PASSED
process CPU time (ns/op) 94000000 91000000 −2.7% (−2500000) −9000000 to +4000000 ≤ 4700000 PASSED
bytes requested through read(2) (rchar delta from /proc/self/io) 25804440 12922594 −49.9% (−12881846) −12881846 to −12881846 < −1290222 PASSED
read(2) calls (syscr delta) 406 209 −48.5% (−197) −197 to −197 ≤ 20.3 PASSED

Retained-only two-source best-case block-stream merge: two disjoint maximum-size blocks and no discarded blocks. · 10 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 6948 6958 −3 −18 to +70 ≤ 347.4 PASSED
MB/s 2358 2355 +0.1% (+1.3) −24.17 to +6.1 ≥ −117.9 PASSED
B/op 1648 1648 0 0 to 0 ≤ 82.4 PASSED
allocs/op 23 23 0 0 to 0 ≤ 1.15 PASSED

Retained-only two-source worst-case block-stream merge with overlapping timestamp ranges and no discarded blocks. · 10 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 1286204 1295822 +0.6% (+7788) −6074 to +21488 ≤ 64310 PASSED
MB/s 6.365 6.32 −0.6% (−0.04) −0.1 to +0.03 ≥ −0.3183 PASSED
B/op 1670 1670 0 0 to +1 ≤ 83.5 PASSED
allocs/op 23 23 0 0 to 0 ≤ 1.15 PASSED

Checks: 7 of 7 passed. Verification: no defect found.

Timeline