Merge sorted raw-row runs before serializing an in-memory part

perfloop/victoriametrics · REDUNDANT SERIALIZATION

https://perfloop.ai/t/oss/case_545z63bb2p

Verdict

OPEN · opened 2026-08-26

Hypothesis

`flushRowssToInmemoryParts` creates one `inmemoryPart` per non-empty raw slice and then repeatedly invokes `mustMergeInmemoryParts`. `InitFromRows` sorts and `rawRowsMarshaler` writes first-level blocks at compression level -5; its source comment says those blocks will be re-compressed during subsequent merges. `mustMergeInmemoryPartsFinal` then opens a `blockStreamReader` for every fresh part before it reaches `mergeBlockStreams`, so a multi-slice flush materializes timestamps, values, index, and metaindex data solely to consume it as merge input.

Keep the parallel sort and the current initial block formation, but carry those blocks as bounded in-memory runs until the fan-in merge writes the final part. The remaining required work is ordering, retention and deleted-series handling, deduplication, and one final serialization. The fresh input part writer, close, metadata creation, reader, and decode path no longer sit between the two stages.

A case should benchmark end-to-end pending-row flushes with 2, 15, and more than 15 8 MiB runs, with both shared-series and disjoint-series distributions. CPU, allocation, and peak-live-heap profiles must show whether the removed marshal/read path is material; reject the change if the raw-run heap or retained blocks erase that benefit. Differential output tests must cover block boundaries, mixed PrecisionBits, equal timestamps with deduplication, deleted metric IDs, and retention trimming.

Change to test: Refactor the multi-slice branch of `flushRowssToInmemoryParts` to sort each `[]rawRow` in parallel, retain each run's quantized and first-pass-deduplicated `Block`s in bounded run cursors, and k-way merge those cursors directly into destination `blockStreamWriter` output parts. Split output at the existing in-memory-size boundary and keep the serialized `inmemoryPart`/`blockStreamReader` path for normal compactions. Preserve source block boundaries, scale and PrecisionBits handling, deleted-MetricID and retention filtering, deduplication, counters, and flush deadlines.

Where it lives

perfloop/victoriametrics · lib/storage/partition.go

Evidence

No usable result yet.

Timeline