Replace per-message MaxAge deletes with consumer-aware block-prefix expiry ranges
perfloop/nats-server · UNDER BATCHING
https://perfloop.ai/t/oss/case_yng2yjnryc
Verdict
VERIFIED · settled 2026-08-12
What happened: The paired measurements met the required improvement.
Hypothesis
The ordinary MaxAge branch in server.fileStore.expireMsgs walks LoadNextMsg and, for every expired message, reacquires fs.mu before entering removeMsgViaLimits, removeMsg, and server.fileStore.removeMsgFromBlock. That leaf can load a block cache, look up and copy the message, mutate message/byte/per-subject and first-or-delete-map state, then emit a single-message storage update; stream.storeUpdates matches consumers by subject and decStreamPending may start an individual termination path for each outstanding delivery. The source already exposes the cheaper storage geometry that this live path misses: expireMsgsOnRecover tests mb.last.ts and removes wholly old blocks, and the public compact/purge paths plus stream.purge and consumer.purge show that sequence-floor cleanup can be handled as a range. This must be a dedicated via-limits range primitive rather than a blind call to public Compact, since removeMsgViaLimits intentionally avoids an index-state write that normal compaction forces. Once the transaction lands, full eligible blocks no longer require one traversal of this leaf per expired record; the remaining necessary work is the boundary block and genuinely pending consumer records or per-message advisories. A later case should falsify the hypothesis with timestamp-ordered, no-SDM MaxAge backlogs spanning many blocks at R1 and R3 while varying unacknowledged density, measuring end-to-end expiry CPU and tail time, removeMsgFromBlock/cache-load counts, range versus per-message replication and consumer updates, and FirstSeq, subject-total, pending/redelivery, and advisory correctness; reject it if eligible full blocks still produce one leaf deletion per message or no tail improvement after the required pending cleanup is included.
Change to test: Add an internal ExpirePrefixViaLimits(first, cutoff) operation and range callback from the store timer to stream processing. For a verified contiguous MaxAge prefix, expireMsgs should use block time metadata to emit one range instead of calling removeMsgViaLimits for every sequence; the file-store implementation should unlink wholly expired msgBlocks, subtract per-subject totals from each block's aggregate fss state, and scan only the boundary block while retaining the recoverable via-limits/no-forced-index-write contract. The stream should coordinate that range once (as one internal replicated range operation for clustered streams) and apply a range-aware consumer invalidation/floor that visits only affected pending and redelivery state needed for termination semantics, rather than routing every stored message through decStreamPending. Keep the existing single-message route for SubjectDeleteMarkerTTL, negative-TTL survivors or other non-contiguous spans, and unordered hash-wheel TTL expiry.
Where it lives
perfloop/nats-server · server/filestore.go
Evidence
cold restarted v4-index file-store MaxAge expiry of 16,384 timestamp-ordered messages across complete blocks · 10 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
13898560 |
4657337 |
−66.7% (−9274083) |
−9844329 to −8613241 |
< 0 |
PASSED |
end-to-end file-store MaxAge expiry of 16,384 messages with 128 pending and redelivered consumer records · 10 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
17978209 |
4165790 |
−75.9% (−13647498) |
−14578089 to −13226715 |
< 0 |
PASSED |
p95-ns/op |
21244853 |
4996952 |
−78.2% (−16612541) |
−18698718 to −14281559 |
< 0 |
PASSED |
end-to-end file-store MaxAge expiry of 16,384 messages with 4,096 pending and redelivered consumer records · 10 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
32100486 |
17394732 |
−45.2% (−14514084) |
−15635062 to −13027041 |
< 0 |
PASSED |
p95-ns/op |
36903397 |
20731092 |
−41.9% (−15448286) |
−21267192 to −13941427 |
< 0 |
PASSED |
Checks: 5 of 5 passed. Verification: no defect found.