Startup retry deletes durable segments after replay refuses an occupied ID

perfloop-oss/weaviate · Identity Collision

https://perfloop.ai/t/oss/case_dq9amkedw4

Verdict

VERIFIED · settled 2026-11-17

What happened: The check fails before this change and passes with it.

Hypothesis

Retrying a refused WAL replay can discard an unrelated committed segment. The commit correctly refuses an occupied successor, but startup classifies that incumbent as another provisional replay chunk. A deterministic native test reproduces the loss; the prevalence of this state in deployed buckets is unknown.

At revision 36357612175f8c6977e27cd99fc8c69feec29938, recoveredRun.commit checks each final path immediately before its rename (bucket_recover_from_wal.go:359–388). Its comment explicitly supports an occupied ID after a backward clock step. Chunk 0 can already be durable when chunk 1 is refused. writeOutReplay propagates the error and retains the source WAL, but discards only staged spellings. On retry, chunkedSegmentIDs includes both the renamed prefix and the contiguous incumbent. removeSegmentsOfSurvivingWALs deletes that entire list before mounting segments. Adjacency does not establish shared replay ownership. TestRecoveredRunCommit already requires the incumbent's bytes to survive the refusal, but does not check the next startup.

The new file adapters/repos/db/lsmkv/hunt_replay_occupied_test.go contained TestHuntReplayOccupiedSuccessorSurvivesRetry. Its bounds were two valid replace-WAL records with 64 payload bytes each, one separately flushed durable segment at WAL ID+1, one refused startup, and one retry. It used the existing bucket fixture, public Put and FlushAndSwitch operations, and real NewBucket startup. The exact repository-root command was: go test ./adapters/repos/db/lsmkv -run '^TestHuntReplayOccupiedSuccessorSurvivesRetry$' -count=1 -v -timeout=90s. It exited 1. Deciding output: 'first open refused occupied successor after chunk 0; incumbent and WAL are unchanged'; 'retry succeeded; WAL records recovered; successor bytes preserved=false'; 'actual : []byte(nil)'; 'startup retry must preserve unrelated committed data after an occupied-ID refusal'. The expected value was the separately committed durable-value. Both WAL records were checked successfully before this assertion failed. Existing product files were not edited, and the scratch test was removed after the run.

The next useful proof is to retain this reduced witness and enumerate commit-prefix/refusal/restart states against an independent ledger of replay-owned and previously committed files. Confirm preservation of committed values and source-WAL bytes, along with ordinary interrupted replay cleanup and successful replay. This test layout is provisional, not an additional frozen implementation requirement.

Current upstream overlap inspected: https://github.com/weaviate/weaviate/pull/13657 carries the same adjacency classifier and per-chunk occupied-ID refusal in a merge-down; its implementation does not distinguish the incumbent on retry. The checked hunks of https://github.com/weaviate/weaviate/pull/13857 add secondary-range reads, and https://github.com/weaviate/weaviate/pull/13864 add idle-collector helpers, not this repair. Older open-PR pagination is still being checked. Local file history is shallow, and the bounded remote commit-history read was forbidden, leaving merged-history coverage incomplete. CONTRIBUTING.md requires tests and a linked accepted issue for an external PR; an accepted issue for this new counterexample has not been established.

Change to test: Make startup cleanup distinguish replay-owned output from pre-existing committed segments even after a prefix commit meets an occupied ID. Preserve the incumbent and WAL replayability across refusal and retry; a durable ownership record or equivalent crash-safe protocol is a provisional implementation choice.

Where it lives

perfloop-oss/weaviate · adapters/repos/db/lsmkv/store.go

Evidence

The check fails before this change and passes with it: `With one startup owner and two valid replace-WAL records, refusing an occupied successor ID preserves the incumbent and source WAL bytes; a whole-WAL retry recovers both records and the committed value.`

The check fails before this change and passes with it: `With one startup owner and two valid replace-WAL records, refusing an occupied successor ID preserves the incumbent and source WAL bytes; a whole-WAL retry recovers both records and the committed value.`

The check fails before this change and passes with it: `With one startup owner, retrying a two-record replace-WAL replay interrupted after publishing its two chunks but before unlinking its WAL preserves the bytes and value of a committed successor segment and recovers both WAL records.`

Older replace WAL startup · 15 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 5775232 8702756 +49% (+2830472) +2464326 to +6510072 loss ≤ stated 1.25 × |baseline median| PASSED
B/op 102447 103973 +1.5% (+1540) +1345 to +1878 loss ≤ 0.05 × |baseline median| PASSED
allocs/op 301 326 +8.3% (+25) +24 to +26 loss ≤ stated 0.2 × |baseline median| PASSED

Single chunk replace WAL startup · 15 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 5530954 8622450 +56.6% (+3128334) +2408298 to +3466819 loss ≤ stated 1.25 × |baseline median| PASSED
B/op 61052 62542 +2.4% (+1458) +1320 to +1762 loss ≤ 0.05 × |baseline median| PASSED
allocs/op 251 274 +9.2% (+23) +23 to +23 loss ≤ stated 0.2 × |baseline median| PASSED

Nonnumeric older WAL startup · 16 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 5656854 8429829 +50.4% (+2850970) +2484464 to +3057194 loss ≤ stated 1.25 × |baseline median| PASSED
B/op 101569 102647 +1% (+1062) +913 to +1177 loss ≤ 0.05 × |baseline median| PASSED
allocs/op 293 316 +7.8% (+23) +23 to +24 loss ≤ stated 0.2 × |baseline median| PASSED

Empty truncated WAL startup · 16 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 350204 352243 +1970 −3238 to +11702 loss ≤ 0.05 × |baseline median| PASSED
B/op 90814 90829 +17.5 −17 to +57 loss ≤ 0.05 × |baseline median| PASSED
allocs/op 231 231 0 0 to 0 loss ≤ 0.05 × |baseline median| PASSED

Surviving WAL ownership cleanup · 16 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 1881123 3382145 +80.2% (+1508933) +1247463 to +1661000 loss ≤ stated 2.5 × |baseline median| PASSED
B/op 51200 52474 +2.5% (+1277) +1037 to +1418 loss ≤ 0.05 × |baseline median| PASSED
allocs/op 188 191 +1.6% (+3) +2 to +3 loss ≤ 0.05 × |baseline median| PASSED

Removed WAL ownership cleanup · 16 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 254845 3301803 +1195% (+3045039) +2859364 to +3243918 loss ≤ stated 20 × |baseline median| PASSED
B/op 10794 11536 +7.5% (+814) +731 to +897 loss ≤ stated 0.25 × |baseline median| PASSED
allocs/op 118 128 +8.5% (+10) +10 to +10 loss ≤ stated 0.25 × |baseline median| PASSED

Unpublished replay ownership cleanup · 16 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 263555 3296416 +1147% (+3021945) +2880702 to +3207750 loss ≤ stated 20 × |baseline median| PASSED
B/op 48261 49825 +3.2% (+1525) +1320 to +1654 loss ≤ 0.05 × |baseline median| PASSED
allocs/op 141 154 +9.2% (+13) +13 to +13 loss ≤ stated 0.25 × |baseline median| PASSED

Two-chunk surviving WAL ownership cleanup · 17 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 3345465 3503028 +4.3% (+145253) +7833 to +276485 loss ≤ stated 0.3 × |baseline median| PASSED
B/op 56054 58077 +3.6% (+2000) +1905 to +2139 loss ≤ 0.05 × |baseline median| PASSED
allocs/op 255 256 +0.4% (+1) +1 to +1 loss ≤ 0.05 × |baseline median| PASSED

Checks: 8 of 8 passed. Verification: no defect found.

Timeline