Startup retry deletes durable segments after replay refuses an occupied ID
perfloop-oss/weaviate · Identity Collision
https://perfloop.ai/t/oss/case_dq9amkedw4
Verdict
VERIFIED · settled 2026-11-17
What happened: The check fails before this change and passes with it.
Hypothesis
Retrying a refused WAL replay can discard an unrelated committed segment. The commit correctly refuses an occupied successor, but startup classifies that incumbent as another provisional replay chunk. A deterministic native test reproduces the loss; the prevalence of this state in deployed buckets is unknown.
At revision 36357612175f8c6977e27cd99fc8c69feec29938, recoveredRun.commit checks each final path immediately before its rename (bucket_recover_from_wal.go:359–388). Its comment explicitly supports an occupied ID after a backward clock step. Chunk 0 can already be durable when chunk 1 is refused. writeOutReplay propagates the error and retains the source WAL, but discards only staged spellings. On retry, chunkedSegmentIDs includes both the renamed prefix and the contiguous incumbent. removeSegmentsOfSurvivingWALs deletes that entire list before mounting segments. Adjacency does not establish shared replay ownership. TestRecoveredRunCommit already requires the incumbent's bytes to survive the refusal, but does not check the next startup.
The new file adapters/repos/db/lsmkv/hunt_replay_occupied_test.go contained TestHuntReplayOccupiedSuccessorSurvivesRetry. Its bounds were two valid replace-WAL records with 64 payload bytes each, one separately flushed durable segment at WAL ID+1, one refused startup, and one retry. It used the existing bucket fixture, public Put and FlushAndSwitch operations, and real NewBucket startup. The exact repository-root command was: go test ./adapters/repos/db/lsmkv -run '^TestHuntReplayOccupiedSuccessorSurvivesRetry$' -count=1 -v -timeout=90s. It exited 1. Deciding output: 'first open refused occupied successor after chunk 0; incumbent and WAL are unchanged'; 'retry succeeded; WAL records recovered; successor bytes preserved=false'; 'actual : []byte(nil)'; 'startup retry must preserve unrelated committed data after an occupied-ID refusal'. The expected value was the separately committed durable-value. Both WAL records were checked successfully before this assertion failed. Existing product files were not edited, and the scratch test was removed after the run.
The next useful proof is to retain this reduced witness and enumerate commit-prefix/refusal/restart states against an independent ledger of replay-owned and previously committed files. Confirm preservation of committed values and source-WAL bytes, along with ordinary interrupted replay cleanup and successful replay. This test layout is provisional, not an additional frozen implementation requirement.
Current upstream overlap inspected: https://github.com/weaviate/weaviate/pull/13657 carries the same adjacency classifier and per-chunk occupied-ID refusal in a merge-down; its implementation does not distinguish the incumbent on retry. The checked hunks of https://github.com/weaviate/weaviate/pull/13857 add secondary-range reads, and https://github.com/weaviate/weaviate/pull/13864 add idle-collector helpers, not this repair. Older open-PR pagination is still being checked. Local file history is shallow, and the bounded remote commit-history read was forbidden, leaving merged-history coverage incomplete. CONTRIBUTING.md requires tests and a linked accepted issue for an external PR; an accepted issue for this new counterexample has not been established.
Change to test: Make startup cleanup distinguish replay-owned output from pre-existing committed segments even after a prefix commit meets an occupied ID. Preserve the incumbent and WAL replayability across refusal and retry; a durable ownership record or equivalent crash-safe protocol is a provisional implementation choice.
Where it lives
perfloop-oss/weaviate · adapters/repos/db/lsmkv/store.go
Evidence
The check fails before this change and passes with it: `With one startup owner and two valid replace-WAL records, refusing an occupied successor ID preserves the incumbent and source WAL bytes; a whole-WAL retry recovers both records and the committed value.`
The check fails before this change and passes with it: `With one startup owner and two valid replace-WAL records, refusing an occupied successor ID preserves the incumbent and source WAL bytes; a whole-WAL retry recovers both records and the committed value.`
The check fails before this change and passes with it: `With one startup owner, retrying a two-record replace-WAL replay interrupted after publishing its two chunks but before unlinking its WAL preserves the bytes and value of a committed successor segment and recovers both WAL records.`
Older replace WAL startup · 15 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
5775232 |
8702756 |
+49% (+2830472) |
+2464326 to +6510072 |
loss ≤ stated 1.25 × |baseline median| |
PASSED |
B/op |
102447 |
103973 |
+1.5% (+1540) |
+1345 to +1878 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
301 |
326 |
+8.3% (+25) |
+24 to +26 |
loss ≤ stated 0.2 × |baseline median| |
PASSED |
Single chunk replace WAL startup · 15 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
5530954 |
8622450 |
+56.6% (+3128334) |
+2408298 to +3466819 |
loss ≤ stated 1.25 × |baseline median| |
PASSED |
B/op |
61052 |
62542 |
+2.4% (+1458) |
+1320 to +1762 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
251 |
274 |
+9.2% (+23) |
+23 to +23 |
loss ≤ stated 0.2 × |baseline median| |
PASSED |
Nonnumeric older WAL startup · 16 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
5656854 |
8429829 |
+50.4% (+2850970) |
+2484464 to +3057194 |
loss ≤ stated 1.25 × |baseline median| |
PASSED |
B/op |
101569 |
102647 |
+1% (+1062) |
+913 to +1177 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
293 |
316 |
+7.8% (+23) |
+23 to +24 |
loss ≤ stated 0.2 × |baseline median| |
PASSED |
Empty truncated WAL startup · 16 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
350204 |
352243 |
+1970 |
−3238 to +11702 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
90814 |
90829 |
+17.5 |
−17 to +57 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
231 |
231 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Surviving WAL ownership cleanup · 16 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
1881123 |
3382145 |
+80.2% (+1508933) |
+1247463 to +1661000 |
loss ≤ stated 2.5 × |baseline median| |
PASSED |
B/op |
51200 |
52474 |
+2.5% (+1277) |
+1037 to +1418 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
188 |
191 |
+1.6% (+3) |
+2 to +3 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Removed WAL ownership cleanup · 16 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
254845 |
3301803 |
+1195% (+3045039) |
+2859364 to +3243918 |
loss ≤ stated 20 × |baseline median| |
PASSED |
B/op |
10794 |
11536 |
+7.5% (+814) |
+731 to +897 |
loss ≤ stated 0.25 × |baseline median| |
PASSED |
allocs/op |
118 |
128 |
+8.5% (+10) |
+10 to +10 |
loss ≤ stated 0.25 × |baseline median| |
PASSED |
Unpublished replay ownership cleanup · 16 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
263555 |
3296416 |
+1147% (+3021945) |
+2880702 to +3207750 |
loss ≤ stated 20 × |baseline median| |
PASSED |
B/op |
48261 |
49825 |
+3.2% (+1525) |
+1320 to +1654 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
141 |
154 |
+9.2% (+13) |
+13 to +13 |
loss ≤ stated 0.25 × |baseline median| |
PASSED |
Two-chunk surviving WAL ownership cleanup · 17 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
3345465 |
3503028 |
+4.3% (+145253) |
+7833 to +276485 |
loss ≤ stated 0.3 × |baseline median| |
PASSED |
B/op |
56054 |
58077 |
+3.6% (+2000) |
+1905 to +2139 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
255 |
256 |
+0.4% (+1) |
+1 to +1 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Checks: 8 of 8 passed. Verification: no defect found.