Merge drops live copies hidden by obsolete source entries
perfloop-oss/weaviate · Unsound pruning
https://perfloop.ai/t/oss/case_yj7744f1ce
Verdict
VERIFIED · settled 2026-11-17
What happened: The check fails before this change and passes with it.
Hypothesis
A successful HFresh merge can erase the only current copy of a live vector. Its duplicate check can reject a candidate because an obsolete source entry has the same ID, even though garbage collection already removed that source entry from the output. This is a correctness and potential ANN recall problem; its occurrence in deployed workloads is unmeasured.
HFresh.doMerge initializes newPosting with p.GarbageCollect at merge.go:68-73, but builds vectorSet by ranging over p at :97-100. GarbageCollect filters in place and returns a shorter slice (posting.go:78-95); the caller's p retains its old length and can still expose removed entries in its tail. The candidate loop first accepts a current version, then skips its ID if it is in that set (:172-185). Both input postings can subsequently be replaced or emptied (:188-224), so the rejected current copy need not remain anywhere.
The new native file hunt_merge_live_union_test.go checked conservation of the set of live current-version IDs through one enqueued merge executed by MergeTask.Execute. Its bounded search domain was 400 ordered pairs of postings, each length 1..2, IDs {0,1}, versions {1,2}, authoritative versions 2 and alive, min posting size 2, maximum 8, equal four-dimensional centroids, no concurrent mutation or faults, and no follow-up workers. It stopped at case 2/400: source=[ID 0/version 1], candidate=[ID 0/version 2]. The separate TestHuntMergeCurrentReplicaReplay repeated exactly that smallest one-ID input. Both failed: live_before=map[0:true], live_after=map[]. The task itself returned nil. Equal centroids also avoid a reassignment task masking the loss.
Command: go test ./adapters/repos/db/vector/hfresh -run '^TestHuntMerge(LiveUnionBounded|CurrentReplicaReplay)$' -count=1 -timeout=90s -v. Exit status was 1 because of the failed conservation assertions. The temporary file was removed. The remaining 398 search cases were not checked, and this fixture does not measure production frequency or claim a full SearchByVector request replay.
The proof target is conservation of live current-version IDs across successful merges and garbage collection for either surviving destination. The smallest next check is this deterministic replay after repair, then small ordered input pairs and a public search replay to confirm discoverability. That checker layout is provisional, not an additional fixed implementation requirement. CONTRIBUTING.md:10 requires tests for contributions, and :38 requires a linked accepted issue before an external PR; no acceptance has been established.
The actual merge patch in https://github.com/weaviate/weaviate/pull/13851 (head 35ce7cda49e76b37cea7af4424810002ac7c0b36) moves destination persistence ahead of centroid deletion and adds WAL writes. It leaves vectorSet and candidate duplicate exclusion unchanged, so this is distinct from that actively owned crash-ordering repair. The inspected https://github.com/weaviate/weaviate/pull/13931 changes recovery and other maintenance paths, not this loop. Older open-PR pagination and shallow local history remain named gaps. User:4's earlier rejection of 'Bound filtered HFresh scoring with ordered replay' concerned a narrow, unsupported filtered-search speedup; this finding instead addresses the ANN recall priority named in that closure.
Change to test: Base duplicate exclusion on live entries actually retained in the merged posting, so an obsolete source entry cannot suppress a current candidate copy. Preserve the union of live current-version IDs regardless of which posting survives.
Where it lives
perfloop-oss/weaviate · adapters/repos/db/vector/hfresh/search.go
Evidence
The check fails before this change and passes with it: `In the ten four-dimensional TestMergeRetainsCurrentVectors fixtures, with other maintenance paused and no injected faults, successful MergeTask.Execute leaves exactly one current-version copy of each expected live ID after garbage collection, and SearchByVector returns those IDs for either surviving destination. Reassignment contains exactly the moved current IDs, excluding obsolete and surviving-resident entries, including when ReassignTask completes between the source snapshot and candidate read and advances version 2 to 3 or 127 to 1.`
HFresh merge · 13 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
509183 |
507127 |
−6629 |
−12145 to +2856 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Checks: 3 of 3 passed. Verification: no defect found.
Timeline
2026-11-17· Case opened