Failed merge enqueue suppresses later retries

perfloop-oss/weaviate · Derived summary drift

https://perfloop.ai/t/oss/case_2y1grgzfw8

Verdict

VERIFIED · settled 2026-11-17 · merged as weaviate/weaviate#13940

What happened: The check fails before this change and passes with it.

Hypothesis

A failed write to the merge queue leaves the posting recorded as pending even though no task was queued. Later searches can therefore report a successful merge enqueue without scheduling the maintenance. The frequency of recoverable queue-write failures and their effect on ANN latency or recall are unmeasured.

SearchByVector enqueues an undersized posting at search.go:155-161. TaskQueue.EnqueueMerge adds its ID to mergeList before DiskQueue.Push, but returns the push error without removing that ID. The next call exits successfully at TryAdd. MergeDone normally removes the bit, but the failed push created no task to execute. DiskQueue.Push increments its record count only after its writer succeeds (queue/queue.go:321-326).

The new native check hunt_merge_enqueue_retry_test.go, TestHuntMergeEnqueueRetryAfterCreateFailure, used one ID, one recoverable chunk-create failure, two enqueue calls, and no queue consumption or concurrency. It temporarily replaced the empty queue directory with a file, restored the directory after the first error, and retried. Command: go test ./adapters/repos/db/vector/hfresh -run '^TestHuntMergeEnqueueRetryAfterCreateFailure$' -count=1 -timeout=90s -v. The run exited 1 with first_failed=true pending_after_failure=true retry_error=<nil> queued_after_retry=0; the assertion expected one queued task. The scratch test was removed. This is a deterministic counterexample, not a production-frequency measurement.

The independent proof target is failure-safe enqueue state: a rejected queue write must not suppress a later successful retry. The smallest next check is the same replay after repair, followed by bounded duplicate and concurrent-enqueue schedules. The test placement is provisional; CONTRIBUTING.md:10 independently requires tests for a contribution, and :38 requires a linked accepted issue before an external PR. No accepted issue has been established here.

Upstream draft https://github.com/weaviate/weaviate/pull/13851 (head 35ce7cda49e76b37cea7af4424810002ac7c0b36) changes merge persistence ordering, not enqueue bookkeeping. The inspected changes in https://github.com/weaviate/weaviate/pull/13931 concern startup recovery, reassign fallback, and split follow-up work, not this failure. Remaining open-PR pagination and the shallow local history are evidence gaps, not a claim of upstream clearance. User:4's closure of the earlier filtered-HFresh speedup concerned its narrow, unsupported performance value; it does not refute this separate maintenance-correctness mechanism.

Change to test: Leave no pending-task claim after an unsuccessful merge enqueue, so a later retry can queue the work. Preserve successful-task deduplication and serialized merge execution.

Where it lives

perfloop-oss/weaviate · adapters/repos/db/vector/hfresh/search.go

Evidence

The check fails before this change and passes with it: `After a recoverable chunk-create failure, TaskQueue.EnqueueAnalyze can retry the same posting ID and queues exactly one task despite a duplicate retry, with no queue consumption.`

The check fails before this change and passes with it: `After a recoverable chunk-create failure, TaskQueue.EnqueueSplit can retry the same posting ID and queues exactly one task despite a duplicate retry, with no queue consumption.`

The check fails before this change and passes with it: `After a recoverable chunk-create failure, TaskQueue.EnqueueMerge can retry the same posting ID and queues exactly one task despite a duplicate retry, with no queue consumption.`

The check fails before this change and passes with it: `When a MergeTask executes while its posting lock is held, its successful replacement enqueue remains pending and suppresses the next duplicate enqueue, with the replacement left unprocessed.`

Checks: 4 of 4 passed. Verification: no defect found.

Timeline