Cancelled batch commits can miss the garbage-collection wakeup

perfloop-oss/dolt · Missed notification

https://perfloop.ai/t/oss/case_maksqy7axt

Verdict

VERIFIED · settled 2026-11-10 · merged as dolthub/dolt#12427

What happened: The check fails before this change and passes with it.

Hypothesis

A batch publisher can keep waiting after its caller cancels because the garbage-collection admission wait loses its cancellation notification. This retains the committing goroutine until collection changes state. The failure needs cancellation during finalization and a narrow event order; its occurrence rate and wall-time impact in ordinary workloads are unmeasured.

The ordinary route is DoltDB.CommitHeadUpdates → hooksDatabase.CommitDatasets → database.CommitDatasets → database.update → database.tryCommitChunks → ValueStore.Commit. In go/store/types/value_store.go:425–451, waitForNotFinalizingGC holds gcMu while checking ctx.Err(), but its cancellation goroutine calls gcCond.Broadcast() without that mutex. Cancellation can broadcast after the check returns nil but before gcCond.Wait registers the waiter. The cancelled condition then holds with no cancellation wakeup remaining. Collection can still complete, so this is not a claim of permanent deadlock. SQL transaction publication can retain its transaction locks across this call.

The native check on prop_nx8bdety3b ran `go -C go test ./store/types -run '^TestPerfloopGCCommitCancellation$' -count=1 -timeout=60s -v`. checkProperty call_Ij7YJoQvrXvXJGzaCjZkYt23 failed the before-registration schedule: cancelled=true, returned-before-GC-completion=false, underlying-commits=0. The already-cancelled and registered-waiter controls returned cancellation before GC completion; the uncancelled control committed once after completion. This is a real ValueStore.Commit call using existing GC transition seams and a context whose first Err read precedes cancellation but returns afterwards. The run returned the reduced replay {schedule: gap, yields: 64}. Quiescence, not a timeout, established that the waiter remained asleep after runnable notifier work drained.

Confidence is medium for the batch-level contribution: the shared admission mechanism has an executable counterexample, while a complete CommitDatasets replay is the independent Case proof target. Reproduce cancellation before publication while GC is paused, assert cancellation completes without finishing GC and leaves the root unchanged, and preserve successful uncancelled publication plus cancellation after wait registration. The checker placement and scheduling technique are provisional, not extra frozen implementation requirements. A small synchronization change and regression test appear sufficient; lock ordering and notifier cleanup still need review.

The complete current open-PR listing showed no matching cancellation repair. The accepted revision is the merge of https://github.com/dolthub/dolt/pull/12338; its implementation narrows GC root-visitor callbacks and centralizes the keeper check, rather than changing this admission wait. No separate review archive was named in the inspected repository guidance. Older merged-file history remains a gap because the checkout is shallow and the bounded GitHub history read was refused.

Change to test: Synchronize the cancellation wakeup with the garbage-collection admission check and wait registration so a cancelled publisher can leave without waiting for collection to finish. Preserve collection exclusion, balanced admission finalizers, ordinary publication and pre-publication cancellation errors without adding an arbitrary timeout.

Where it lives

perfloop-oss/dolt · go/libraries/doltcore/doltdb/doltdb.go

Evidence

The check fails before this change and passes with it: `For a two-branch CommitDatasets batch waiting behind finalizing GC, cancellation between a nil context-error read and wait registration returns context.Canceled with no datasets before GC completes, without changing the root or either head.`

Checks: 6 of 6 passed. Verification: no defect found.

Timeline