Keep MOVE atomic when destination insertion fails
perfloop-oss/dragonfly · Ignored Failure
https://perfloop.ai/t/oss/case_renqvdp41x
Verdict
VERIFIED · settled 2026-11-16
What happened: The check fails before this change and passes with it.
Hypothesis
MOVE deletes its source before attempting destination insertion. When that insertion is rejected for memory pressure, the source can disappear even though the reply says nothing moved. A native check reproduced this on a single string; how often the condition occurs in deployed workloads is unmeasured.
At revision 583b1a67ef1297e681c6482efbee096400634eca, OpMove removes index state, moves the value out, and calls DelMutable before AddNew. AddNew can return OUT_OF_MEMORY through AddOrFind. GenericFamily::Move only sends an error and suppresses its manual journal entry for IO_ERROR; it converts OUT_OF_MEMORY to integer 0 and still records MOVE. A thrown allocation failure is also caught by Transaction::RunCallback, but the entry ignores ScheduleSingleHop's returned status.
The reproducible check is `bash tests/move_failure_atomic_property/run.sh`, stored on prop_v4nez5kz1q. checkProperty run call_xgjdtZc1nRyNnn1jQqOhGoMd exited 1 on the real command path. Its reduced replay is db0 to db1, key `k`, value `v`, no expiry or stickiness, and destination insertion budget -1. One destination callback fired. The output reports reply i0, source value absent, source deadline -2, destination absent and one MOVE journal record. The existing pre-mutation callback sets the same memory-budget field used by EngineShard::CacheStats; this is controlled fault injection, not an observed production OOM. The returned replay is retained on the property node.
The Case should first rerun that witness with ordinary successful and destination-collision controls. The independent proof target is conservation of source/destination logical state and agreement of reply and journal effects when destination publication fails, including the existing expiry, sticky, indexed-hash and tiered-list paths touched by the repair. The present run does not establish disk-error or indexed/list failure behavior. The stored check and its placement are provisional proof scaffolding, not an extra frozen requirement on the final patch.
The inspected implementation in https://github.com/dragonflydb/dragonfly/pull/8698 and its linked https://github.com/dragonflydb/dragonfly/issues/8552 addresses lookup-triggered expiry and its journal order, not MOVE destination publication. https://github.com/dragonflydb/dragonfly/pull/8697 changes snapshot deletion gating. Actual generic-family hunks in https://github.com/dragonflydb/dragonfly/pull/8674 and https://github.com/dragonflydb/dragonfly/pull/8432 change SCAN and transaction clocks, not this handoff. Open work was refreshed before submission; none of those inspected implementations supplies this repair.
Change to test: Guarantee destination publication before the destructive ownership handoff so a rejected insertion retains the source and its logical metadata. Surface the actual failure without journaling an uncommitted MOVE, while preserving successful transfers, destination collisions, expiry, stickiness, search indexes and tiered-list bookkeeping.
Where it lives
perfloop-oss/dragonfly · src/server/generic_family.cc
Evidence
The check fails before this change and passes with it: `When destination insertion is rejected for memory pressure or its pre-mutation callback throws bad_alloc, MOVE preserves the source value, absolute expiry, stickiness and DB0 hash search membership, leaves the destination absent, returns an out-of-memory error and emits no MOVE journal record; this holds for the tested strings, hashes and resident or tiered lists moving in both directions between DB0 and DB1. For tiered lists under the controlled successful byte transport, held stash completion remains owned by the source DB after rejection, subsequent LRANGE returns every original element, and consumed or deleted nodes leave no tiered entries or allocated storage. For long keys with destination top-key sampling in real cluster mode, a sampled allocation failure also leaves slot counts and destination object memory unchanged, including with client tracking enabled; rollback makes no allocation attempts.`
Checks: 4 of 4 passed. Verification: no defect found.
Timeline
2026-11-05· Case opened2026-11-06· PR opened