Async storage writes: MsgVote leaves before the candidate's term and vote are durable, so a crash lets one node lead a term twice
perfloop-oss/raft · Uncatalogued correctness
https://perfloop.ai/t/oss/case_jwheb7ead4
Verdict
VERIFIED · settled 2026-10-31
What happened: The check fails before this change and passes with it.
Hypothesis
With asynchronous storage writes, a candidate can publish vote requests before its new term and self-vote are durable. The submitter reports that a crash during the pending write lets the same node lead the same term twice and causes replicas to apply different entries at the same index and term. This run has not independently reproduced that outcome. The occurrence rate and full affected-user scope remain unmeasured.
The edit subject is (*raft).send at raft.go:514-601, specifically the queue-selection condition at line 546. It delays MsgAppResp, MsgVoteResp and MsgPreVoteResp through msgsAfterAppend, but sends MsgVote through msgs (lines 546-599). Its comment requires election votes to be durable before publication (lines 547-578). The supported consumer route is (*RawNode).Campaign (rawnode.go:83-87), (*raft).Step (raft.go:1089-1271), hup (973-990), campaign (1025-1073), and send. Campaign sends peer vote requests at line 1071 while deliberately deferring its own vote response at lines 1053-1060.
Ready.Messages explicitly permits immediate delivery with AsyncStorageWrites (node.go:98-106). readyWithoutAccept places msgs in Ready.Messages and adds local append work separately (rawnode.go:139-187). newStorageAppendMsg carries HardState fields and attaches msgsAfterAppend as responses (rawnode.go:223-260). The storage contract requires those writes to be durable before the attached responses are delivered (raft.go:153-187).
The submitted failure chain is that two remote grants elect the three-node candidate while its own append is stalled, letting it replicate entries at an unpersisted term. A crash then restores the old term, allowing another campaign for the same term. Voters can repeat a vote for the same sender, subject to log freshness (raft.go:1212-1257). The submitter reports that this produces a different leader log while a follower retains the old payload because findConflict compares entry index and term, not data (log.go:144-167). The resulting state-machine divergence, rather than simply an early message, is the safety property to investigate.
The submitter reports reproducing on main c51f010, the accepted revision c51f010c851edd0dada5af41bafbf68e54617767, with output: "index 2: 1/2 y applied by [1 3]; 1/2 x applied by [2]; state machines diverged". They characterize the impact as a Raft safety violation for every AsyncStorageWrites user after a candidate crash during a slow disk write. The Case must check that reported outcome and conditional scope; the report is not an independently established result or an incident-frequency measurement.
The smallest useful next check is a new regression in package raft, for example TestAsyncStorageWritesCandidateCrashSafety. Node 1 must use AsyncStorageWrites with a manually driven append thread; nodes 2 and 3 must use ordinary Ready/Advance. Stall node 1's disk and deliver only vote messages while its append work remains pending. Let the two grants elect node 1, then have it propose "x" and send that entry to node 2 only. Crash node 1 and rebuild it from its persisted MemoryStorage, without its lost volatile state or pending append work. Re-elect it at the same term and propose "y". Assert that no two nodes applied different entries at one (index, term). The requested regression must fail on c51f010 and pass with the fix; the applied-entry records and election schedule are the confirmation or rejection signal for the submitted mechanism.
The outcome also includes preserving immediate MsgPreVote delivery, updating testdata/async_storage_writes.txt and testdata/async_storage_writes_append_aba_race.txt for the changed message order, and a passing go test -race ./.... These fixtures currently expose immediate peer MsgVote delivery before append-thread completion (testdata/async_storage_writes.txt:23-49 and testdata/async_storage_writes_append_aba_race.txt:85-114).
Fresh upstream comparison found related but different open proposals. https://github.com/etcd-io/raft/pull/127, head 636ff4572ca3e2b4c55ea17d8439a0603086260b, adds tests for winning before a delayed self-vote, but its inspected patch does not defer MsgVote or test crash/restart divergence. https://github.com/etcd-io/raft/pull/544, head b60b90c06086e29f67bf35fff3b503458b7b8f1a against base 1931ff5beacc1e2539003c0c70f096285f04c425, addresses https://github.com/etcd-io/raft/issues/543 by holding snapshot append work until earlier entries finish applying; that is a different async ordering problem and does not change send. https://github.com/etcd-io/raft/pull/452, head a775bccb5d28521b656769192299295894714d2d, only changes unstable-log diagnostic severity and the shared async-storage fixture. The remaining difference is candidate HardState durability before peer vote-request publication and its crash-safety regression. These remain open proposals, not evidence of maintainer acceptance. Other implementation hunks, closed work and review discussions were not inspected, so broader upstream overlap remains an explicit gap.
Change to test: Make candidate vote requests wait for durable HardState{Term,Vote} by adding MsgVote to the msgsAfterAppend condition in (*raft).send at raft.go:546, like MsgVoteResp; keep MsgPreVote immediate. Add the package-raft crash regression described above, update both named async-storage fixtures for the new message order, and require go test -race ./... to pass.
Where it lives
perfloop-oss/raft · rawnode.go
Evidence
The check fails before this change and passes with it: `With AsyncStorageWrites enabled only on node 1 in a controlled three-voter RawNode schedule with PreVote disabled, after x is sent only to node 2 and node 1 crashes with append work pending, no replicas may apply different payloads at the same (index, term).`
Checks: 4 of 4 passed. Verification: no defect found.