In-flight NACK repair can mix packets across receiver restarts

perfloop-oss/livekit · Identity Collision

https://perfloop.ai/t/oss/case_vt9pmts6t2

Verdict

VERIFIED · settled 2026-11-17 · merged as livekit/livekit#5213

What happened: The check fails before this change and passes with it.

Hypothesis

A NACK worker can combine pre-restart packet metadata with post-restart media bytes. The resulting repair carries the old outgoing sequence number and timestamp but different media. A deterministic native schedule reproduces this; the frequency of overlapping repairs, restarts and source-sequence reuse in deployed traffic is unknown.

retransmitPackets obtains independently copied metadata from sequencer.getExtPacketMetas before calling Receiver.ReadRTP with only layer and sourceSeqNo. The sequencer lock is no longer held during that read. ReceiverRestart flushes the sequencer, but it cannot invalidate the worker's already-returned epms slice. BufferBase.restartStreamLocked resets RTP statistics and resynchronizes the packet bucket. A supported new stream can reuse a source sequence number, so that key alone does not distinguish the two generations. ReceiverBase.restartInternal stops normal forwarding workers before resetting buffers and notifying DownTracks; it does not join the separate retransmission worker.

The new pkg/sfu/retransmit_hunt_restart_test.go ran four explicit schedules around one normally forwarded VP8 packet: no restart, restart before metadata lookup, restart after metadata lookup but before cache read, and restart after cache read. It used the existing real-Pion DownTrack binding fixture, negotiated RTX, S0T0, no budget, and the actual Buffer.RestartStream, Write and GetPacket operations. A new test-only in-package helper called the existing retransmitPackets seam synchronously, so a correctly dropped repair could be distinguished from a completed wrong repair without a timing-based absence assertion. The only reused source sequence was 6000. The chosen schedules are a bounded check, not an exhaustive proof over all interleavings.

Command: go test ./pkg/sfu -run '^TestHuntRestart' -count=1 -timeout=60s -v. Both the schedule search and the reduced deterministic TestHuntRestartAfterMetadataBeforeRead failed with: 'counterexample stage=2 OSN=6000 oldTS=90000 repairTS=90000 originalTail=11 repairTail=22 replacementTail=22', followed by '--- FAIL: TestHuntRestartAfterMetadataBeforeRead'. The old media ended in 0x11 and the replacement in 0x22. No malformed packet, short-buffer error or budget rejection was required. Scratch files were removed after the run.

The smallest useful next Case check is the same boundary under the real receiver restart coordinator, followed by a generation-aware implementation checked against both permitted outcomes: original bytes or no obsolete repair. The test layout is provisional. A repair must coordinate the cache reset with metadata snapshots, not merely add another sequencer clear. That adds generation/synchronization complexity at the lookup handoff; the smallest safe representation remains to be chosen. This is a demonstrated integrity violation, not an alleged incident or an unsynchronized-memory-access claim.

The deciding upstream record is merged https://github.com/livekit/livekit/pull/4760 (68ecd38c00afc236bf5e8aa553660c7e7e7a912b), whose explicit purpose is preventing metadata from surviving a resynced bucket. Its implementation and test cover lookups performed after flush, not snapshots already held before flush. This proposal retains that fix and addresses the remaining in-flight boundary; it does not resubmit sequencer flushing. Current open-PR pages and the available DownTrack/pacer hunks in https://github.com/livekit/livekit/pull/5195 and https://github.com/livekit/livekit/pull/4963 did not implement this generation handoff. The community archive's later RTX search was rate-limited and Slack discussion was unavailable, so upstream review outside the code host remains a named evidence gap.

Change to test: Keep packet metadata and receiver bytes tied to the same stream generation across restart, and discard a repair whose generation is obsolete. Use a generation-aware cache lookup or an equivalent atomic ownership handoff rather than relying only on flushing sequencer entries that an in-flight worker has already copied.

Where it lives

perfloop-oss/livekit · pkg/sfu/downtrack.go

Evidence

The check fails before this change and passes with it: `For a forwarded VP8 packet with reused source sequence number, timestamp and SSRC, NACK repair across ReceiverBase.Restart emits the original media or drops the obsolete repair in the four before/after metadata and cache-read schedules.`

Timeline