Full record channels can deadlock polling startup and shutdown

perfloop-oss/kinesumer · CYCLIC WAIT

https://perfloop.ai/t/oss/case_6asg64xqpw

Verdict

VERIFIED · settled 2026-10-23

What happened: The assertion is violated on the comparison and satisfied with this change.

Hypothesis

Polling workers can wait for record-channel space while lifecycle code waits for those same workers. This can prevent multi-stream Consume from returning its channel, or prevent shutdown and rebalancing from finishing when the caller is not reading. A temporary test reproduced the startup blockage; deployment frequency is unknown.

At revision a377be60e78d28bd662e420620e10290e4dfc31f, consumeLoop checks stop and close only outside its batch-delivery loop. The send at kinesumer.go:654 cannot observe either signal. syncShardInfoForStream pauses and restarts all workers for each changed stream assignment (syncclient.go:123-129). pause closes stop and waits for workers (kinesumer.go:501-505). Close likewise waits before closing the records channel, with a comment saying the caller should drain the remaining records afterward (847-855).

The temporary TestPerfloopStartupBackpressure used public NewKinesumer and Consume with polling enabled, a fake StateStore, and a local HTTP Kinesis fixture. The first stream returned one 21-record batch into the initially empty, default 20-slot channel. The second stream's shard lookup waited until that channel filled before allowing its assignment update. Consume remained blocked during a 100 ms observation. One package-internal receive released the first worker and allowed Consume to return. That receive is unavailable through the public API until Consume returns. This establishes the constructed correctness failure, not its prevalence or an elapsed-time saving in ordinary traffic.

The repair must account for consumeOnce storing NextShardIterator before delivery (kinesumer.go:695). Merely returning from a cancelled send could skip the remaining records when a retained shard restarts. The independent Case should reproduce the two-stream startup cycle, then verify that a worker blocked in delivery terminates on stop or close without further receives. With completing dependency calls, it should verify replay of the interrupted batch tail and that automatic checkpoints never cover unsent records. Normal complete-batch order and manual marking of received records are regression controls. Scan and commit timeouts, and Close's final synchronization sleep, are not claimed to disappear.

README.md:5-13 documents multi-stream consumption, automatic rebalancing, and checkpoint resumption. Its usage at 60-69 calls Consume before reading the returned channel. This repair needs no new exported operation, type, or configuration. Longer-term maintenance direction beyond the inspected repository documentation remains unknown.

Change to test: Make blocked polling record sends interruptible by stop and close signals. Preserve replay of the undelivered batch tail across restart, without checkpointing records that were not delivered.

Where it lives

perfloop-oss/kinesumer · kinesumer.go

Evidence

The assertion is violated on the comparison and satisfied with this change: `In the tested polling schedules with completed Kinesis and state-store calls, a full records channel does not prevent multi-stream Consume from returning after an assignment change or Close from finishing; a retained shard replays the interrupted batch tail in order, and automatic checkpoints do not include unsent records.`

Checks: 5 of 5 passed. Verification: no defect found.

Timeline