Checkpoint lookup failures can launch uninitialized polling workers

perfloop-oss/kinesumer · IGNORED FAILURE

https://perfloop.ai/t/oss/case_0vqwenwz9y

Verdict

VERIFIED · settled 2026-10-23

What happened: The assertion is violated on the comparison and satisfied with this change.

Hypothesis

A failed checkpoint lookup still starts polling workers for the new shard assignment. On first startup, this can turn a returned StateStore error into an unhandled background panic because the workers' checkpoint state does not exist. A temporary fault-injection test reproduced that sequence; deployment failure frequency is unknown.

At revision a377be60e78d28bd662e420620e10290e4dfc31f, the changed-assignment branch of syncShardInfoForStream pauses workers, writes k.shards[stream], and registers defer k.start() at syncclient.go:127-129. ListCheckPoints can then fail at 146-149, before the checkpoint and offset maps are initialized at 151-156. The deferred restart runs despite that error. NewKinesumer initially creates empty outer maps, so a first nonempty polling assignment has no per-stream checkpoint map when this branch fails.

The temporary TestPerfloopCheckpointFailure invoked public NewKinesumer and Consume with EFOMode false, one alive client, one cached assigned shard, and a StateStore whose ListCheckPoints returned a sentinel error. An isolated subprocess first reported that Consume returned that error, then panicked in getNextShardIterator at kinesumer.go:716 through consumeOnce and consumeLoop. The nil map receiver was reached before a Kinesis request was needed. This test observed background startup after the error, not merely a hypothetical unsafe access.

The claim applies to checkpoint-loading failure after a changed assignment has entered the pause/install branch. Earlier GetShards errors and the unchanged-assignment return do not schedule this restart. Later rebalances with existing maps need not produce the same nil-pointer panic; they still require consistent assignment installation.

The independent Case should inject this failure through the existing Config.StateStore contract and require the original error to return without creating a poller or issuing a shard-iterator request for the rejected assignment. Cleanup must not encounter uninitialized worker state. For an existing assignment, a failed replacement must preserve the prior valid assignment and its checkpoint state. Successful startup and successful reassignment are regression controls. A worker starting for the failed first assignment, or any partially installed replacement, would refute the repair.

README.md:5-13 and 83-88 offer automatic shard assignment and checkpoint-based continuation as library behavior. The proposal repairs that existing contract without adding exported operations, types, or configuration. Longer-term maintenance direction beyond the inspected repository documentation remains unknown.

Change to test: Prepare checkpoint and iterator state before publishing a new shard assignment, and start workers only for a complete assignment. If checkpoint loading fails, return the error without publishing partial state or spawning workers for that assignment, retaining any previous valid assignment.

Where it lives

perfloop-oss/kinesumer · kinesumer.go

Evidence

The assertion is violated on the comparison and satisfied with this change: `For a polling Kinesumer with no prior assignment, when Consume encounters a changed shard assignment and StateStore.ListCheckPoints returns an error, Consume returns that error without publishing the assignment, launching its poller, or issuing GetShardIterator for the rejected shard, and Close completes safely.`

The assertion is violated on the comparison and satisfied with this change: `For a polling Kinesumer with a prior valid assignment, when StateStore.ListCheckPoints fails during replacement, syncShardInfoForStream returns that error, preserves and resumes the previous assignment and its checkpoint, offset, and iterator state, and starts no poller or iterator request for the rejected shard.`

Checks: 4 of 4 passed. Verification: no defect found.

Timeline