Polling stops duplicate checkpoint batches and can persist stale offsets

perfloop-oss/kinesumer · N+1 Calls

https://perfloop.ai/t/oss/case_qw29hkqz69

Verdict

VALIDATED BY A DIFFERENT CHANGE · settled 2026-10-23 · pull request opened as daangn/kinesumer#29

What happened: The check fails before this change and passes with it.

Hypothesis

Pausing or closing a polling consumer makes every exiting worker write checkpoints for the whole consumer. This multiplies storage work and can leave an older checkpoint in storage when snapshots finish out of order. Deployment frequency and actual DynamoDB latency remain unmeasured.

At revision a377be60e78d28bd662e420620e10290e4dfc31f, consumePolling starts one worker per assigned shard. Each worker taking the stop or close branch calls Commit before returning (kinesumer.go:615-634). Commit scans every stream's marked offsets and starts a write for each nonempty stream. Neither pause nor Close performs a final checkpoint after waiting for workers (501-505, 847-855). The benefiting library operations are Refresh, documented at README.md:60-62, and Close, documented by its exported Go comment. Their ordinary polling route reaches this work without a failure or retry. The README's two-stream client uses polling and automatic marking by default; the relevant state is that its assigned workers have completed batches and marked their offsets. How frequently consumers refresh or close is unknown.

The relevant storage cost is inferred, not a cloud timing measurement. With W workers taking these exit branches, a stable marked-key set containing C_s rows in each of S nonempty streams causes W*S UpdateCheckPoints calls and W*sum(C_s) submitted rows. A single final snapshot needs S calls and sum(C_s) rows. These counts exclude independent periodic/manual commits and do not apply to workers that already exited through the closed-shard cleanup branch. For the documented two-stream shape with one marked worker per stream, the expected reduction is four submissions and four rows to two submissions and two rows. Checkpoint submissions are Refresh's direct StateStore write work in polling mode; Close also has a client deregistration. Thus the removed work is material in storage-operation count, without implying a similar elapsed-time gain.

This is not one network call per checkpoint row. stateStore.UpdateCheckPoints already batches rows through dynamo.Batch.Write.Put.RunWithContext (statestore.go:260-280). The pinned github.com/guregu/dynamo v1.10.4 batchwrite.go creates separate BatchWriteItem requests for each invocation, in chunks of at most 25 rows, without coalescing concurrent invocations. With successful requests, no retries, and all rows processed, the corresponding request count is W*sum(ceil(C_s/25)), rather than sum(ceil(C_s/25)). Rows contain the checkpoint key, shard ID, sequence number, and timestamp, not record payloads. Cross-stream flattening would affect the existing per-stream timeout and error granularity (kinesumer.go:84-85, 792-803), which are constraints on a broader batching change.

Temporary TestPerfloopPollingCheckpointCounts exercised public NewKinesumer, Consume, Refresh, and Close with default commit settings, a local HTTP Kinesis fixture, and a counting StateStore. Initial marks were seeded through public MarkRecord after receiving each shard's first record, producing the same marked state automatic complete-batch delivery reaches. Refresh and Close each submitted 1/1, 4/4, 32/1024, and 128/16384 batches/rows for one worker, two one-worker streams, 32 workers, and 128 workers respectively. These are fixture counts, not evidence of consumer shard distributions; the larger widths are optional stress samples. The combined race-detector run also reported a separate stop-channel race, so it was not a clean package test.

Temporary TestPerfloopConcurrentFinalCheckpoint reproduced the ordering failure on three runs under the race detector. Worker A's global snapshot contained worker B's earlier mark 100 and was held before persistence. B then completed its in-flight GetRecords batch, automatically marked 200, and persisted a newer snapshot. Releasing A last left the store at 100 after Close returned, while the in-memory mark remained 200. Every checkpoint call returned success. The fake store used the same unconditional replacement semantics as the default backend's Put operations; this was not an observed production incident.

The independent Case should reproduce the public lifecycle count check with the default checkpoint-write implementation and record actual backend request and row counts. The required two-stream, one-marked-worker-per-stream check should reduce four successful checkpoint submissions to two, with one lifecycle-triggered submission per nonempty stream. A one-worker run is a no-regression control. Controlled completion orders must leave every final worker mark persisted, including when an older periodic commit is already in flight. A stale final offset, lost stream checkpoint, or unchanged duplicated submission count would refute the contribution. Manual marking after workers have finished is outside that guarantee. Blocked record sends, dependency timeouts, and Close's final synchronization sleep are not claimed to disappear.

README.md:5-13 and 83-88 offer multi-stream consumption, rebalancing, and checkpoint resumption. The change needs no new exported operation, type, configuration, or protocol. That README contains no retirement notice; maintenance direction beyond the inspected repository evidence remains unknown.

Change to test: Make each polling stop or close cycle issue one consumer-wide checkpoint flush after its workers have finished, rather than one global flush per worker. Order that final snapshot and its writes against other in-flight commits so final worker marks survive, while retaining per-stream timeout and error behavior.

Where it lives

perfloop-oss/kinesumer · kinesumer.go

Evidence

The check fails before this change and passes with it: `For a polling Kinesumer with one successfully marked worker per nonempty stream, each Refresh and Close submits exactly one default DynamoDB BatchWriteItem request and one row per stream (two requests and rows for two streams, one for the one-worker control), assuming each successful write returns no unprocessed rows.`

The check fails before this change and passes with it: `For a polling Kinesumer using a synchronous checkpoint store with unconditional last-write-wins puts, with one worker per stream and final worker marks of 200 in two streams, when Refresh or Close stops those workers while an older successful local periodic snapshot of 100 is in flight, both final checkpoints remain 200 after the lifecycle operation returns, assuming all local writes succeed and no independent writer changes those keys.`

Polling Refresh and Close checkpoint writes · 9 sample pairs

metric baseline candidate paired median change confidence range required result
refresh_two_streams_batchwrite_requests 4 2 −50% (−2) −2 to −2 gain > 0; gain > 0.05 × |baseline median|; arm medians separated PASSED
refresh_two_streams_checkpoint_rows 4 2 −50% (−2) −2 to −2 gain > 0; gain > 0.05 × |baseline median|; arm medians separated PASSED
refresh_one_worker_batchwrite_requests 1 1 0 0 to 0 loss ≤ 0.05 × |baseline median| PASSED
refresh_one_worker_checkpoint_rows 1 1 0 0 to 0 loss ≤ 0.05 × |baseline median| PASSED
close_two_streams_batchwrite_requests 4 2 −50% (−2) −2 to −2 gain > 0; gain > 0.05 × |baseline median|; arm medians separated PASSED
close_two_streams_checkpoint_rows 4 2 −50% (−2) −2 to −2 gain > 0; gain > 0.05 × |baseline median|; arm medians separated PASSED
close_one_worker_batchwrite_requests 1 1 0 0 to 0 loss ≤ 0.05 × |baseline median| PASSED
close_one_worker_checkpoint_rows 1 1 0 0 to 0 loss ≤ 0.05 × |baseline median| PASSED

Polling Refresh one-worker local elapsed-time control · 11 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 2705 2539 −3.8% (−103) −374 to −29 loss ≤ 0.05 × |baseline median| PASSED

Checks: 7 of 7 passed. Verification: no defect found.

Timeline