Complete failed producer tickets without waiting for unrelated reserved permits

perfloop-oss/s2 · Cyclic wait

https://perfloop.ai/t/oss/case_xv5daw63n2

Verdict

VERIFIED · settled 2026-11-18 · pull request opened as s2-streamstore/s2#868

What happened: The check fails before this change and passes with it.

Hypothesis

A producer can report a failed flush while an earlier record's acknowledgement future remains pending. A caller that keeps a public submission reservation while awaiting that ticket can then wait indefinitely. A bounded public-API test reproduced the dependency; its prevalence in applications is unknown.

At revision 19c0c21c627dc1a10e14aabd44d27e62ea3dde5d, terminate_producer dispatches batch acknowledgements that are ready, but retains unfinished PendingBatchAck values and their record-response senders. It publishes the terminal error and rejects other pending records and controls, then awaits cmd_rx.recv() until the closed command channel drains. A public RecordSubmitPermit owns a channel permit that prevents that receive from returning None. Producer::run cannot resume polling or drop the unfinished batch acknowledgements while awaiting this cleanup. This applies when an earlier batch acknowledgement is still pending at terminal entry, not to tickets already dispatched successfully.

The new temporary native test sdk/tests/producer_terminal_checks.rs::terminal_failure_does_not_wait_for_reserved_permit uses one empty valid record, one one-byte record, an 8-byte batch limit, one-record batches, and one held public reservation. A loopback S2S peer receives the first append and withholds its acknowledgement. The later record exceeds the configured batch limit. Run: cargo test --locked -p s2-sdk --test producer_terminal_checks terminal_failure_does_not_wait_for_reserved_permit -- --exact --nocapture. The run exited 101 with 'flush=Validation; later_ticket=Validation; earlier_pending_with_permit=true', followed after releasing the permit by 'earlier_after_permit_release=Ok(Err(Validation(ValidationError("record size in metered bytes (9) exceeds max_batch_bytes (8)"))))'. Its assertion failed with 'earlier submitted ticket is pending after terminal flush while an unrelated public permit remains held'. Bounds were one fixed event order, a 100 ms readiness window, and 2 s setup/post-release limits. The timing alone is bounded evidence; the retained senders and channel-permit dependency explain the unbounded wait.

Merged https://github.com/s2-streamstore/s2/pull/698 establishes durable flushes and sticky terminal errors, but its implementation retains this cleanup dependency. Open https://github.com/s2-streamstore/s2/pull/842 changes zero-linger batching rather than terminal cleanup. The reviewed open-PR page contains no producer-cleanup change. The SDK contribution documentation uses GitHub issues and pull requests, not a separate review archive.

The independent Case target is to establish that terminal handling settles earlier unresolved record obligations before waiting for unrelated reservations, while preserving already-ready successful acknowledgements and rejection of late submissions. Rerunning the reduced test is the smallest useful check. Extending its schedule matrix to the other pending-record states is a provisional proof method, not a required test layout or a claim of exhaustive coverage.

Change to test: Make terminal cleanup settle unfinished record tickets and release their associated capacity before any command-drain wait that can depend on a caller-held reservation. Preserve already-ready acknowledgements, the sticky terminal cause, and rejection of submissions sent through previously reserved permits.

Where it lives

perfloop-oss/s2 · sdk/src/producer.rs

Evidence

The check fails before this change and passes with it: `With a 1 MiB producer byte limit, when a valid record and flush fail with stream_not_found while an unrelated public reservation retains byte capacity, both an already-waiting valid submission and a fresh valid submission fail with the sticky terminal error before that reservation is released.`

The check fails before this change and passes with it: `When an earlier valid append has reached the peer but remains unacknowledged and a later record fails producer batching validation, the earlier ticket resolves without claiming no side effects before any held public reservation is released; unsent records, controls, and late submissions keep the validation error, with and without a held reservation.`

The check fails before this change and passes with it: `When later record batching validation fails, the producer closes and drains earlier submitted appends independently of held public record reservations: a valid peer acknowledgement sent after validation begins is delivered with its exact indexed fields, and a peer error is delivered with its actual cause; unsent records, controls and late reserved submissions retain the sticky validation error.`

The check fails before this change and passes with it: `When an append session fails with stream_not_found, byte-capacity and batch-count waiters, fresh submissions, and submissions through earlier public reservations receive the terminal error without requiring those reservations to be released.`

Checks: 5 of 5 passed. Verification: no defect found.

Timeline