Full-log selection errors leave appends waiting for a response

perfloop-oss/chroma · IGNORED FAILURE

https://perfloop.ai/t/oss/case_a6vd7j1r0g

Verdict

VERIFIED · settled 2026-10-15 · pull request opened as chroma-core/chroma#7917

What happened: The assertion is violated on the comparison and satisfied with this change.

Hypothesis

An append can remain unresolved when a full log refuses to assign another fragment. The selection error is logged, but the queued append's one-use response channel is not completed. This is a source-grounded correctness concern, not a measured incident or latency claim.

At revision 5f30478c11ffdcf8cba1f872993c29d8f84ce0d9, ManagerState::select_for_write maps assign_timestamp returning None to Error::LogFull at batch_manager.rs:59–63. The S3 ManifestManager::assign_timestamp implementation at interfaces/s3/manifest_manager.rs:593–614 can return None when its starting offset is u64::MAX. Once the time and active-writer guards permit selection, take_work propagates that error at line 470 before removing queued work or completing its response channels. This claim excludes the teardown, empty-queue, outstanding-pin, and temporary readiness exits.

LogWriter::append_many_with_options calls OnceLogWriter::append through its retry closure. For a nonempty batch, OnceLogWriter::append pushes work, calls take_work, merely logs an Err at writer.rs:701–703, and then awaits the append response at 705–708. The flusher in OnceLogWriter::open also logs selection errors and sleeps at 617–622 without resolving the queued responses. The outer error handler cannot propagate LogFull or retire the epoch while the append is still awaiting that response. With the writer left open, the append not cancelled, and assignment continuing to refuse the full offset, later selection attempts cannot make the append durable.

The Case proof target is a retained valid full-offset manifest, or a writer brought to that boundary by its last valid record, followed by a one-record call to the public append_many_with_options API using the real BatchManager. Exercise both immediate selection and selection first driven by the flusher. Retain the LogFull failure witness and inspect the queued sender and pending response, rather than treating a timeout alone as proof. The repaired operation must return the assignment error and create no new durable fragment or manifest entry. Ordinary successful appends, temporary batch-delay and active-writer waits, pin-held batches, and contention recovery are regression controls.

Confidence is medium from source inspection; no failure reproduction has been run. This direction adds no public API, configuration, or storage format. The README describes the ordinary append-and-durability contract at rust/wal3/README.md:325–339; this proposal concerns that path, not the temporary migration bootstrap helper described at writer.rs:176–180.

Change to test: Complete the affected append response channels with the terminal fragment-assignment error, whether selection is driven by an append or the flusher. Preserve batch ordering, durability-only success acknowledgements, and existing handling of temporary no-work states and contention.

Where it lives

perfloop-oss/chroma · rust/wal3/src/writer.rs

Evidence

The assertion is violated on the comparison and satisfied with this change: `For a live LogWriter using the production S3 fragment and manifest managers and real BatchManager with local test storage, a valid retained manifest ending at offset u64::MAX, and no other append or teardown, an uncancelled one-record LogWriter::append_many_with_options call returns Error::LogFull and completes its queued response without creating a fragment or manifest entry, whether selection is immediate or first driven by the flusher after a held pin is released.`

Checks: 5 of 5 passed. Verification: no defect found.

Timeline