Full delayed queues keep the worker scanning instead of waiting

perfloop-oss/helix-db · Polling Loop

https://perfloop.ai/t/oss/case_xzkn0dkpyh

Verdict

VERIFIED · settled 2026-11-15 · pull request opened as HelixDB/helix-db#1301

What happened: The check fails before this change and passes with it.

Hypothesis

Automatic scheduling repeatedly resolves a full queue even when every operation is delayed. Finishing the scan still counts as progress, so the dispatcher never reaches its existing wait path. A native fixture confirms sustained CPU use at the 64-item page boundary. How often ordinary writers retain that many simultaneously delayed operations remains unknown; confidence is medium.

At revision 4b0982f, schedule_operation_task returns Continue after a full page. When a later page exhausts the prefix with a cursor present, it resets the cursor but also returns Continue because wrapping is true. dispatch_worker_cycle resets lanes_without_work on Continue and waits only after four idle lane attempts. With stable future-deadline pointers spanning at least one full 64-item page, available permits, automatic lanes and no other task completions, the cyclic scans never let that idle count reach four. Each pass calls observe_operation_pointer again, performing its dependent reads and validation without making an operation eligible.

This is not the bounded recovery-sweep timer running expensively. The timer is never reached in this state. The normal runtime installs automatic vector/text scheduling and enabled secondary scheduling. release_transient_claim persists same-writer failure deadlines using backoff_millis, bounded from one to thirty seconds. The observer honors those deadlines. Thus delayed records have a production source, although the measured queue was directly seeded rather than generated by a real failure incident.

Measurement receipt call_oi4H4SGNofguPLc1wBcS8OO6, output:1, records `python3 .perfloop_worker_measure.py`, including `cargo run --release -p db --features bench-internals --example perfloop_worker_probe -- compare 64 500`. It drives the actual run_worker_cycle with three automatic lanes, real in-memory SlateDB, no publication work, valid secondary Activate records and same-epoch thirty-second transient backoff. Temporary counters measure scans, observations and wait entries. An external shutdown thread bounds each run; no driver step may execute, and every sampled deadline remained in the future.

For 64 records, the baseline made 1,112 scans and 35,648 observations with zero wait entries in 501.313 ms. Process CPU accounting reported 490 ms. A minimal comparator changed only end-of-sweep reporting: six scans, 192 observations and one wait entry, with CPU reported below the counter's resolution. At 65 and 128 records, the baseline again had zero waits and about 480 ms CPU; the comparator waited after 195 and 384 observations, reporting zero to ten milliseconds CPU. Empty, one-item and 63-item controls waited in both arms. These are local diagnostics with counting overhead, not production savings or a zero-CPU guarantee.

CPU-profile receipt call_A4bHO8FXBvWZinSsmbFs4qVG separately runs the 64-record baseline for three seconds. The retained grouped summary attributes 89.31% of its 2.90 sampled CPU seconds to observe_operation_pointer frames. This costs the repeated classification within that capture, including its fixture setup; it does not measure production prevalence.

The provisional first Case proof should reproduce the page-boundary discontinuity and show that an unchanged delayed queue reaches a bounded wait. It must also preserve deadline wakeups, shutdown, new-work notifications, task joins, fair ready-work discovery and lost-claim retries, including work arriving behind a scan cursor. The small comparator and checker layout are provisional, not a requirement to ship that exact edit. Reducing per-observation copies cannot stop this independent repetition.

Current open-PR implementation coverage showed no worker idle-accounting repair. Merged [PR #1297](https://github.com/HelixDB/helix-db/pull/1297) deliberately counts lost claims as work to prevent an unrelated long idle; that behavior must remain intact. Its inspected implementation retains the full-page/wrapping decisions described here. The PR's reported 122-index tenant batch establishes a relevant queue-cardinality shape, not the frequency of simultaneous backoff. Local history is shallow. GitHub is the documented review venue; the bounded public Discord read was refused with HTTP 403, leaving discussion coverage unavailable.

Change to test: Treat completion of a no-dispatch queue sweep as idle and wait on the existing wake, task-join and deadline sources instead of continually counting scan progress as work. Preserve bounded pages, family fairness, prompt new-work discovery and the existing claim-conflict rescheduling behavior.

Where it lives

perfloop-oss/helix-db · crates/db/src/index_lifecycle/worker.rs

Evidence

The check fails before this change and passes with it: `With automatic lanes, available permits and no publication work, 64, 65 and 128 operations backed off by retryable text-store HEAD failures reach a wait within 30 further page scans once all retries are delayed, without running an early driver step.`

Automatic delayed queue (64 operations) · 9 sample pairs

metric baseline candidate paired median change confidence range required result
cpu_ns 2120000000 340000000 −84.4% (−1790000000) −1870000000 to −1730000000 gain > 0; gain > 0.05 × |baseline median|; arm medians separated PASSED

Automatic delayed queue (65 operations) · 11 sample pairs

metric baseline candidate paired median change confidence range required result
cpu_ns 2110000000 390000000 −81.5% (−1720000000) −1830000000 to −1570000000 gain > 0; gain > 0.05 × |baseline median|; arm medians separated PASSED

Automatic delayed queue (128 operations) · 12 sample pairs

metric baseline candidate paired median change confidence range required result
cpu_ns 2110000000 410000000 −81.3% (−1715000000) −1840000000 to −1620000000 gain > 0; gain > 0.05 × |baseline median|; arm medians separated PASSED

Automatic ready index builds · 13 sample pairs

metric baseline candidate paired median change confidence range required result
completion_ns 27200000000 27100000000 0 −800000000 to +400000000 loss ≤ 0.05 × |baseline median| PASSED

Empty automatic worker wakes · 45 sample pairs

metric baseline candidate paired median change confidence range required result
elapsed_ns 560723225 544657418 −9156442 −52938511 to +26040084 loss ≤ 0.05 × |baseline median| PASSED

Checks: 4 of 4 passed. Verification: no defect found.

Timeline