Full delayed queues keep the worker scanning instead of waiting
perfloop-oss/helix-db · Polling Loop
https://perfloop.ai/t/oss/case_xzkn0dkpyh
Verdict
VERIFIED · settled 2026-11-15 · pull request opened as HelixDB/helix-db#1301
What happened: The check fails before this change and passes with it.
Hypothesis
Automatic scheduling repeatedly resolves a full queue even when every operation is delayed. Finishing the scan still counts as progress, so the dispatcher never reaches its existing wait path. A native fixture confirms sustained CPU use at the 64-item page boundary. How often ordinary writers retain that many simultaneously delayed operations remains unknown; confidence is medium.
At revision 4b0982f, schedule_operation_task returns Continue after a full page. When a later page exhausts the prefix with a cursor present, it resets the cursor but also returns Continue because wrapping is true. dispatch_worker_cycle resets lanes_without_work on Continue and waits only after four idle lane attempts. With stable future-deadline pointers spanning at least one full 64-item page, available permits, automatic lanes and no other task completions, the cyclic scans never let that idle count reach four. Each pass calls observe_operation_pointer again, performing its dependent reads and validation without making an operation eligible.
This is not the bounded recovery-sweep timer running expensively. The timer is never reached in this state. The normal runtime installs automatic vector/text scheduling and enabled secondary scheduling. release_transient_claim persists same-writer failure deadlines using backoff_millis, bounded from one to thirty seconds. The observer honors those deadlines. Thus delayed records have a production source, although the measured queue was directly seeded rather than generated by a real failure incident.
Measurement receipt call_oi4H4SGNofguPLc1wBcS8OO6, output:1, records `python3 .perfloop_worker_measure.py`, including `cargo run --release -p db --features bench-internals --example perfloop_worker_probe -- compare 64 500`. It drives the actual run_worker_cycle with three automatic lanes, real in-memory SlateDB, no publication work, valid secondary Activate records and same-epoch thirty-second transient backoff. Temporary counters measure scans, observations and wait entries. An external shutdown thread bounds each run; no driver step may execute, and every sampled deadline remained in the future.
For 64 records, the baseline made 1,112 scans and 35,648 observations with zero wait entries in 501.313 ms. Process CPU accounting reported 490 ms. A minimal comparator changed only end-of-sweep reporting: six scans, 192 observations and one wait entry, with CPU reported below the counter's resolution. At 65 and 128 records, the baseline again had zero waits and about 480 ms CPU; the comparator waited after 195 and 384 observations, reporting zero to ten milliseconds CPU. Empty, one-item and 63-item controls waited in both arms. These are local diagnostics with counting overhead, not production savings or a zero-CPU guarantee.
CPU-profile receipt call_A4bHO8FXBvWZinSsmbFs4qVG separately runs the 64-record baseline for three seconds. The retained grouped summary attributes 89.31% of its 2.90 sampled CPU seconds to observe_operation_pointer frames. This costs the repeated classification within that capture, including its fixture setup; it does not measure production prevalence.
The provisional first Case proof should reproduce the page-boundary discontinuity and show that an unchanged delayed queue reaches a bounded wait. It must also preserve deadline wakeups, shutdown, new-work notifications, task joins, fair ready-work discovery and lost-claim retries, including work arriving behind a scan cursor. The small comparator and checker layout are provisional, not a requirement to ship that exact edit. Reducing per-observation copies cannot stop this independent repetition.
Current open-PR implementation coverage showed no worker idle-accounting repair. Merged [PR #1297](https://github.com/HelixDB/helix-db/pull/1297) deliberately counts lost claims as work to prevent an unrelated long idle; that behavior must remain intact. Its inspected implementation retains the full-page/wrapping decisions described here. The PR's reported 122-index tenant batch establishes a relevant queue-cardinality shape, not the frequency of simultaneous backoff. Local history is shallow. GitHub is the documented review venue; the bounded public Discord read was refused with HTTP 403, leaving discussion coverage unavailable.
Change to test: Treat completion of a no-dispatch queue sweep as idle and wait on the existing wake, task-join and deadline sources instead of continually counting scan progress as work. Preserve bounded pages, family fairness, prompt new-work discovery and the existing claim-conflict rescheduling behavior.
Where it lives
perfloop-oss/helix-db · crates/db/src/index_lifecycle/worker.rs
Evidence
The check fails before this change and passes with it: `With automatic lanes, available permits and no publication work, 64, 65 and 128 operations backed off by retryable text-store HEAD failures reach a wait within 30 further page scans once all retries are delayed, without running an early driver step.`
Automatic delayed queue (64 operations) · 9 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
cpu_ns |
2120000000 |
340000000 |
−84.4% (−1790000000) |
−1870000000 to −1730000000 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
Automatic delayed queue (65 operations) · 11 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
cpu_ns |
2110000000 |
390000000 |
−81.5% (−1720000000) |
−1830000000 to −1570000000 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
Automatic delayed queue (128 operations) · 12 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
cpu_ns |
2110000000 |
410000000 |
−81.3% (−1715000000) |
−1840000000 to −1620000000 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
Automatic ready index builds · 13 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
completion_ns |
27200000000 |
27100000000 |
0 |
−800000000 to +400000000 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Empty automatic worker wakes · 45 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
elapsed_ns |
560723225 |
544657418 |
−9156442 |
−52938511 to +26040084 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Checks: 4 of 4 passed. Verification: no defect found.
Timeline
2026-11-14· Case opened2026-11-15· PR opened