Overlap independent root comparisons across hosts

perfloop/weaviate · SERIAL WORK

https://perfloop.ai/t/oss/case_fyw4ea4gt7

Verdict

CLOSED · opened 2026-08-12

What happened: Closed; the assigned hypothesis was retired.

Hypothesis

I ran an isolated two-peer mock-RPC probe against `PrefilterShardRoots`. Each `CompareHashTreeRoots` call slept for 75 ms, and the prefilter completed in 151.6 ms for two peers. This confirms serial waiting in a controlled test, not production peer latency.

The source builds `byHost` before issuing requests, then flushes chunks from nested host and shard loops. `flush` synchronously calls `CompareHashTreeRoots`; the REST path reaches `http.Client.Do` and the gRPC path makes a unary call. Each request already has its host and root chunk, while results only union diverging shards and increment per-call status counters. Errors force full descent only for the failed chunk's shards. These facts make the waits structurally independent, although safe production concurrency and its benefit remain hypotheses. `runBatch` invokes this prefilter for each asynchronous reconciliation batch with more than one entry before it chooses skip or descent, so every extra serialized wait holds that batch.

A case session should trace root-compare spans and `runBatch` duration for batches that produce at least two remote host/chunk requests. It should show that baseline spans do not overlap, then compare bounded overlap while verifying identical `needFull` and stats outcomes and unchanged RPC count. The success signal is prefilter and batch wall time moving from the sum of comparable RPC waits toward their maximum; with two similarly slow requests, that removes roughly one remote wait.

Change to test: Create immutable root chunks and run independent host comparisons through a bounded concurrency group. Merge divergence sets and outcome counters after completion so each failed or unsupported chunk still forces only its own shards to full descent.

Where it lives

perfloop/weaviate · adapters/repos/db/async_replication_scheduler.go

Evidence

Timeline