Interim cross-cluster async search returns the wrong hit page

perfloop-oss/elasticsearch · UNCATALOGUED MECHANISM

https://perfloop.ai/t/oss/case_z73jpc17ge

Verdict

VERIFIED · settled 2026-10-03

What happened: The assertion is violated on the comparison and satisfied with this change.

Hypothesis

An in-progress cross-cluster async search can return the wrong page of hits. The coordinator widens the search sent to each cluster by changing a shared request source. An interim response then builds a new merger from that changed source rather than the user's requested page. This is a source-derived prediction; the visible result has not been measured in a running search.

At revision f7db0f8e, `TransportSearchAction.executeRequest` reaches `ccsRemoteReduce` for remote indices when `shouldMinimizeRoundtrips` is true. In its multi-participant branch, `ccsRemoteReduce` creates the final-response merger before registering a task supplier that calls `createSearchResponseMerger` again. That helper records the original `from` and `size` in the first merger, then changes the source to `from=0` and `size=from+size`; `SearchRequest.subSearchRequest` shares this source for the outgoing searches. `MutableSearchResponse.toAsyncSearchResponse` calls the supplier for a partial response and merges available cluster results. `SearchResponseMerger.getMergedResponse` uses the second merger's offset and length when it merges hits. For an original positive offset `f` and positive page size `s`, the first merger retains `(f,s)` but subsequent snapshot mergers read `(0,f+s)`.

The affected operation is `/_async_search` followed by an interim-result poll with `ccs_minimize_roundtrips=true`, a positive `from`, and at least two participating clusters, such as local plus remote, with one completed while another remains running. `CrossClusterAsyncSearchIT.testGetResultIntermediateResultsFalseOnRunningSearchDoesNotIncludeIntermediateResults` already exercises that configuration and returns interim hits, but uses the default offset. The single-remote/no-local fast path and a completed search do not use the erroneous interim merger; an empty interim result cannot show the page mismatch.

A Case can extend that integration test with stable, uniquely sorted documents, `from=3,size=2`, enough local matches, and a blocked remote cluster. Poll with intermediate results enabled after local completion and compare the exact hit IDs with a completed local-only search using the same page; then release the remote cluster and compare the completed two-cluster result with a completed two-cluster control. An interim page that already matches the requested local-only page under these conditions, or no interim hits when the local result has arrived, would refute the predicted visible failure. Production frequency and user exposure remain unknown.

Change to test: Separate the requested page from the wider window sent to each cluster. Preserve one immutable original `from` and `size` for both completed and interim mergers, while widening only the outgoing subsearches to `[0, from + size)`.

Where it lives

perfloop-oss/elasticsearch · server/src/main/java/org/elasticsearch/action/search/TransportSearchAction.java

Evidence

The assertion is violated on the comparison and satisfied with this change: `For /_async_search with one local and one remote cluster, ccs_minimize_roundtrips=true, from=3,size=2, eight uniquely ranked match-all documents per cluster, and local hits published while the remote remains running, an interim-results poll returns exactly the two local hits at offsets 3 and 4 in ascending rank order.`

Checks: 3 of 3 passed. Verification: no defect found.

Timeline