Keep successful upgrades from opening the backend circuit breaker

perfloop-oss/skipper · Value Kind Confusion

https://perfloop.ai/t/oss/case_vq976y36j2

Verdict

VERIFIED · settled 2026-11-15 · pull request opened as zalando/skipper#4438

What happened: The check fails before this change and passes with it.

Hypothesis

A cleanly completed upgrade can open the backend's circuit breaker even though the backend returned 101 and exchanged data successfully. Later requests to that healthy backend then receive 503 without being sent upstream. This requires ExperimentalUpgrade and an applicable circuit breaker; their production use together is unknown.

The experimental branch of makeBackendRequest returns a non-nil proxyError with handled=true after makeUpgradeRequest returns, including a successful session. The type's own comment distinguishes that ownership marker from an actual proxy error. Proxy.do instead treats every non-nil result as a backend failure at lines 1508–1514: it calls the breaker's done(false) and increments the backend-error counter before handling the already-served response. The native check below observes the breaker effect, not the metric count. Active sessions have not returned through this branch yet; the claim concerns completed calls.

circuit/doc.go:12–18 and 132–140 and docs/reference/filters.md:2755–2764 define failures as connection failures or backend statuses of at least 500. A normal 101 is neither. In an initially closed consecutive breaker, successive completed upgrades to the same breaker key can therefore supply the erroneous failure count, absent resets or other traffic.

The provisional new native file proxy/hunt_upgrade_breaker_test.go ran from the repository root with `go test ./proxy -run '^TestHuntUpgradeBreaker(Replay|SuccessSequences)$' -count=1 -timeout=30s -v`. The reduced replay performs one valid WebSocket handshake and echoed message, waits for normal peer closure and handler completion, then sends one healthy HTTP GET with a threshold-one breaker. It failed with `healthy backend rejected after 1 successful upgrades (threshold 1): status=503 X-Circuit-Open="true" body="Service Unavailable\n"; want 200`. The bounded sequence check ran all nine combinations of thresholds one through three and zero through threshold completed successful upgrades before the GET. The GET was rejected when the success count reached each threshold. The shorter prefixes passed. Operations had one-second deadlines. The scratch test was removed after the run.

The Case proof target is to report the actual upgrade/backend outcome rather than confuse an already-handled response with failure, while retaining breaker accounting for genuine connection errors and 5xx results. The smallest useful next check is the two-operation replay plus an actual-failure control. Test layout is provisional. Simply treating every handled marker as success would hide real failures from the same upgrade handler, so outcome and connection-ownership bookkeeping need to remain distinct. This entails internal result plumbing but need not add a public option.

Concrete prior searches for circuit/upgrade and handled/circuit found no matching repair. The actual patches in https://github.com/zalando/skipper/pull/4424, https://github.com/zalando/skipper/pull/4425 and https://github.com/zalando/skipper/pull/4427 do not repair this classification. In particular #4424 excludes upgrade requests from hedging and retains done(false) for the ordinary non-nil error result; its new guard only avoids double-counting a primary hedging error. The other patches change round-trip hedging and adaptive delay. Recent merged history remains a gap because the checkout is shallow and the bounded history reads were refused. The separate proposal Drive archive was refused as well. Repository rules require a human-authored issue and PR description for any eventual contribution.

Change to test: Separate an already-handled upgrade response or session from the backend success/failure result before updating circuit breakers and backend-error metrics. Count successful upgrades according to the documented backend policy while preserving accounting for real connection failures and 5xx responses, including failures handled inside the upgrade path.

Where it lives

perfloop-oss/skipper · proxy/proxy.go

Evidence

The check fails before this change and passes with it: `With ExperimentalUpgrade enabled and an initially closed threshold-one consecutive circuit breaker, a completed WebSocket upgrade that echoes data leaves the next healthy HTTP request to the same backend permitted and does not increment backend-error accounting.`

Checks: 4 of 4 passed. Verification: no defect found.

Timeline