Finalize failed rollbacks so later upgrades are not locked out
perfloop-oss/helm · Uncatalogued correctness
https://perfloop.ai/t/oss/case_cj9f882jcr
Verdict
VERIFIED · settled 2026-11-08 · pull request opened as helm/helm#32871
What happened: The check fails before this change and passes with it.
Hypothesis
Rollback can return a hook error while its newly stored revision still says the operation is running. A subsequent ordinary upgrade then refuses to proceed, even though the rollback has ended. This also occurs at other execution-error exits. The frequency of these failures in customer deployments is unmeasured.
Run creates targetRelease before calling performRollback, then returns immediately on its error. Its final Releases.Update is success-only. performRollback records StatusFailed for Update, Wait, and WaitWithJobs errors, but directly returns errors from manifest building, pre-rollback hooks, waiter acquisition, post-rollback hooks, and the deployed-release query. StatusPendingRollback is documented as an operation underway. Upgrade.prepareUpgrade treats any latest pending revision as a lock. This finding concerns completed error returns after successful revision creation, not pre-creation exits, dry runs, crashes, or failed storage writes.
The native property prop_ffx2hmsmcp failed in checkProperty run call_lJlVXTZVPpNSX1mSPpsN7vw4 and reproduced from its retained inputs in call_wcKTv8K1ioY4KJ7WUHGPxqEP. Run `go test ./pkg/action -run '^TestPerfloopRollbackTerminalStateProperty$' -count=1 -timeout=45s -v` with the stored files. The fourteen-case matrix starts with two revisions and requests one public rollback to revision one. Eight of eleven single-fault execution schedules left revision 3 pending and made a subsequent public client-dry-run upgrade return errPending. The eight schedules cover both manifest builds, hook creation and watching before and after the update, waiter acquisition, and the final deployed-release query. Update and both readiness-error controls persisted failed status and allowed the successor. Success and both dry-run controls passed. Release records use the real serializing Secret driver against a fake API, rather than relying on memory-driver pointer aliasing.
At head 916eda6abd45d01b15f33e201ea03a008e47f0ec, https://github.com/helm/helm/pull/32314 adds failRollback for its new sequenced path but leaves the ordinary rollback's direct error exits and Run's success-only final write unchanged. That is useful adjacent implementation, not coverage of this remaining path. https://github.com/helm/helm/pull/32850 changes hook deletion waiting, not the release's terminal state after a hook error. https://github.com/helm/helm/pull/32585 instead introduces an opt-in override during a later upgrade. In https://github.com/helm/helm/issues/32584#issuecomment-5670349134, scottrigby directs crash-recovery and lock-override design to the HIP discussion. This proposal neither overrides an active lock nor adds recovery policy.
The first Case check should replay the pre-hook failure through Run, read the persisted revision after return, and attempt an ordinary upgrade. Confirm that the original error is retained, the new revision is failed, and the pending-state guard no longer blocks that successor. Then cover the other post-creation error boundaries while retaining success, update/readiness, and dry-run controls. The stored native matrix and its placement are provisional. Centralizing finalization must not accidentally supersede an untouched deployment or disturb existing cleanup behavior. Cross-write atomic recovery and production incidence remain outside what this witness establishes.
Change to test: Give ordinary rollback failure finalization one owner so every execution error after revision creation records a failed terminal status when storage is writable. Preserve the original execution error, prior-revision status semantics, and dry-run behavior without adding a lock-override flag.
Where it lives
perfloop-oss/helm · pkg/action/rollback.go
Evidence
The check fails before this change and passes with it: `With writable Secret storage, Rollback.Run returns its execution error and persists the new revision as failed before returning from manifest-build, hook-create/watch, waiter-acquisition, update, readiness, or deployed-release-query failures, so a subsequent ordinary upgrade is not rejected by the pending-state guard.`
Rollback manifest-build failure · 13 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
2457322 |
4994752 |
+105.2% (+2584631) |
+2300233 to +2909329 |
loss ≤ stated 1.5 × |baseline median| |
PASSED |
B/op |
2636446 |
5028060 |
+90.7% (+2392143) |
+2385497 to +2393604 |
loss ≤ stated 1.5 × |baseline median| |
PASSED |
allocs/op |
3645 |
7016 |
+92.8% (+3383) |
+3358 to +3392 |
loss ≤ stated 1.5 × |baseline median| |
PASSED |
Rollback pre-hook creation failure · 14 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
5023148 |
7486263 |
+50.3% (+2528437) |
+2263557 to +2742786 |
loss ≤ stated 0.8 × |baseline median| |
PASSED |
B/op |
5039178 |
7424519 |
+47.3% (+2385620) |
+2381060 to +2390209 |
loss ≤ stated 0.8 × |baseline median| |
PASSED |
allocs/op |
7104 |
10437 |
+46.9% (+3335) |
+3317 to +3348 |
loss ≤ stated 0.8 × |baseline median| |
PASSED |
Rollback template-free manifest-build failure · 14 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
2392082 |
4892681 |
+102.7% (+2456730) |
+2267513 to +2783408 |
loss ≤ stated 1.5 × |baseline median| |
PASSED |
B/op |
2603323 |
4983843 |
+91.5% (+2380919) |
+2377281 to +2382675 |
loss ≤ stated 1.5 × |baseline median| |
PASSED |
allocs/op |
3480 |
6845 |
+96.8% (+3367) |
+3356 to +3373 |
loss ≤ stated 1.5 × |baseline median| |
PASSED |
Rollback template-free pre-hook creation failure · 15 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
4921194 |
7233122 |
+49% (+2409173) |
+1848320 to +2927402 |
loss ≤ stated 0.8 × |baseline median| |
PASSED |
B/op |
4997147 |
7371430 |
+47.5% (+2373986) |
+2370713 to +2380243 |
loss ≤ stated 0.8 × |baseline median| |
PASSED |
allocs/op |
6927 |
10247 |
+47.9% (+3320) |
+3306 to +3345 |
loss ≤ stated 0.8 × |baseline median| |
PASSED |
Checks: 4 of 4 passed. Verification: no defect found.
Timeline
2026-11-08· Case opened2026-11-08· PR opened