Stop lookup failures from sweeping retained image content
perfloop-oss/distribution · Ignored Failure
https://perfloop.ai/t/oss/case_j01t3vsp92
Verdict
VERIFIED · settled 2026-11-13 · pull request opened as distribution/distribution#5083
What happened: The check fails before this change and passes with it.
Hypothesis
Garbage collection can delete an image configuration still reachable from a tagged index after a temporary storage error. The reproduced failure uses --delete-untagged with no concurrent writes. A bounded native check demonstrates data loss; the frequency of this failure with real storage backends is unknown.
markManifestReferences marks a descriptor's digest and discards the error returned by manifestService.Exists before deciding whether to visit its descendants. MarkAndSweep skips untagged manifest payloads during initial enumeration when RemoveUntagged is true. Filtering the child's marked digest out of the manifest-deletion candidates does not protect its unvisited configuration or layers. Other unexpected marking errors already abort before sweep. The preservation contract is stated in docs/content/about/garbage-collection.md:73-87 and exercised without injected errors by TestTaggedManifestlistWithUntaggedManifest.
The stored check on prop_d0tta90j10 ran at revision 64bbe136631b9ed568c7415c5e6eaef6c79697f5 with `go test -count=1 -v -run=^TestPerfloopGCReachabilityFaultSchedules$ ./registry/storage`. It exhausts five schedules on fresh real in-memory stores: no fault, or one error at each of the first four matching child-data Stat calls. Each fixture has one tagged OCI index, one untagged image, a two-byte configuration, and no layers. The ordinal-1 schedule returned nil from GC, issued two Delete calls, and left the configuration unreadable with unknown blob. Ordinals 2 and 3 returned marking errors and issued no deletes. The healthy control and untriggered ordinal-4 schedule preserved all three blobs. The test's exit code 1 is the expected preservation-assertion failure, not a build failure. The retained deterministic ordinal-1 subtest is a reduced replay.
The larger class-removal alternative of repository-local traversal marks is already actively implemented by https://github.com/distribution/distribution/pull/4884 and is excluded from this proposal. Its checked implementation and the two-pass GC rewrite at https://github.com/distribution/distribution/pull/4795 both leave this Exists error discard unchanged. Those are open implementations, not an accepted replacement decision. Older GC proposals inspected here change enumeration order, age thresholds, or storage layout rather than this error boundary.
The smallest useful Case check is to rerun the retained fault test against a minimal error-boundary repair, requiring the failing lookup to prevent destructive work without changing healthy completion or ordinary missing-reference handling. Wider graph shapes, Docker manifest lists, and backend-specific error types are remaining assurance work, not claims established by this finite OCI fixture.
Change to test: Treat an indeterminate manifest-existence result as a marking failure so destructive sweep cannot follow an incomplete traversal, while preserving existing handling of a known missing reference. Establish retained-content safety with reproducible fault coverage; the native checker and fixture layout used here are provisional, not extra repository requirements.
Where it lives
perfloop-oss/distribution · registry/root.go
Evidence
The check fails before this change and passes with it: `For a tagged OCI index referencing one untagged OCI image with a two-byte configuration and no layers in an in-memory registry with no concurrent writes, a one-shot child-data Stat failure during the manifest-existence lookup makes MarkAndSweep with RemoveUntagged return an error before any Delete call and preserve all stored blob bytes.`
Checks: 2 of 2 passed. Verification: no defect found.
Timeline
2026-11-12· Case opened2026-11-12· PR opened