Concurrent Manifest Dependency Verification

perfloop/distribution · SERIAL WORK

https://perfloop.ai/t/oss/case_zptzacr9b7

Verdict

VERIFIED · settled 2026-07-09 · pull request opened as distribution/distribution#4906

What happened: The parallelized manifest verification implementation has been fully validated on both Docker Schema 2 and OCI manifest handlers under concurrent benchmarks with injected driver latency. Under the schema2 benchmark (verifyManifest_latency), the median latency was reduced from 128.7ms to 21.5ms (an 83.33% reduction, p-value < 0.0001). Under the OCI manifest benchmark (verifyManifest_oci_latency), which serves as a co-measured guardrail, the median latency was reduced from 128.0ms to 21.4ms (an 83.30% reduction). The candidate latency of ~21ms is exactly double the simulated 10ms single-step latency limit because the manifest service Put operation consists of two sequential phases: first, verifyManifest executes all descriptor presence checks in parallel (taking max(10ms) = 10ms), and second, the manifest itself is written to the blob store using ms.blobStore.Put, incurring an additional sequential 10ms driver Stat check. This mathematically confirms the sum-to-max transformation.

Hypothesis

The verifyManifest method validates each descriptor referenced by the schema2 manifest in a sequential loop. It performs independent I/O-bound storage existence checks (using manifestService.Exists and blobsService.Stat) for each layer or manifest. Since these checks are completely independent of one another and block on downstream storage I/O, serial execution forces the total verification latency to scale linearly with the number of references (O(N) * latency of a single check). Under the Upload Image Manifest workload, parallelizing these checks using golang.org/x/sync/errgroup allows the I/O-bound operations to overlap, reducing latency from the sum of the check durations to the maximum of those durations, significantly improving manifest put performance.

Change to test: Parallelize the independent descriptor verification checks in verifyManifest using a concurrency primitive like golang.org/x/sync/errgroup.

Where it lives

perfloop/distribution · registry/handlers/manifests.go

Evidence

Timeline