Bulk-build duplicate-free JSON objects

perfloop/protobuf-py · INEFFICIENT ALGORITHM

https://perfloop.ai/t/oss/case_pd2a2asnmv

Verdict

VERIFIED · settled 2026-07-22

What happened: Validated and review-passed. On the sealed public nested duplicate-free 1,000-member ASCII `Maps.stringToString` `Maps.from_json` component selector, ns/op fell from 893,362.83 to 363,796.30 (59.28% lower). The one-member ASCII guard held, and the added non-ASCII string-map and int32-map destination guards held; they are regression bounds, not claimed wins. All duplicate-rejection allocation/process-cap checks held, and the new baseline/candidate public structured-value validation check passed. The claim remains limited to the sealed ASCII 1,000-member component selector, not all map kinds or whole-parser performance.

Hypothesis

`merge_from_json` passes `_no_duplicates` as `json.loads(..., object_pairs_hook=...)`, then sends the resulting dictionary to `_read_message`, which checks for a dict and iterates `json.items()`. The current target performs `k in obj` followed by `obj[k] = v` for every pair, so each valid member takes the explicit membership lookup plus Python-level loop work before the dictionary consumer can run. The hook cadence is per parsed object rather than merely per document: a focused nested-JSON check invoked it twice for an inner one-member object and its outer two-member object. Thus the trigger is each duplicate-free object, including nested objects, with the predicted benefit conditional on objects having 100–1,000 members; source does not establish how often that shape occurs in the workload.

A focused CPython check compared the current target to the guarded `dict(pairs)` fast path. It preserved output and the same `ValueError('duplicate key: a')` for representative valid lists and duplicate-at-second/third lists. Direct target timings were 25.4% faster at 10 pairs, 46.6% at 100, and 33.7% at 1,000, while it regressed 22.4% at one pair. A separate `json.loads`-with-hook check showed total parse changes of -1.7% at 10 members, +16.2% at 100, and +7.0% at 1,000; these are local measurements, not evidence of production end-to-end impact. A case session should benchmark `Message.from_json` with flat and nested duplicate-free documents spanning 1, 10, 100, and 1,000 members, measure end-to-end CPU/latency plus `_no_duplicates` exclusive CPU, and differentially test duplicate errors. The confirming signal is roughly a 35% reduction in this symbol's own cost on the 100–1,000-member successful-object path without changing the returned mapping or first duplicate-key error; small-object crossover and whole-message impact must be reported separately.

Change to test: Use a valid-object fast path that constructs `obj = dict(pairs)` and returns it when its length equals `len(pairs)`; on a length mismatch, rescan `pairs` with a seen-key set solely to raise the current first-duplicate `ValueError` and preserve its key text.

Where it lives

perfloop/protobuf-py · src/protobuf/_message.py

Evidence

Timeline