Large package pushes retain whole-part buffers instead of bounded streams
perfloop-oss/cloudsmith-cli · Avoidable Copy
https://perfloop.ai/t/oss/case_s1v32njqfg
Verdict
VALIDATED BY A DIFFERENT CHANGE · settled 2026-11-16
What happened: The paired measurements met the required improvement.
Hypothesis
Large package pushes allocate a complete 100 MiB part before sending it. The previous response can retain an earlier request body while the next part is read. This can make temporary heap use much larger than the buffers needed for incremental transfer. The proposed benefit is lower memory use, not a demonstrated upload speedup.
At revision fb5b8068a5e8637e558fef15769247c2ad02424d, the installed CLI reaches push_handler, upload_files_and_create_package, cli.commands.push.upload_file, and this helper. push.py:506-508 automatically selects multipart for files at least CHUNK_SIZE, which is 104857600 bytes. Both output branches reach this helper. For an unchanged regular file of S bytes, a successful transfer materializes one payload byte string per nonempty part and sends ceil(S/CHUNK_SIZE) parts before completion. Early failures need not reach that count or completion. Requests retains the bytes as PreparedRequest.body, and the previous resp remains live while the next read and PUT are evaluated.
This is an ordinary supported large-file push route, not a retry-only path. README.md:521-533 documents push raw, and CHANGELOG.md:199 specifically records a repair for uploads larger than 100 MB using SSO or OIDC. The smallest eligible files already require a 100 MiB materialization by source. How frequently users push this size class is unknown. The internal change needs no new command, option, public API, or wire protocol.
The reproducible baseline is the receipt's retained perfloop_multipart_probe.py, run after uv sync --locked --no-dev. It invokes the registered push raw command with JSON output and the documented no-wait-for-sync option. SDK control replies are stubbed; checksum calculation, multipart selection, HTTP part transfer, completion ordering, and package creation sequencing execute. Receipt call_HK5pYQ4Bb2vNLNV4LHBMdvfi contains its cProfile capture; that capture is wall accounting, not an allocation profile.
An exact-script rerun with tracemalloc on Python 3.13.5, requests 2.34.2, and urllib3 2.8.0 measured 134217794 live bytes attributed to files.py:115 during the second part of the 128 MiB push, and 209715266 bytes during the second part of the 256 MiB push. These are two live payload objects totaling 128 and 200 MiB, plus object overhead. Both commands exited zero and verified total bytes, whole-file checksum, ordered part numbers, and exactly one completion before package creation. By comparison, the unchanged checksum pass uses 1 MiB blocks in core/utils.py:106-122. The expected removed footprint is the whole-part payload buffers, not upload traffic or the checksum pass. RSS reduction and elapsed-time effects remain unmeasured.
The required Case proof is a materially lower peak live heap for the registered large-file push while preserving the same bytes, wire part boundaries, authentication, callbacks, retry replay, error reporting, and completion-after-all-parts behavior. It should measure process RSS separately and inject a failed part to ensure completion is not attempted. The recorded sizes are optional reproducible samples. Subthreshold files and exact part-size multiples are regression controls. Reject the memory claim if the full-part allocations are already avoided or the consumer operation's peak heap does not materially fall. Confidence is medium.
Change to test: Stream each existing part from a bounded, seekable view of the open file instead of materializing a whole-part bytes object. Keep one PUT in flight and preserve content lengths, part boundaries, credentials, replay on retry or redirect, per-success callbacks, failure propagation, file cleanup, and completion only after every part succeeds.
Where it lives
perfloop-oss/cloudsmith-cli · cloudsmith_cli/cli/commands/push.py
Evidence
registered raw push, 128 MiB with update check due · 14 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
peak_traced_heap_bytes |
258157960 |
153436462 |
−40.6% (−104721269) |
−104723266 to −104719285 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
Checks: 3 of 3 passed. Verification: no defect found.
Timeline
2026-10-21· Case opened2026-11-01· PR opened