lz4c: truncate existing output files and never write into the input

perfloop-oss/lz4 · Uncatalogued correctness

https://perfloop.ai/t/oss/case_csvpxe8dah

Verdict

VERIFIED · settled 2026-11-11 · merged as pierrec/lz4#297

What happened: The check fails before this change and passes with it.

Hypothesis

A shorter restore can retain the old file's tail, and a destination that names the source can overwrite the only compressed copy. Compression uses the same output-open flags. The submitter reports corruption and source destruction; the examples and their prevalence remain unconfirmed in this run.

At the accepted revision 35976251339c01d649496a51e9de573693d42a7b, Uncompress is defined in cmd/lz4c/uncompress.go:20-124. Its ordinary named-file branch opens the input at lines 32-37, derives the destination with strings.TrimSuffix at line 71, and opens it with O_CREATE|O_WRONLY, without O_TRUNC, at line 72. Decoding writes to that destination at lines 104-108. A name without the .lz4 suffix therefore selects the input pathname itself. Compress, defined in cmd/lz4c/compress.go:31-189, appends the suffix at line 163 and uses the same non-truncating flags at line 164 before writing at lines 168-180. Neither handler checks input/output file identity before writing. These are documented CLI file operations (README.md:20-46), registered by main in cmd/lz4c/main.go:17-30; the scope here is the normal named-file branches, not benchmark mode or stdin-to-stdout operation.

The submitter reports a failure at HEAD 6a24197 and says the cited file is unchanged at upstream 3597625. Their command is `seq 1 2000 > data.txt; lz4c compress data.txt; seq 1 50000 > data.txt; lz4c uncompress data.txt.lz4`. They report exit status 0 and a 288,894-byte data.txt instead of the expected 8,893 bytes: decoded bytes followed by the old file's tail. They describe restoring any shorter version over a longer existing file as corrupted. They also report that compressing a 3-byte file over a 300,015-byte .lz4 leaves 300,015 bytes that no longer decode. These are attributed reports to reproduce at the Case base, not measurements established here.

For a copy of valid compressed data named backup, without a .lz4 suffix, the submitter reports that a 300,000-random-byte payload exits 0 and the file becomes the payload plus 15 old bytes. For 46.9 MB of text, they report exit status 1 with `lz4: invalid block size`, leaving 4 MB of payload followed by old compressed bytes and destroying the only copy. They report the same destructive behavior when the output name is a hard link or symlink to the input. Adding O_TRUNC alone would empty an aliased source before decoding, so the two repairs must stay together. Their control is a .lz4 input with no existing output: correct decoded bytes and an unchanged source.

The smallest useful first check is the reported shorter-restore command in a temporary directory, comparing the complete restored bytes, exact length, exit status, and compressed-input hash. The Case must check all the reported compression and self-alias examples and the distinct-output control, then establish exact-length output and error-before-mutation for identical-path, hard-link, and symlink destinations in both commands. The submitter explicitly requests CLI tests for shorter restores and non-.lz4 input names. Reusing TestMain and runCLI in cmd/lz4c/main_test.go:17-63 is provisional test placement, not an additional frozen requirement.

The submitter reports no matching upstream report among 24 lz4c issues and PRs and distinguishes issue 287 with PR 288 as the exit-status defect. This run read https://github.com/pierrec/lz4/issues/287 and the implementation in https://github.com/pierrec/lz4/pull/288, merged as 47027fde2524b6e9f86ec4e161716523bd254a27: it changes main.go error reporting and adds main_test.go, but does not change either output-open site. That fix is already present at the accepted revision; truncation and same-file refusal remain distinct work. The fresh open-PR inventory contains https://github.com/pierrec/lz4/pull/293 (writer.go and writer_test.go) and https://github.com/pierrec/lz4/pull/294 (CHANGELOG.md), neither changing these CLI sites. README.md:76-81 directs contributors to issues and pull requests with tests; no separate review venue was identified there. Recent file history remains an evidence gap: the local checkout is shallow, and bounded code-host history reads for the two handler files were refused with HTTP 403.

Change to test: Truncate the output in both compress and uncompress, but return an error without changing the input whenever the output path is the input or resolves to the same file, including hard links and symlinks. Add CLI tests for a shorter restore over a longer file and for a non-.lz4 input name.

Where it lives

perfloop-oss/lz4

Evidence

The check fails before this change and passes with it: `Named-file lz4c compress and uncompress replace longer distinct regular output files with exactly the new bytes without changing the input, including empty payloads.`

The check fails before this change and passes with it: `Named-file lz4c compress and uncompress return an error before changing input contents when the derived output names the input file, directly or via a hard link or symlink.`

The check fails before this change and passes with it: `Named-file lz4c uncompress with an existing distinct regular destination returns an error for an invalid header or truncated frame checksum, leaves no stale destination tail, and does not change the compressed input.`

CLI equal-size restoration · 9 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 5138896 5446893 +5.6% (+287743) +151340 to +814712 loss ≤ stated 0.3 × |baseline median| PASSED

CLI fresh-file restoration · 24 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 5373446 5374677 +2343 −242905 to +216029 loss ≤ 0.05 × |baseline median| PASSED

CLI file replacement and alias refusal · 12 sample pairs

metric baseline candidate paired median change confidence range required result
shorter-restore-output-bytes/op 288894 8893 −96.9% (−280001) −280001 to −280001 loss ≤ 0.05 × |baseline median|; base median within 288894 to +∞ PASSED
shorter-restore-expected-payload-bytes/op 8893 8893 0 0 to 0 loss ≤ 0.05 × |baseline median|; base median within 8893 to +∞ PASSED
overwrite-compress-input-bytes/op 3 3 0 0 to 0 loss ≤ 0.05 × |baseline median|; base median within 3 to +∞ PASSED
overwrite-compress-output-bytes/op 300015 18 −100% (−299997) −299997 to −299997 loss ≤ 0.05 × |baseline median|; base median within 300015 to +∞ PASSED
self-alias-random-payload-bytes/op 300000 300000 0 0 to 0 loss ≤ 0.05 × |baseline median|; base median within 300000 to +∞ PASSED
self-alias-random-stale-tail-bytes/op 15 0 −100% (−15) −15 to −15 gain > 0; gain > 0.05 × |baseline median|; arm medians separated; base median within 15 to +∞ PASSED
self-alias-text-input-MB/op 46.9 46.9 0 0 to 0 loss ≤ 0.05 × |baseline median|; base median within 46.85 to +∞ PASSED
self-alias-text-overwritten-payload-MB/op 4.194 0 −100% (−4.194) −4.194 to −4.194 gain > 0; gain > 0.05 × |baseline median|; arm medians separated; base median within 3.5 to +∞ PASSED

Checks: 2 of 2 passed. Verification: no defect found.

Timeline