norm Writer and Reader insert a U+034F joiner into already-normalized text when each 3-byte chunk splits a combining mark

perfloop-oss/text · Derived summary drift

https://perfloop.ai/t/oss/case_dw4vcx1tg8

Verdict

VERIFIED · settled 2026-11-11 · pull request opened as golang/text#79

What happened: The check fails before this change and passes with it.

Hypothesis

Streaming normalization is reported to add an unwanted character to text that is already normalized. A combining grapheme joiner (U+034F) normally bounds long runs of combining marks; the reported insertion instead changes the caller's text without an error. Exposure among callers receiving very small stream pieces remains unknown.

The submitter's smallest found input is 16 repeats of "\u0634\u064e\u0651": ARABIC LETTER SHEEN, FATHA, and SHADDA. They report 96 input bytes, already in both NFC and NFD. Writing this input to norm.NFC.Writer in three-byte Write calls, or passing it through norm.NFC.Reader with an underlying source returning three bytes per Read, reportedly produces 98 bytes. A single U+034F appears before the final two marks, leaving the tail "\u0634\u034f\u064e\u0651". The expected result is the original 96 bytes, which the submitter reports norm.NFC.String returns. The Case must independently check these reported results and the smallest-found qualification.

The input has at most two consecutive non-starters, below the limit of 30 in UAX #15 Stream-Safe Text Format; the implementation also sets maxNonStarters to 30 (unicode/norm/composition.go:9-18). The submitter describes the narrow trigger as "every chunk must begin inside a combining mark and hold no later starter". They report that, for these six-byte clusters, only chunk size 3 failed among sizes 1 through 300, while one large Write was correct. The exact chunk-phase interpretation and reported exclusivity need checking rather than being generalized to arbitrary small reads or writes.

The public contracts say Writer writes f(b) and Reader returns f(data) (unicode/norm/readwriter.go:62-70 and 117-125). The result should therefore not acquire a character merely because the bytes were split. Writer and Reader retain a reorderBuffer across calls and feed it to doAppend (readwriter.go:9-49 and 72-115). doAppend patches a split UTF-8 tail before its quick-copy branch (unicode/norm/normalize.go:216-249). At the submitted locality, appendQuick copies an already-normalized span into rb.out without updating rb.ss (normalize.go:182-189). quickSpan uses its own local streamSafe counter (304-355), while patchTail subsequently advances rb.ss and inserts a joiner on overflow (149-180; composition.go:33-65). The submitted mechanism is that this retained counter misses copied starters and later completed marks accumulate across clusters until a false overflow occurs. Counter evolution at this exact copy boundary remains the mechanism to verify.

Comparison with the newly accepted iterator report is source-distinct. That report concerns NFD/NFKD Iter.Next on adjacent clusters with out-of-order marks and targets the ccc-0 break in doNormDecomposed (unicode/norm/iter.go:341-359). Its proposed starter reset is inside the iterator loop. The Reader/Writer route here instead calls doAppend and appendQuick directly (readwriter.go:30,104; normalize.go:247), with already-ordered NFC input and three-byte chunk boundaries. Repairing the iterator's break does not maintain the counter across these streaming quick copies. Both streaming APIs and the submitted appendQuick locality therefore remain part of this report, rather than being replaced by the iterator-only repair.

The comparison now also includes the accepted NFKC compatibility-letter boundary-query report. Its owner, Properties.BoundaryBefore (unicode/norm/forminfo.go:96-106), checks rune properties and does not maintain rb.ss. That report requests correcting the exported boundary answer and conditionally removing doAppend's extra leading-non-starter guard. This report instead concerns incorrect NFC text output despite the existing guard at normalize.go:231, with already-normalized Arabic clusters. A boundary-predicate correction or conditional guard cleanup is not the requested stream-counter repair and does not replace either streaming regression test.

The package's stream-test buffer sizes include 1 through 8 (unicode/norm/readwriter_test.go:13). Its Reader helper changes the consumer's output-buffer size, not the underlying source's read size (15-27), so the requested Reader regression must actually deliver input in three-byte pieces; the Writer helper already splits Write input (36-49). The smallest useful next check is a focused package test reconstructing the reported clusters, checking that NFC.String and NFD.String leave the input unchanged, then comparing complete Writer output after Close and complete Reader output against NFC.String. Check the reported base lengths and inserted code point, rerun the 1-through-300 chunk-size controls and one-large-Write control, and inspect the counter around quick copying and tail completion. This diagnostic arrangement is provisional, not an additional required test layout. The requested repository regression coverage remains both Reader and Writer with repeated two-mark clusters at size 3.

Upstream review found no same open change in the complete code-host PR listing, its refreshed exact-path pages, or a bounded Gerrit query for open unicode/norm changes. The inspected patch in https://github.com/golang/text/pull/58 adds decomposition-table bounds checks, not cross-chunk counter maintenance. Recent merged implementations https://go-review.googlesource.com/c/text/+/794100 and https://go-review.googlesource.com/c/text/+/794101 change invalid-character handling and decomposeSegment progress; they do not maintain the counter in appendQuick. The older counting repair https://go-review.googlesource.com/c/text/+/40454 preserves accounting around multi-segment decompositions and flushing, a distinct concern that the new repair must not undo. Local Git history is shallow, so the bounded Gerrit implementation reads supplement it rather than establishing exhaustive historical absence.

Change to test: Keep the reorderBuffer's stream-safe counter correct across chunks: when appendQuick copies a quick span, set the counter from the last copied rune, or reset it at the start of each doAppend before patchTail counts. Add native Reader and Writer regression tests with repeated two-mark clusters and three-byte input chunks, so splitting the stream does not add a joiner to already-normalized text.

Where it lives

perfloop-oss/text · unicode/norm/readwriter.go

Evidence

The check fails before this change and passes with it: `NFC.Reader returns sixteen repetitions of U+0634 U+064E U+0651 unchanged when its error-free source supplies three-byte pieces and the output is read through EOF.`

The check fails before this change and passes with it: `NFC.Writer preserves sixteen repetitions of U+0634 U+064E U+0651 written in three-byte pieces to an error-free output buffer, including buffered output emitted by Close.`

Checks: 1 of 1 passed. Verification: no defect found.

Timeline