Borrow simple fused input instead of copying it

perfloop/basetenkenizer · AVOIDABLE COPY

https://perfloop.ai/t/oss/case_sramqj8th2

Verdict

PROPOSED · opened 2026-09-07

Hypothesis

For a borrowed normalized one-text segment, the current builder deliberately creates an owned `String` even though the eligible fused route subsequently only re-slices that buffer before raw BPE. A private, call-scoped borrowed view can avoid the full input copy while retaining the existing owned representation wherever text is transformed or boundaries require it.

Add route counters and an allocation profile for the proposed private branch, then differentially compare exact IDs and errors against the existing path for borrowed and owned normalization, added/special-token cases, and varied input lengths. Reject if eligible bytes are negligible, allocation/copy bytes do not fall, or any fallback or result differs.

Keep `PreTokenizedString`'s public owned API unchanged. Select only before the existing copy, retain the current owned path for owned normalization, multiple segments, added-token splits, and generic pre-tokenizers, and do not combine this change with scanner/BPE fusion.

Change to test: Add a private borrowed single-text-span pretokenized view selected before `PreTokenizedString::from_text` for borrowed-normalization fused encodes, while retaining the owned representation for all transformed or multi-segment paths.

Where it lives

perfloop/basetenkenizer · src/lib.rs

Evidence

No usable result yet.

Timeline