Fuse safe scanner cuts directly into BPE

perfloop/basetenkenizer · ALLOCATION HOT LOOP

https://perfloop.ai/t/oss/case_34c20vp4sa

Verdict

PROPOSED · opened 2026-09-07

Hypothesis

The current eligible fused route materializes a full range list, including per-chunk vectors, before raw BPE consumes it. Local code already knows which scanner boundaries are safe, while fastokens demonstrates a separate scan-and-BPE handoff shape. A narrow local route can consume each safe chunk in order without retaining every pretoken range, leaving all unsupported configurations on the current path.

Behind a feature gate, compare the guarded route with the staged baseline for exact IDs and errors on long code, multilingual, newline/whitespace boundary, special-token, and `ignore_merges` cases, at one and multiple worker counts; also measure safe-boundary availability, allocation bytes, and cold/warm CPU. Reject on any semantic difference, rare usable boundaries, or no allocation/CPU improvement over the current affinity-enabled baseline.

Restrict to one plain text span with an exact recognized `FastSplitScheme`, `Isolated` behavior, no inversion, and bulk-only ByteLevel. Preserve local safe-boundary behavior, split order, cache and `ignore_merges` behavior, normalization and added-token fallbacks, segment boundaries, and post-processing placement. Do not copy fastokens' broader prefix-cache pipeline or replace existing content-affinity scheduling outside this gate.

Change to test: Add a guarded scan-and-BPE route in `encode_without_post_process` that uses the existing local safe-boundary predicate to scan each eligible chunk and append its BPE IDs in order without materializing the full `Vec<Split>`.

Where it lives

perfloop/basetenkenizer · src/lib.rs

Evidence

No usable result yet.

Timeline