Generic Call branch builds duplicate type-argument vectors

perfloop/baml · ALLOCATION HOT LOOP

https://perfloop.ai/t/oss/case_ckgjt8y54f

Verdict

VERIFIED · settled 2026-09-16 · pull request opened as BoundaryML/baml#4898

What happened: The paired measurements met the required improvement.

Hypothesis

Generic bytecode calls build a type-argument vector for temporary rooting and then clone the same lanes into the callee frame. The maintained `compute::generic apply with inferred type args 1m` workload repeats this inside a one-million-iteration loop, so it may create avoidable allocation and copy traffic. Its contribution to CPU time, allocator bytes, and end-to-end latency is not measured in this hunt.

At revision `29fcfe5e6c4513417c037f2388dc3efb5acecb77`, the workload defines the free generic function `gapply<T>` and calls it for every `i` from 0 through 999,999. Its source explicitly says that `T` is inferred at every call site and that the callee frame is seeded with `[int]`. `runtime_benchmark.rs` invokes `BexEngine::call_function("main", ...)`; `call_function_with_trace` delegates to the same `call_function_bound_args_with_trace` implementation used by the accepted entry API, which starts the VM event loop and calls `BexVm::exec`. Compact execution reaches this accepted `step_compact` target. On the successful direct-call branch, `ntypeargs != 0` selects vm.rs:8504, where the owned `TakenTypeArgs` is reduced to borrowed `CallOptions` slices before call setup; the workload has one function type lane and no source-level early exit from its loop.

The called setup path materializes `options.type_args.to_vec()` into the `pending_call_type_args` field, then, after a bytecode frame was pushed, calls `frame.type_args.extend_from_slice(options.type_args)`. For the documented one-lane `[int]` call, source therefore bounds one nonempty temporary vector construction and two `RealizedTy` lane clones per successful iteration: one million temporary vectors and two million lane clones per `main()` execution. This count excludes the conditional exact-value lane, whose extra work applies only when a runtime definition value is present. The temporary crosses into a VM field and the final vector is stored in the frame, so it is allocation-shaped work rather than a register-only value copy.

The caller's `TakenTypeArgs` is otherwise dead after call setup. An ownership-transfer path can keep it rooted while a native callee runs, then move it into a newly pushed bytecode frame; the Case must verify the existing error, yield, nested-call, and metadata semantics before treating that as safe. Confidence is medium.

The proof target is the generated Divan benchmark for `compute::generic apply with inferred type args 1m` in `baml_language/crates/baml_tests/benches/runtime_benchmark.rs`, filtered as `generic_apply_with_inferred_type_args_1m`, with its normal compiled-once engine, fresh call context per sample, and `BAML_PROFILE=0`. Compare allocation calls and bytes attributed to generic call setup plus CPU and wall time for the ordinary `main()` call, while retaining the benchmark result and generic type behavior. The expected source-level delta for this workload is removal of the one million temporary vectors and two million lane-copy operations; measured savings remain a Case question.

Change to test: Pass owned `TakenTypeArgs` from the generic Call branch into an owned call-setup path, so the pending root temporarily owns the lanes and a successfully pushed bytecode frame takes them instead of cloning them twice. Preserve native-call rooting, restoration of outer pending lanes on errors or yields, type-argument order, and sparse `FrameTypeMetadata` alignment.

Where it lives

perfloop/baml · baml_language/crates/bex_engine/src/lib.rs

Evidence

generic apply with inferred type args 1m · 10 sample pairs

metric baseline candidate paired median change confidence range required result
alloc_calls 10000031 9000031 −10% (−1000000) −1000002 to −999999 < −800000 PASSED
alloc_bytes 616265794 488265794 −20.8% (−127999993) −128000030 to −127999985 < −100000000 PASSED

generic apply with inferred type args 1m ordinary timing · 10 sample pairs

metric baseline candidate paired median change confidence range required result
cpu_ns 600000000 530000000 −11.7% (−70000000) −80000000 to −60000000 < −30000000 PASSED
wall_ns 607594801 537813765 −11.5% (−69961786) −72486635 to −61544162 < −30379740 PASSED

interfaces polymorphic dispatch 100k ordinary timing · 10 sample pairs

metric baseline candidate paired median change confidence range required result
wall_ns 84020000 81215000 −3.6% (−3000000) −3570000 to −2090000 ≤ 1000000 PASSED

closure apply 1m ordinary timing · 10 sample pairs

metric baseline candidate paired median change confidence range required result
wall_ns 551000000 496400000 −9.8% (−54200000) −58600000 to −50400000 ≤ 1000000 PASSED

pure call 1m ordinary timing · 10 sample pairs

metric baseline candidate paired median change confidence range required result
wall_ns 183700000 174900000 −4.6% (−8500000) −9900000 to −7800000 ≤ 1000000 PASSED

classes method call 100k ordinary timing · 10 sample pairs

metric baseline candidate paired median change confidence range required result
wall_ns 37715000 37840000 −0.1% (−30000) −470000 to +370000 ≤ 1000000 PASSED

Checks: 6 of 6 passed. Verification: no defect found.

Timeline