Pool OTLP converter scratch state

perfloop/prometheus · ALLOCATION HOT LOOP

https://perfloop.ai/t/oss/case_2vbpy6vz77

Verdict

VERIFIED · settled 2026-08-10 · merged as prometheus/prometheus#19388

What happened: The paired measurements met the required improvement.

Hypothesis

ServeHTTP reaches rwExporter.ConsumeMetrics once per decoded OTLP request, and ConsumeMetrics creates a fresh converter at lines 120–125. NewPrometheusConverter initializes two label builders and make(map[string]string, 64) on that per-batch cadence. I ran the existing single-worker sum/cumulative handler benchmark with a memory profile: `go test ./storage/remote -run '^$' -bench='BenchmarkOTLP/type=sum/temporality=cumulative/cfg=default$' -benchtime=1s -count=1 -cpu=1 -memprofile=/tmp/otlp-write.mem`. It reported 36,636 iterations, 18,373 B/op, and 111 allocs/op; `go tool pprof -alloc_space -list NewPrometheusConverter` attributed 220.5 MB flat allocation to the 64-entry map and 239 MB cumulative (23.89% of that 1,000 MB profile) to the constructor. That benchmark uses test storage, so it demonstrates the allocation site rather than production GC or latency impact. A shorter cumulative default-versus-convert benchmark also showed no stable latency difference, so CPU improvement remains hypothesized. A case session should add a production-shaped multi-request benchmark and compare allocs/op, B/op, CPU, and alloc-space profiles before and after; the confirming signal is that constructor/map allocation disappears or sharply falls without conversion-result changes, races, or retaining request/appender data after Commit/Rollback.

Change to test: Have each rwExporter reuse a converter from a pool, reset all request-local converter state and install the current appender on checkout, then clear request references and return it only after Commit or Rollback; discard unusually large cached maps or buffers rather than retaining them indefinitely.

Where it lives

perfloop/prometheus · storage/remote/write_otlp_handler.go

Evidence

OTLP write handler decoded request, serial at GOMAXPROCS=1 · 10 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 28072 26359 −7.5% (−2119) −3275 to −767 ≤ 0 PASSED
B/op 19169 14025 −26.8% (−5144) −5144 to −5144 < 0 PASSED
allocs/op 190 185 −2.6% (−5) −5 to −5 < 0 PASSED

OTLP write handler decoded requests, parallel at GOMAXPROCS=4 · 10 sample pairs

metric baseline candidate paired median change confidence range required result
ns/op 14046 12239 −11.8% (−1661) −2297 to −611 ≤ 0 PASSED
B/op 19173 14058 −26.7% (−5115) −5116 to −5114 < 0 PASSED
allocs/op 190 185 −2.6% (−5) −5 to −5 < 0 PASSED

Checks: 5 of 5 passed. Verification: no defect found.

Timeline