Performance profiling and optimization

Profile before you optimize. Most Go performance problems are allocation rate (GC pressure) or contention (scheduler stalls), rarely the algorithm. Measure with pprof/trace, change one thing, A/B with benchstat. The toolchain already does constant folding, DCE, bounds-check elimination, inlining, devirtualization, escape analysis, and memclr, so don't hand-roll those.

Verified against Go 1.26.4 (stable, 2026-06) and the final Go 1.27 release notes (GA 2026-08-19). Tags [1.N] mark the release a claim lands in; [exp 1.N] = experimental behind a GOEXPERIMENT. Triangulated from multi-source research; resolved to go.dev/pkg.go.dev where reports diverged.

TL;DR: the modern deltas (read first)

• for b.Loop() [1.24] replaces for i := 0; i < b.N: setup runs once per -count, params/results stay alive (no manual sink), timer is automatic. Trap: 1.24/1.25 b.Loop suppressed inlining of the body (phantom allocs); [1.26] fixed it; re-measure any allocs/op from 1.24/1.25.
• Green Tea GC is the default [1.26] (span-granular, locality-aware marking): 10-40% less GC overhead on GC-heavy programs, +~10% on AVX-512 amd64. The GOEXPERIMENT=nogreenteagc opt-out remains available in Go 1.27.3 on linux/amd64. It reduces, not eliminates, GC cost; allocation reduction still compounds.
• runtime.AddCleanup [1.24] supersedes runtime.SetFinalizer: multiple cleanups/object, interior pointers, no cycle leaks, no resurrection, no extra-GC-cycle delay.
• Zero-copy str↔[]byte = unsafe.String/unsafe.Slice [1.20], never reflect.StringHeader/SliceHeader (deprecated, GC-invisible, broken under modern stack allocation).
• defer is ~1 ns since open-coded defers [1.14]: "avoid defer in hot paths" is stale (exception: defers in a tight loop or >8 per frame fall back to the heap path).
• The compiler stack-allocates slice backing stores in far more cases [1.25][1.26] (incl. append growth and some escaping slices moved to heap only at return). "Variable-sized backing store always heaps" is stale; verify with -gcflags=-m.
• benchstat changed: columns are sec/op/B/op/allocs/op + vs base with ~ for non-significant (Mann-Whitney U). The library golang.org/x/perf/benchstat is deprecated; use golang.org/x/perf/cmd/benchstat.

Table of contents

1. Runtime cost reference
2. Profiling workflow
3. CPU profiling
4. Execution tracer & flight recorder
5. Profile-guided optimization (PGO)
6. Memory profiling and escape analysis
7. Allocation reduction patterns
8. sync.Pool for hot-path objects
9. String and slice optimization
10. Inlining
11. Memory layout & false sharing
12. GC tuning
13. Benchmarking methodology
14. SIMD
15. Caching strategies
16. Library landscape & status
17. Version map 1.20→1.27
18. Anti-patterns / what models get wrong

---

Runtime cost reference {#runtime-costs}

Order-of-magnitude estimates on modern amd64/arm64, not guarantees. There is no authoritative portable ns/op table (varies by microarch, OS, patch level, PGO/GC state, workload); benchmark on target. Treat as relative.

| Operation | Rough cost | Notes | |---|---|---| | Direct call | ~1-2 ns | plus stack-growth check | | Interface / indirect call | ~2-5 ns | blocks inlining; PGO can devirtualize hot ones | | defer (common case) | ~1 ns | open-coded [1.14]; was ~50 ns pre-1.14. Loop/>8 defers fall back to heap path | | Small heap allocation | ~20-100 ns | + amortized GC scan/sweep; [1.27] size-specialized malloc cuts <80 B allocs up to ~30% | | Map access (hit) | ~10-30 ns | Swiss Tables [1.24] improved this | | Slice append (no grow) | ~1-3 ns | grow = alloc + copy | | Channel send/recv (uncontended) | ~50-120 ns | buffered slightly cheaper; contended far higher | | sync.Mutex lock+unlock (uncontended) | ~10-25 ns | CAS fast path inlined; slow path outlined lockSlow; new internal mutex [1.24] | | Atomic load/store/CAS | ~1-10 ns | CAS under contention much higher | | Goroutine create / context switch | ~1-2 µs / ~100-200 ns | ~2 KiB initial stack; switch far cheaper than an OS thread | | cgo call | ~40 ns | [1.26] cut baseline ~30% (streamlined thread-state mgmt) | | L1 hit / main memory | ~1 ns / ~100 ns | the gap is why false sharing and poor locality dominate |

---

Profiling workflow {#profiling-workflow}

Measure → identify the limiting resource → use the matching profiler → fix one bottleneck class → re-benchmark. The GC guide says start with heap profiles when GC cost matters, and that alloc_space (allocation rate) is usually the most useful view for cutting GC cost, not inuse_space (live memory).

Two front-ends, one backend: runtime/pprof (manual/batch) and net/http/pprof (import _ "net/http/pprof" registers on http.DefaultServeMux). All /debug/pprof/ paths require GET [1.22].

| Profile | Samples | Enable | Endpoint | |---|---|---|---| | CPU | on-CPU time, ~100 Hz | pprof.StartCPUProfile/Stop | /debug/pprof/profile?seconds=30 | | Heap (inuse/alloc) | live + cumulative allocs, every MemProfileRate (512 KiB) | pprof.WriteHeapProfile | /heap (?gc=1), /allocs | | Block | time blocked on chan/select/timer/sync | SetBlockProfileRate(n) (off by default) | /block | | Mutex | contention on Mutex/RWMutex | SetMutexProfileFraction(n) (off by default) | /mutex | | Goroutine | live goroutine stacks | none | /goroutine | | goroutineleak [exp 1.26 / GA 1.27] | goroutines on an unreachable primitive | GOEXPERIMENT=goroutineleakprofile (1.26 only; GA in 1.27, experiment deleted) | /debug/pprof/goroutineleak |

/allocs and /heap are the same profile, different default sample type. Block/mutex are useless unless sampling is enabled in-process first; skipping it is the #1 mistake. SetBlockProfileRate(1)/SetMutexProfileFraction(1) cost 5-20% CPU under load; sample instead:

// WRONG: every event; catastrophic under concurrency
runtime.SetBlockProfileRate(1); runtime.SetMutexProfileFraction(1)
// RIGHT: ~1 event per 10000 ns blocked; ~1/100 contention events
runtime.SetBlockProfileRate(10000); runtime.SetMutexProfileFraction(100)

Enable globally only at conservative rates, or via an admin endpoint during an incident. Interpretation trap: mutex-profile stacks point to the end of the critical section that caused contention, not the waiter; since [1.25] runtime-internal lock contention matches this too.

Reading: go tool pprof -http=:8080 cpu.pprof (flame graph is the default view [1.26]; graph under View → Graph). REPL: top20 -cum, list <fn>, peek <fn>, disasm. flat = self, cum = self + callees (hot leaves by flat, hot paths by cum). Differential: go tool pprof -diff_base=old.pprof new.pprof. [1.23] profile stack depth went 32→128 frames (better deep attribution).

---

CPU profiling {#cpu-profiling}

import "runtime/pprof"

func main() {
    f, _ := os.Create("cpu.prof"); defer f.Close()
    if err := pprof.StartCPUProfile(f); err != nil { log.Fatal(err) }
    defer pprof.StopCPUProfile()
    // ... workload
}
// In benchmarks the framework wires it: go test -bench=. -cpuprofile=cpu.prof

Stale-model note: flame graphs do not need uber/go-torch (deprecated); go tool pprof -http renders them natively and they're the default view in 1.26.

---

Execution tracer & flight recorder {#tracing}

runtime/trace records goroutine create/block/unblock, scheduler latency, syscalls, GC, heap-size changes, P start/stop with ns timestamps + stacks. It reveals what CPU sampling cannot: a channel bottleneck shows as the absence of execution. Use CPU profile for where time goes; trace for why goroutines aren't running. Annotate with a task (spans goroutines via context.Context, one request) and a region (same-goroutine interval, one stage):

trace.Start(f); defer trace.Stop()             // or: go test -trace=trace.out; go tool trace trace.out
ctx, task := trace.NewTask(ctx, "http_request"); defer task.End()
trace.WithRegion(ctx, "db", func() { queryDB(ctx) })

Overhead is stale folklore. Pre-1.21 tracing cost ~10-20% CPU; [1.21] cut traceback cost ~10× (≈1-2%), and [1.22] made the format splittable, enabling continuous tracing/flight recording. [1.23] flushes trace data on uncaught panics.

Flight recorder [1.25]

A sliding in-memory ring buffer over the trace: snapshot only the window before a rare event (timeout, latency spike) instead of spooling continuously. Lives in runtime/trace (the golang.org/x/exp/trace prototype is outdated).

fr := trace.NewFlightRecorder(trace.FlightRecorderConfig{
    MinAge:   10 * time.Second, // retain ≈ this long; set ~2× your debug window
    MaxBytes: 10 << 20,         // memory cap: TAKES PRECEDENCE over MinAge
})
if err := fr.Start(); err != nil { log.Fatal(err) }
defer fr.Stop()
if dur > 500*time.Millisecond { f, _ := os.Create("incident.trace"); fr.WriteTo(f); f.Close() }

API: Start() error, WriteTo(io.Writer) (int64, error), Stop(), Enabled() bool; defaults MinAge 10 s, MaxBytes 10 MiB. Only one recorder per process; one WriteTo at a time (Stop blocks until in-flight writes finish); may coexist with a trace.Start consumer.

Goroutine-leak profile [exp 1.26 → GA 1.27]

Piggybacks on the GC reachability graph: flags a goroutine blocked on a channel/Mutex/Cond whose primitive is unreachable from any runnable goroutine or global root (topologically impossible to unblock): zero false positives, zero overhead unless in use. Cannot detect leaks via primitives still reachable through globals/live locals. Makes leaks runtime-profiled data, superseding go.uber.org/goleak for production detection. GA in [1.27]: profile type goroutineleak in runtime/pprof, endpoint /debug/pprof/goroutineleak; the GOEXPERIMENT=goroutineleakprofile setting was deleted.

---

Profile-guided optimization (PGO) {#pgo}

PGO feeds a representative CPU pprof profile back to the compiler, which then inlines hot functions more aggressively and devirtualizes hot interface calls. Whole-program: all packages incl. dependencies are recompiled. Preview [1.20], GA [1.21] with -pgo=auto default. Realistic gain 2-14%, zero code change, the cheapest big win.

curl -o cpu.pprof "http://service:6060/debug/pprof/profile?seconds=30"
mv cpu.pprof ./cmd/server/default.pgo   # MAIN package dir; commit to VCS (auto-detected [1.21+])
go build ./cmd/server                   # explicit: -pgo=./path.pprof or -pgo=off
go version -m ./server | grep pgo       # verify applied
go tool pprof -proto a.pprof b.pprof > default.pgo   # merge profiles for robustness

Truths & pitfalls: microbenchmarks are bad PGO profiles (sliver of program → tiny gains); use real traffic; the go command errors if default.pgo is in a non-main package; profiles are portable across GOOS/GOARCH; source-stable (last week's profile matches today's build by in-function line offsets; PGO-on-PGO doesn't flap, so no two-stage canary); a stale profile optimizes cold paths but won't slow hot ones. Build overhead is no longer a reason to skip it [1.23]: from 100%+ on large builds down to single-digit %. Field data: Datadog ~5.4% (1.21), >10% on some 1.22 services; Uber runs it fleet-wide. "PGO only affects layout, not inlining" is wrong; it raises the inlining budget for hot call sites (see Inlining).

---

Memory profiling and escape analysis {#memory}

The compiler decides stack (pointer bump, auto-freed) vs heap (G