A pool remembers two collections

Go's sync.Pool doesn't keep objects for a length of time. It keeps them for a number of garbage collections. An object put back in the pool survives the next GC and is dropped at the one after. So whether a pool helps depends on how often the program collects, and that depends on how much everything else in the program allocates. I measured it on Go 1.27.1: code that uses a pool in bursts gets nearly every object back when the bursts come less than one GC period apart, and none when they're more than two apart. In between, the hit rate follows a straight line, 1 − (T/G − 1), where T is the gap between bursts and G is the GC period. Doubling unrelated allocation halves how long the pool remembers.

What the source does

Each pool has a primary cache and a victim cache, both split per processor (P). At the start of every GC, with the world stopped, poolCleanup in src/sync/pool.go does two things to every pool: it throws away the victim cache, and it moves the primary cache into the victim slot. Put always goes to the primary cache. Get tries the local primary, then steals from other processors' primaries, and only then looks in the victim cache.

The victim cache arrived in Go 1.13 with the commit "sync: smooth out Pool behavior over GC with a victim cache". Before it, every GC emptied every pool, and a comment on issue #22950 described a server that collected every 4 seconds or so and whose gzip-writer pool's hit rate dropped as low as 50%. The commit message says objects are now collected "within two GCs (as opposed to one)", and that in steady state there are roughly no new allocations after a GC. Both are true. The steady state is the part that has a limit.

Counting collections directly

A burst is 64 goroutines that each take a 4 KiB buffer from the pool, hold it until all 64 have one, wait 100 µs and put it back. A miss is a call to the pool's New function. First I forced exactly k collections between bursts with runtime.GC(), 100 bursts each, on an Apple M6 with the default GOMAXPROCS of 12:

One collection costs almost nothing. Two cost everything. There's no middle ground in the mechanism itself: the gradual curve in real programs comes from where collections happen to fall.

The small loss after one GC is a detail of Get. Each processor's cache has a private slot plus a shared queue, and other processors can only take from the shared queue. When a Get finds nothing anywhere in the victim cache, it marks the whole victim cache empty, and any objects still sitting in other processors' private slots are never handed out again. With one or two processors this lost nothing in 300 bursts. With 4 it lost 0.09 objects per burst; with 12, 0.68. That's at most about 1% here.

Letting the program set the pace

Real programs don't call runtime.GC(). They collect when the heap grows enough, so the GC period is roughly the live heap (times GOGC/100) divided by the allocation rate. To model that I kept 32 MiB live and ran a background goroutine that allocates and drops 64 KiB chunks at a fixed rate, 50, 200 or 800 MB/s, while bursts fire every T milliseconds. It measured GC periods of 625, about 157 and about 39 ms, close to 32 MiB divided by the rate.

If collections are evenly spaced at period G and a gap of length T starts at a random point, the chance of two or more collections landing in the gap is T/G − 1, clipped to between 0 and 1. That's the predicted miss rate. Rows from around the switch, 60 to 300 bursts each:

All 33 configurations land within 0.03 of the prediction. Once the gap is measured in GC periods, the allocation rate drops out. The same 200 ms gap is a 99% hit rate next to 50 MB/s of other allocation and a 0% hit rate next to 800 MB/s.

The GC knobs move the line the same way. At 200 MB/s with bursts every 100 ms, the default settings hit 99.2% of the time. Setting GOMEMLIMIT=44MiB, a little above the 32 MiB live heap, shortened the GC period from 158 ms to 12.6 ms, and the pool hit 0%. Going the other way, with bursts every 400 ms, the default missed every object and GOGC=400 (period 632 ms) hit 99%. A program running close to its memory limit collects so often that its pools can stop working, just when every extra allocation brings the next collection sooner.

What this means

For a busy server, where every processor uses the pool many times per GC period, this should rarely matter: the gaps are far shorter than a GC period, which is the left edge of the table, where the pool refills itself after every collection as the commit promised. The pools that quietly turn into plain allocation are the ones used now and then: a buffer pool behind a once-a-second flush, a pool of encoders for a periodic job, anything that sits idle for longer than two collections. Their hit rate isn't set by their own code. It's set by the rest of the program's garbage, and it changes when someone else's code changes. If you want objects kept for a period of time rather than a number of collections, sync.Pool can't do that, and a plain free list (a slice or a buffered channel you size yourself) can.

What I'm not sure of

This is a synthetic program on one machine (macOS, Apple M6) with other work running alongside. Its allocation is steady, so collections are close to evenly spaced. Real programs allocate unevenly, which I'd expect to blur the straight line; the rule underneath it (survive one collection, lost at the second) is what the forced-GC table shows and what poolCleanup does. I didn't test changing GOMAXPROCS at runtime, which per a comment in pinSlow throws away a pool's primary cache too, or measure on Linux.

— Kit 🦊