Go's cancelled timeouts keep requests alive too

Earlier today I wrote that a cancelled asyncio timeout keeps your request alive: the cancelled timer stays in the event loop's heap, still holding the request's context, until a lazy cleanup gets to it. Go's context.WithTimeout has the same shape. Calling cancel() stops the timer, but the runtime can only mark a stopped timer as dead and leave it in the heap, and the dead timer's callback still points at the context, its parents, and every value stored in them. Go cleans up sooner than asyncio: a heap can hold dead timers up to about a third of its live ones, where asyncio allows about as many as it has live. In a net/http server with 10,000 idle keep-alive connections and handlers that do 10 ms of work, about 1,600 finished requests' values were alive at a time, 2,800 at the worst sample. A ten-line change to the context package brings it to zero.

Where the context goes

In Go 1.27.1, WithDeadlineCause arms the deadline like this:

c.timer = time.AfterFunc(dur, func() {
    c.cancel(true, DeadlineExceeded, cause)
})

The closure holds c, the new context, and c holds its parent, all the way up the chain. If a middleware did context.WithValue(r.Context(), userKey, session) before the timeout, the session is reachable from the timer. When the handler finishes, cancel() calls c.timer.Stop() and sets c.timer = nil. That drops the context's pointer to the timer, but not the timer's pointer to the context.

The timer itself lives in a per-P heap in the runtime. timer.stop() says why it doesn't take the timer out: "It may be on some other P, so we can't actually remove it from the timers heap. We can only mark it as stopped. It will be removed in due course by the P whose heap it is on." The mark is a timerZombie flag. The callback and its argument stay set, because Reset after Stop reuses them.

A zombie leaves the heap in three ways:

• It reaches the top of the heap, meaning every live timer due before it has fired or gone. Each time a timer is added, the runtime first pops zombies off the top.
• It sits in the last slot of the heap's array. The same pass trims zombies off the end, which is cheap because nothing has to move.
• The P that owns the heap sees more than a quarter of its entries are zombies and rebuilds the heap without them. If Z is the number of zombies and L the number of live timers, that triggers when Z > (L + Z)/4, so when Z passes L/3.

The first two catch the easy case. One goroutine that creates a timeout and cancels it at once leaves its zombie at the top or the end of the heap, and the next timer clears it; I measured zero kept contexts that way. What keeps zombies around is a live timer due earlier sitting above them, such as a request that started before them and is still running. Then only the quarter rule frees them, and how many it lets pile up depends on how many live timers the heap has. On a server, those include every connection's read deadline.

Measuring it

Each request allocated a 32 KB value, stored it with context.WithValue, wrapped that in context.WithTimeout, and cancelled when done. A cleanup registered with runtime.AddCleanup marked each value when the garbage collector freed it. Every 250 ms I listed the requests that had already finished and whose value wasn't freed yet, forced a collection, and counted the ones still not freed. The same program with WithCancel in place of WithTimeout kept zero in every configuration, so whatever the count shows is the timer.

First a synthetic load on one P: 64 goroutines each holding a request open for 0–2 ms, a 30 s timeout, and L extra timers that stay armed throughout, standing in for connection deadlines. Four runs per row (the extra timers due in 10 s in two, in an hour in the other two), Linux on arm64, GOMAXPROCS=1:

The count saws between zero and about L/3 and averages around a sixth of L. At 100,000 live timers the worst sample was 33,059 kept requests, about a gigabyte of 32 KB values. With four Ps the totals were similar when the extra timers were due in 10 s, before the requests' deadlines, and three to ten times lower when they were due in an hour. I didn't pin down why the deadline mattered there and not on one P; which P's heap each timer lands on is the likely part.

Then a real net/http server on loopback, four Ps, IdleTimeout of 120 s. Some connections made one request and then sat idle, each holding an armed read deadline. 256 client connections sent requests whose handler did WithValue, WithTimeout(5 s), 0–20 ms of simulated work, and cancel(), at about 23,000 requests a second:

Going from 10,000 to 20,000 idle connections didn't raise it further; I didn't chase why. The work matters more than the connections. With handlers that returned at once, every idle count kept between zero and six requests, because no in-flight request sat above the zombies and they were popped off the top as soon as the next timer arrived.

As with asyncio, this isn't a leak. Memory stops growing. It's a bound that scales with how many timers the process has, which grows with the same load that makes the kept values add up.

The fix

The runtime can't clear a stopped timer's callback, because a later Reset needs it. The context package can stop handing the timer a direct pointer. I gave timerCtx one more field, an atomic.Pointer[timerCtx] that the closure goes through, and made cancel set it to nil:

ref := new(atomic.Pointer[timerCtx])
ref.Store(c)
c.ref = ref
c.timer = time.AfterFunc(dur, func() {
    if c := ref.Load(); c != nil {
        c.cancel(true, DeadlineExceeded, cause)
    }
})

A zombie then holds only an 8-byte pointer cell with nil in it. I built both programs against the patched package with go build -overlay, which swaps one standard library file:

The cost is one more small allocation per WithTimeout: the package's own BenchmarkWithTimeout, which creates ten contexts per operation, went from 40 to 50 allocations and from 2,720 to 2,960 bytes per operation. Its timings moved by less than the run-to-run noise on my machine. The context package's tests pass with the change, with and without -race. I looked for an existing Go issue about this and didn't find one.

Until something like it lands, the workaround is the same as the cause: a request's values stay reachable from its timeout's timer, so keep big things out of the context chain above a WithTimeout, or pass them alongside the context instead. Measured with Go 1.27.1 on Linux arm64.