Found it: a 64-byte pad on a 128-byte line

This morning I wrote that free-threaded Python stops at the fast cores: on a 12-core Apple chip, 12 threads in one python3.14t process got 4.2× over one thread while 12 processes got 5.8×, and I couldn't say why. Now I can. Every thread writes a small counter on every loop iteration, CPython pads each thread's counter to 64 bytes, and this chip's cache lines are 128 bytes. So pairs of threads share a line and fight over it. Keep the neighbor slot idle and threads match processes: 5.7× against 5.8×.

Same instructions, more cycles

The first post ended with "finding the shared line would take hardware counters". The kernel on this macOS version refuses to program the chip's configurable counters, even for root, so the events I wanted (loads served from another cluster) were out of reach. The two fixed counters still work: cycles and instructions retired, per core, across the whole machine.

On the empty loop (for _ in repeat(None, n): pass, 1.2 billion iterations), they settled what kind of problem it is. Twelve threads and twelve processes retired the same work, 101 to 105 billion instructions in each of two runs. The threads needed 26 to 29 billion cycles to do it, the processes 18. That's 3.7 to 3.9 instructions per cycle against 5.7. With 6 workers on the fast cores, both setups ran at 6.0 to 6.5. So the threads weren't doing extra work, retrying or spinning. They were waiting on memory.

The instruction that waits

Next, sampling: dtrace's profile provider records which instruction each core is on, about 5,000 times a second. Mapped back to the interpreter binary, two neighboring addresses stood out. In the 12-thread run they got about 14,000 samples between them. In the 12-process run, one got 1,640 and the other didn't make the top 60. Both are in the handler for JUMP_BACKWARD, the bytecode at the bottom of every loop:

ldr   x8, [x22, #0x380]   ; this thread's QSBR state
ldr   x9, [x8, #0x8]      ; → pointer to the shared state   ← hot
ldapr x9, [x9]            ; load the global write sequence
stlr  x9, [x8]            ; store it as this thread's seq
ldr   x8, [x22, #0x18]    ; then check for signals etc.     ← hot

That's _Py_qsbr_quiescent_state. Free-threaded CPython frees some shared memory lazily, with a scheme called quiescent-state-based reclamation: each thread periodically announces "I'm not holding any pointers right now" by copying a global sequence number into its own slot. A loop's back edge is a safe point: the periodic check that JUMP_BACKWARD runs calls QSBR_QUIESCENT_STATE before it looks for pending signals, so the interpreter announces on every iteration. The two hot instructions sit right after the loads and the store-release on that slot. That's where a core waits when the line it needs is held by another core.

The slots live in one array, and each is padded against exactly this problem:

// Padding to avoid false sharing
struct _qsbr_pad {
    struct _qsbr_thread_state qsbr;
    char __padding[64 - sizeof(struct _qsbr_thread_state)];
};

64 bytes is the cache line on most x86 chips. On this one, sysctl hw.cachelinesize says 128. Two threads' slots land on each line, and both threads write their halves on every loop iteration. That's classic false sharing, the thing the padding was meant to prevent. The header on CPython's main branch still pads to 64.

Testing it without rebuilding Python

In a fresh process, threads appear to take slots in the order they start (the result below depends on that, and bears it out), so the start order decides who shares a line. A thread blocked on a lock doesn't run bytecode, so it doesn't write its slot while it waits. So I started an idle thread between each pair of workers: worker, idle, worker, idle. If false sharing is the cause, every worker now shares its line with a thread that never writes, and the gap should close. As controls, I started the same idle threads all before or all after the workers, which changes nothing about which workers sit side by side.

On the empty loop with 12 workers (four runs each): workers alone took 0.66 to 0.73 seconds, idle threads before or after 0.66 to 0.76, and interleaved 0.44 to 0.46. Twelve processes took 0.41 to 0.46 across today's runs.

Then the first post's loop (s += i * i % 7, 600 million iterations), 1 to 12 workers, best of three:

With idle neighbors, 12 threads took 2.30 seconds; plain threads took 3.36. The interleaved line tracks the processes line within 0.2× the whole way, and the processes come from a separate run. That's nearly all of the gap from the first post, from one source. (Plain threads reached 3.9× here and 4.2× in the first post's run; the spread between runs is about that size.)

Why only past 6 cores

This part I can only partly explain. Up to 6 workers, the shared lines cost nothing measurable, and the first post found the same on the efficiency cores alone. The cost appears when the workers span the fast clusters and the efficiency cluster. My reading is that passing a line back and forth inside a cluster is cheap and across clusters is not, but checking that takes the configurable counters I couldn't use. The 128-byte line and the per-iteration write are measured. The cluster explanation is a guess that fits.

What to do about it

The fix in CPython is one constant: pad the slots to 128 bytes on Apple silicon, or to whatever the platform's line size is. Until then, a free-threaded program can work around it the way the test did: start an idle thread between workers so no two busy threads share a line. That's a hack that depends on slot order, which is an internal detail. The safer advice from the first post still holds: on a Mac with efficiency cores, stop a CPU-bound thread pool at the fast-core count, or use processes past that.

I also take back a sentence from the first post. I'd noticed the 64-byte padding and this chip's 128-byte lines, then dismissed them because the structures I could check didn't explain the slowdown. I'd checked the wrong structure.