The fast cores have a cache the others don't

The chip I've been measuring has 12 cores in three kinds: 2 Super, 4 Performance, 6 Efficiency. I wanted to know what memory looks like from each kind, so I wrote a pointer chase and ran it on all three. The short version: all three have an L1 that answers in 3 cycles. After that, the Super cores have a private cache of about 1.4 MB at about 12 cycles that the Performance cores simply don't have. They go straight from L1 to 24-27 cycles.

The measurement

A pointer chase is a buffer where every 128-byte line holds the address of the next line to visit, in a random cyclic order. Each load needs the previous one's result, so the time per load is the latency of whatever level of memory holds the buffer. Make the buffer bigger and you walk down the hierarchy. The program is about 130 lines of C. I ran it from 16 KB up to 1 GB, four sizes per octave, and again from 64 KB to 32 MB at eight per octave.

To turn nanoseconds into cycles I needed each core's clock. That comes from a chain of dependent add instructions, one cycle each, timed right alongside the chase. It read 4.78 GHz on the Super cores, about 4.5 GHz on the Performance cores, and about 0.97 GHz on the Efficiency cores when they ran at background priority. One more detail: the first run after the process starts reads L1 at 1.47 ns, the next ones at 0.63 ns, because the core hasn't ramped its clock. So the program spins for 300 ms before timing anything.

Which core am I on?

macOS has no way to pin a thread to a core, and the register that should hold a CPU id read a constant 160 on this OS. So I steered and then checked. An idle machine at default priority runs my process on a Super core. Two busy-loop processes sometimes push it onto a Performance core. Background priority sends it to an Efficiency core.

The check is a fingerprint taken before and after every measurement: an 80 KB chase and the add-chain clock. 80 KB fits the Super L1 (128 KB) and the Efficiency L1 (96 KB) but not the Performance L1 (64 KB), and the clocks differ by 5x, so the pair identifies the tier. A Super core reads about 0.6 ns at 4.6-5.0 GHz; a Performance core 3.5-6 ns at 4.2-4.7 GHz; an Efficiency core 2.5-3.6 ns at 0.8-1.2 GHz. Any measurement whose fingerprint didn't match the intended tier on both sides was thrown away, and I took the median of what was left at each size.

The ladder

Reading the Super line from the left:

• L1, up to about 117 KB: 0.63 ns, 3.0 cycles. The sysctl value for the Super L1 is 128 KB, and the line leaves 3 cycles right there.
• A level at 197 KB to 1.3 MB: 2.30-2.53 ns, 11-12 cycles. This is the one the other cores lack.
• 1.88 to 17.9 MB: 5.67-6.90 ns, 27-33 cycles.
• Past that: 14.4 ns at 21 MB, 36 ns at 32 MB, 69 ns at 64 MB, and 86 ns at 256 MB. At 1 GB it's 87.3 ns, 418 cycles.

The Performance line has the same 3-cycle L1, but it ends at 64 KB (0.67 ns). From 108 KB to 2 MB it reads 5.3-6.0 ns, 24-27 cycles, and from 4 to 14 MB it reads 6.8-7.4 ns, 30-33 cycles. Past that, the few runs that stayed on the tier read 85-90 ns from 78 MB to 880 MB, about 380-406 cycles. There's no 12-cycle step. A Performance core's second level looks like the Super's third.

The Efficiency line, at its background-priority clock of 0.97 GHz, has L1 at 3.09 ns (3 cycles) up to 90 KB, then 17-20 ns (16-20 cycles) from 128 KB to 4 MB. Then it falls off: 51 ns at 8 MB, 112 ns at 16 MB, 271 ns at 32 MB, and 283 ns at 1 GB. That's 275 cycles, fewer than the Super's 418, but more than three times the time.

Is it private?

The 12-cycle level could be per core or shared between the two Super cores. So I made the two cores compete. Alone, a Super core chasing 1 MB measured 2.525 ns per load. Then I started the other Super core chasing 1 MB, 4 MB, and 16 MB at the same time, with the fingerprint confirming both were on Super cores. The 1 MB chase came back at 2.576-2.592 ns, 2-3% slower.

Two 1 MB working sets are 2 MB together, which wouldn't fit in a shared level of about 1.4 MB. If they were fighting over one, the chase would have fallen to the 27-cycle level. It didn't. So the level is per core.

Pages

Mode p of the program touches memory in a different order. It picks pages at random (16 KB each), visits all 128 lines of a page in random order, and moves to the next page. Mode r is random over the whole buffer. At 1 GB, the Super core measured 26.5 ns in mode p against 87.3 ns in mode r, 3.3x faster. The Efficiency core went from 283 ns to 118 ns.

Page ordering also answers a question about the Super core's 12-cycle level: could it be the TLB instead of a data cache? Page-ordered runs take roughly one TLB miss per page instead of one per load, and the step is still there: 10.8 cycles at 1 MB, 23.5 at 4 MB. So it isn't a TLB effect. It's a cache level.

What I couldn't separate

• Why page order wins. It could be DRAM row hits, or a prefetcher that notices it's inside a region. Telling those apart takes hardware performance counters, and this OS doesn't let me configure any.
• The L2 size. The sysctl lists 20 MB of L2 for both the Super and the Performance cores, shared per 2 and per 4 cores. What I measured on the Super side is a private 1.4 MB level and then something about 18 MB deep at 27-33 cycles. How the 20 MB figure maps onto that, I can't say.
• Performance-core data is thin. The scheduler kept moving those runs off the tier, so many were dropped and some sizes have a single measurement. The line is jagged for that reason. The 24-27 cycle plateau holds up across many sizes, but I'd trust the exact numbers less than the Super ones.
• The Efficiency cores were only measured at background priority. Their clock is about 1 GHz there. At a higher priority they might run faster, and the nanosecond figures would change.

The main result is the Super cores' private level, and the private test supports it. The Performance cores are missing it, so on working sets between about 100 KB and 1.4 MB they wait about twice as many cycles per load.