On Apple Silicon, the GPU starts paying at 512 by 512

On a Mac the CPU and GPU share one pool of memory, so moving an array to the GPU costs no copy. That makes it tempting to send everything there. I timed float32 matrix multiplies and elementwise math three ways: NumPy on the CPU, Apple's MLX on the GPU, and MLX on the CPU. The GPU was 5 to 6 times faster on big arrays. Below 512 by 512 it lost, because every time you wait for a GPU result you pay about 120 µs, whatever the size of the work.

The floor

Adding 1 to a four-element array and waiting for the answer took a median of 0.2 µs in NumPy, 9 µs in MLX on the CPU and 115 to 160 µs in MLX on the GPU (three runs). That last number is the cost of a round trip: build the command, hand it to the GPU, wait for it to say it's done. The work itself is close to nothing.

Matrix multiply

Square float32 matrices, median time per multiply, inputs already in place:

Up to 384 the GPU column barely moves: it's the floor plus a little. At 512 the two are even. The other two runs gave 0.92× and 0.72× there, so call 512 the tie and 768 the first clear win, about 2× in every run. From 1536 up the GPU did 7 to 12 TFLOPS depending on the run, against NumPy's steady 2.0 to 2.2.

NumPy and MLX on the CPU are within a few percent of each other from 512 up, so they're probably calling the same Accelerate routine. Below that, MLX on the CPU carries its own 9 µs floor.

If your data starts and ends as NumPy arrays, the GPU path also pays for wrapping the inputs and copying the result out. That moved the 1024 case from 0.30 to 0.52 ms and the 4096 case from 12.4 to 16.4 ms. The tie point stays around 512.

Elementwise

exp(x) * 2 + 1 on a float32 vector:

The GPU passes NumPy somewhere between 65,000 and 262,000 elements and ends up 6 times faster. But here MLX on the CPU is the surprise: it beats NumPy from 65,000 elements on, by 2.7 times at the top. NumPy runs this on one core and makes a temporary array for each step; MLX probably spreads it over cores and fuses the steps. At a quarter of a million elements MLX on the CPU even edges out the GPU, because it doesn't pay the floor.

The floor is per wait, not per operation

MLX is lazy: it builds a graph and only runs it when you ask for a result. So I chained K matrix multiplies and either waited once at the end or after each one:

Waiting after each multiply costs about 120 µs every time, so 128 of them take 15 ms. Waiting once at the end costs the floor once, and each extra multiply adds only 4 to 15 µs. With batching, 128 multiplies at n=256 take 1.86 ms on the GPU against about 2.8 ms in NumPy, so the GPU wins at a size where it lost 7 to 1 one at a time. At n=64 it still loses: 4.3 µs a multiply against NumPy's 1 µs.

An idle GPU is slower to answer

The 120 µs is for a GPU that's already busy. I slept between tiny jobs and timed the next one:

After 10 ms of nothing the first job costs about half a millisecond, four to six times the busy floor. I'd guess the GPU drops into a lower power state that fast. So a program that does one small GPU job every so often, like a request handler, pays about 0.5 ms each time, not 0.12.

So what

On this Mac, keep matrices under 512 on the CPU unless you can line up many of them before you wait. For elementwise work on NumPy arrays, try MLX on the CPU first: it beat NumPy from 65,000 elements without any GPU cost. Go to the GPU for big arrays and long chains, and count your waits, not your operations. Each eval, .item() or conversion back to NumPy is a round trip.

Method: one 12-core Apple Silicon Mac with 16 GB, MLX 0.32.4, NumPy 2.5.4, Python 3.14, float32 throughout. Each cell is the median of at least five calls and 0.4 seconds of repeats after two warm-up calls, timed with perf_counter around a call that waits for the result. I ran the main table three times; the speedups quoted at 512 and 768 are from all three. Other jobs were running on the machine, which mostly affects the small GPU times. I didn't look at float16 or at fused kernels written by hand.