Python's zstd threads sit idle above level 3

Python 3.14 ships zstd in the standard library as compression.zstd, and it exposes zstd's multithreading through one option, CompressionParameter.nb_workers. On a 16 MB input, eight workers made level 1 compress 4.2× faster. At level 3 they gave 1.7×. At level 9 and above they did nothing at all. Setting two more parameters, job_size and, at the top levels, overlap_log, turned that into a 3.4× speedup at level 9 and up to 5× at level 19, for a ratio loss of 1 to 4%.

The test

The input is every .py file in the CPython 3.14 standard library, concatenated in sorted order: 16.3 MB of real source code. Each configuration is a one-shot compress(data, options=…) call, timed five times. I report the median of three such rounds, each run in a fresh process. The setup is CPython 3.14.8 with its bundled zstd 1.5.7, in a Linux aarch64 container with 10 cores. The machine had other work on it, so treat differences under about 10% as noise.

Level 3 stops at two workers, and from level 9 up the extra threads might as well not exist. Setting nb_workers=1 isn't free either: it moves compression onto a background thread, which was 2 to 5% slower than 0 at the fast levels and changed the output slightly.

Why: one big job

zstd's multithreaded mode doesn't split the input across workers evenly. It cuts the input into jobs and hands each job to a worker. With job_size left at 0, the job size scales with the compression window, which grows with the level. The numbers fit jobs of about four windows: 2 MB at level 1 gives eight jobs for eight workers, 8 MB at level 3 gives two jobs, and 16 MB or more from level 9 up puts the whole input into a single job. A 16 MB file is simply too small to split at the default job size. A much larger file would be split, so this mostly affects inputs in the tens of megabytes and below.

Setting the job size yourself fixes level 9:

Level 19 needs the overlap too

At level 19, job_size did nothing. With 8 MB, 4 MB, 2 MB and 1 MB jobs the output was the same size to the byte, and the time stayed around 2.5 s. The reason is the overlap: each job is primed with the tail of the data before it, so matches can cross job boundaries. At the strongest levels that overlap defaults to the whole window, 8 MB here, and zstd won't make a job smaller than its overlap. overlap_log sets the overlap as a fraction of the window: 9 is the whole window, and each step down halves it.

The useful comparison is with level 15 on one thread: 1.23 s for a ratio of 5.31. Level 19 with 2 MB jobs and overlap_log=6 takes half that time and still compresses 6% better. Below overlap_log 5 the time barely improves and the ratio keeps falling. Lowering overlap_log without setting job_size changed nothing, because the default job is still bigger than the input.

Takeaway

Setting nb_workers on its own only reliably helped at level 1. For anything stronger on inputs under a few tens of megabytes, set the job size as well:

from compression.zstd import compress, CompressionParameter as P
opts = {P.compression_level: 19, P.nb_workers: 8,
        P.job_size: 2 << 20, P.overlap_log: 6}
out = compress(data, options=opts)

Every setting here round-tripped through decompress correctly, and the output is an ordinary zstd frame that any decoder reads. Method: one input, one machine, one zstd version. Code with a different mix of repetition, or a much larger file, will land in different places, but checking whether your threads are working at all takes one timed run.