A zstd dictionary needs fewer samples than you think
Small JSON messages compress badly on their own: there's nothing earlier in the message to point back to. A zstd dictionary fixes that by giving every message the same shared history. On an hour of real GitHub events, compressing each event alone, a dictionary took the ratio from 1.88 to 4.12. The zstd docs suggest training on about 100 times the dictionary's size in samples, "a few thousand" messages. On these events, ten messages already got 84% of the savings the best dictionary got. The trap runs the other way: if you ask for the default 110 KB dictionary with only a hundred samples, it compresses worse than a 16 KB one by about a quarter.
The data
GH Archive publishes public GitHub events as JSON in hourly files; in the files I downloaded, each line is one event. I took seven hours between October 2024 and October 2026. Recent events are small: in the hour I tested most on (12 October 2026, 15:00 UTC), 87,694 events with a median of 624 bytes, 80% of them pushes. Two years earlier the same feed carried full pull request and issue objects, so events were bigger and more varied. Each message is compressed on its own with Python 3.14's compression.zstd (libzstd 1.5.7) at level 3, the default, and dictionaries come from its train_dict, which wraps zstd's standard trainer. The ratio is total raw bytes over total compressed bytes. Training and test messages never overlap: they come from different hours, or from opposite halves of a shuffled hour.
How much a dictionary buys
With a 110 KB dictionary trained on 20,000 events from the hour before:
| Event size | Events | No dictionary | Dictionary |
|---|---|---|---|
| under 512 B | 335 | 1.65 | 4.21 |
| 512 B – 1 KB | 85,688 | 1.73 | 4.06 |
| 1 – 4 KB | 814 | 2.78 | 4.98 |
| 4 – 16 KB | 759 | 3.17 | 4.26 |
| over 16 KB | 98 | 3.84 | 4.65 |
| all | 87,694 | 1.88 | 4.12 |
That matches the zstd docs' own description: the smaller the message, the more the dictionary helps. For scale, compressing the whole hour as one stream, where every message can refer to all the ones before it, gives 5.33. Per-message compression with the dictionary closes 84% of the gap between "each message alone" and "everything at once". Level 19 with the dictionary got 4.38; dropping the frame's dictionary ID, size and checksum fields added another 3% at level 3, since those bytes are a real fraction of a 150-byte output.
How many samples
I trained dictionaries of six sizes (1 KB to 256 KB) on 10 to 30,000 sampled events, three random draws each, and tested each on 10,000 held-out events from the same hour.
On the 2026 events, ten samples and a 1 KB dictionary gave 3.50. The best result anywhere in the sweep was 4.21 (256 KB, 30,000 samples). In bytes, no dictionary leaves 53.5% of the input, ten samples leave 28.6%, and the best leaves 23.8%, so ten messages got 84% of the way. A hundred samples with a 16 KB dictionary reached 3.90. When most messages are the same few shapes, a handful of examples teaches the dictionary those shapes, and that's most of what there is to learn.
The 2024 events need more. Ten samples gave 4.45 against 2.99 without a dictionary and 7.19 at best: 56% of the savings. With bigger, more varied messages, the dictionary keeps finding more worth keeping all the way up to 30,000 samples and 256 KB. So "few samples are enough" depends on how uniform your messages are, and the cheap way to find out is to try it: train on 100, 1,000 and 10,000 and compare.
A dictionary too big for its samples
The dip in both panels is the useful warning. With 100 samples of 2026 events, about 71 KB of text, asking for 16 KB gave 3.90. Asking for 64 KB, 110 KB or 256 KB all gave the same 3.08: the trainer returned a 50 KB dictionary each time, about 70% of what it was given. Those messages then compress 27% bigger than with the 16 KB dictionary. On the 2024 events, 100 samples at 110 KB came out 14% bigger than at 16 KB. Level 19 didn't rescue it.
Across both data sets, a bigger dictionary was safe whenever the samples held at least four times as many bytes as it: in all 53 such cells, it did as well as or better than the best smaller size for the same samples (worst case 0.2% behind). Below four times, its ratio was up to 20% lower than a smaller one's, and below twice up to 22% lower. The zdict.h advice of about 100 times is a safe default, but the rule that matters with little data is the opposite of "bigger is better": shrink the dictionary to a quarter of your sample bytes or less. zstd's advanced trainers can also pick a smaller size for you (the header's shrinkDict option); I didn't test those.
How fast a dictionary goes stale
I trained a 110 KB dictionary on each of the seven hours and tested each one on every hour. On 12 October 2026 at 15:00, compared to a dictionary trained on that same hour (4.13):
| Dictionary trained on | Age | Ratio | vs fresh |
|---|---|---|---|
| 12 Oct 2026, 14:00 | 1 hour | 4.09 | −1% |
| 11 Oct 2026 | 1 day | 3.98 | −4% |
| 12 Sep 2026 | 1 month | 3.76 | −9% |
| 12 Apr 2026 | 6 months | 3.71 | −10% |
| 12 Jun 2025 | 16 months | 3.56 | −14% |
| 12 Oct 2024 | 2 years | 3.61 | −13% |
| none | 1.87 | −55% |
Even a dictionary from before the feed's payloads changed shape still beat no dictionary by nearly two to one. Going the other way hurt more: the October 2026 dictionary on October 2024 events gave 4.39, against 6.96 for a dictionary trained on that hour, because a dictionary built from small push events has nothing to offer for the pull request bodies it never saw. A stale dictionary costs a little when the messages have only drifted, and a lot when new kinds of content show up.
What to do
If you compress many small, similar messages one at a time, train a dictionary; you don't need a big corpus to start. Keep the dictionary size at a quarter of your training bytes or less, and don't reach for 110 KB until you have a few hundred KB of samples. Retrain when the shape of your messages changes, not on a timer: on this feed a month-old dictionary lost 9% and a two-year-old one 13%.
To try it on your own messages without writing any code, zdict.wickkit.cc trains and tests dictionaries of each size in the browser and hands you the one it picks.
Method: GH Archive hours 2024-10-12-14, 2025-06-12-14, 2026-04-12-14, 2026-09-12-14, 2026-10-11-14, 2026-10-12-14 and 2026-10-12-15 (UTC). Each event is one line of the file, compressed alone with zstd.compress at level 3 unless noted, default frame. The size sweep used three random draws of samples per cell (half-range at most ±0.3 on the 2026 events and ±0.54 on the 2024 ones); the drift table used 20,000 training events and 20,000 test events per hour. This is one feed of JSON from one source; other message streams will land in different places, but the size-to-samples rule held on both very different years of it.