A fixed sleep in a test is a bet on how busy the machine is

A test starts a helper server, sleeps a bit, then connects. The sleep is tuned on a quiet laptop and passes every time. On a loaded CI runner, it starts failing about one run in five. I wanted to know how fast that margin disappears, so I timed how long a freshly spawned Python server takes to accept a connection while busy processes fight it for the CPU. On an idle machine it took 17 ms. With 16 busy processes per core it took 377 ms at the median and 604 ms at p99. A sleep of half a second, 30 times the idle median, guessed wrong 17% of the time.

The test

A container with 4 CPUs ran Python 3.14. Each trial spawned python -c with a script that imports http.server, binds a loopback port, prints it and listens. The parent then retried connect() every 2 ms until it worked, and timed spawn to first connection. Before the trials began, N processes running while True: x += 1 were started in the same container. Each load level got 500 trials. The last three columns show how often a fixed sleep of that length would have been too short.

Until the busy processes outnumber the cores, nothing much changes: the new process finds an idle core or one that frees up soon. Past that point it gets roughly its fair share of the CPU, so startup grows by about 6 ms for each extra busy process here. The growth isn't smooth. I ran the 12- and 16-process levels three times each, and the medians moved by ±20% between runs (68–96 ms at 12, 72–93 ms at 16). The tail grows faster than the median: at 16 per core, p99 is 36 times the idle median.

Why this bites tests in particular

A test suite run in parallel is the load. With a runner that starts one job per core, and each job spawning a server, a compiler or a browser of its own, the machine is oversubscribed by design. A sleep that's 10 times the idle startup time looks generous. Under 2 to 4 jobs per core, the test that relies on it fails a few percent of the time, which is exactly the rate that gets labelled "flaky" and retried rather than fixed.

Making the sleep longer only moves the line. A one-second sleep never failed in my 6,500 loaded trials, but it charges every run a full second, even on an idle machine where the server is up in 17 ms.

So what

Poll for the condition, with a deadline. Something like this:

def wait_until(check, deadline=30, every=0.01):
    end = time.monotonic() + deadline
    while time.monotonic() < end:
        if check():
            return
        time.sleep(every)
    raise TimeoutError(check)

On a quiet machine it returns as soon as the server is up, at most one interval late. On a loaded one it waits as long as it has to. The deadline can then be huge (30 seconds, a minute), because it only matters when something is really broken. A test that sleeps pays the worst case every time and still loses when the load passes it. A test that polls pays the actual startup time and only fails when the server never comes up.

Method: one machine, a 40-line Python script, Python 3.14 in a container limited to 4 CPUs, busy loops in the same container, 500 trials per run and 7,500 in total. Readiness was a loopback connect(). Servers that load more code at startup will scale the same way from a higher base.