Eight VMs, One Lock

Apple's open-source container tool runs every Linux container in its own small virtual machine instead of sharing one Linux kernel. That buys real isolation, and the usual worry is the cost: a VM has to boot. I wanted the actual number, where it goes, and what changes it. The answer to the last part turned out to be the interesting one. Almost nothing changes the time for one VM, but starting several at once is no faster than starting them one after another, and the reason is a single lock.

One VM: about 0.6 seconds

On a 12-core Apple Silicon Mac with tool version 1.5.0, container run alpine true took a median of 607 ms over 60 runs (10th–90th percentile 573–625 ms). For scale, measured the same way from a script:

So a fresh VM costs about ten Node startups, or 400 bare processes. Once a VM is up, running something in it costs about the same as starting Node.

To see where the 0.6 seconds goes, I had the guest print its own uptime as the first thing it does, and read the guest kernel's boot log afterwards, which timestamps events from the moment the guest kernel starts. Comparing the uptime with when that output reached the host splits the time into what happens before the guest starts and what happens inside it. Medians of the same 60 runs:

The Linux kernel itself is the cheap part: under a tenth of a second. Most of the time goes on host-side setup and on the back-and-forth between host and guest that prepares the container once the kernel is up. Guest uptime is only readable to 10 ms, so the last two rows are good to about that.

What doesn't matter

I swept the obvious settings, 30 runs each, shuffled together so slow drift couldn't favour one setting:

• Memory, 256 MB to 8 GB: 600–613 ms. 8 GB added about 20 ms of host setup and nothing else.
• CPUs, 1 to 8: 594–607 ms.
• Image size, 4 MB to 156 MB as the tool reports it: 592–607 ms. The image is already unpacked on disk, and a bigger one doesn't take longer to attach.
• No network: 580 ms, about 20 ms faster. A quiet kernel command line and skipping DNS setup made no difference.

All of those are within a few percent of each other. The cost is fixed.

Many VMs: a queue

Then I started k VMs at the same moment, six times each for k from 1 to 32. With 12 cores I expected most of the 0.6 s to overlap. None of it did.

Thirty-two VMs took 18.9 seconds, which is 32 × 0.59. Starting them together saved nothing. The machine was nearly idle the whole time, so it wasn't CPU. The timeline on the left shows two separate queues. The guest kernels start one at a time, almost exactly half a second apart. And the commands don't run when their own VM is ready: the first kernel started at 0.2 s, which on its own would have meant a command running around 0.6 s, but that command waited until 2.6 s, and the other seven ran in a cluster at 4.2–4.6 s, once the last kernels had started.

Splitting the steps didn't help: creating 16 containers first and then starting all 16 at once took 8.9 s, and 16 runs at once with networking off took 9.0 s, against 9.3 s for 16 ordinary runs at once. By contrast, 16 exec calls into one already-running VM finished in 0.3 s together against 1.05 s one after another, so running commands does overlap. Only starting VMs doesn't.

The lock

The tool's source is public, so I looked. The API server's container service has one lock for the whole service, an AsyncLock created once. Bringing a container up (bootstrap) takes that lock and holds it while it registers the per-container runtime helper and waits for the whole VM boot over IPC. Starting the container's process (startProcess) takes the same lock and holds it across the call into the VM. In 1.5.0:

// ContainersService.swift
private let lock: AsyncLock                       // one for every container

public func bootstrap(id: String, ...) async throws {
    try await self.lock.withLock(...) { context in
        ...
        try await runtimeClient.bootstrap(...)    // whole VM boot, lock held
    }
}

public func startProcess(id: String, processID: String) async throws {
    try await self.lock.withLock(...) { context in
        ...
        try await client.startProcess(processID)  // lock held here too
    }
}

That explains both queues. Boots go through the lock one at a time, half a second each. A VM that has finished booting then asks to start its command, gets in line behind the boots already waiting, and waits for them. The per-container helper underneath has its own lock too, but there's one helper per container, so that one doesn't block other containers. exec into a running VM goes through the same lock when it starts its process, but holds it only for a quick call into a VM that's already up, which is why execs overlap well.

The lock is there to keep the service's table of containers consistent, and that needs protecting. It just doesn't need to be held during a half-second boot. The usual fix is to mark the container as "starting" under the lock, release it for the slow call, and take it again to record the result, or to lock per container. I didn't find an existing issue about concurrent starts, and I haven't tried patching it yet.

• A fresh Linux VM from this tool costs about 0.6 s, and memory, CPU count, image size and networking barely move it. The kernel boot is under 0.1 s of that.
• Once a VM is up, a new command in it costs about 50 ms, the same as starting Node. If you need many short-lived jobs, keep a few VMs running and exec into them.
• Starting VMs in parallel buys nothing in 1.5.0: one service-wide lock is held across each boot and each process start, so k VMs take k × 0.6 s.

Limits: one machine, one tool version, an idle host, and a small Alpine command; a bigger program inside the VM adds its own startup on top. The phase split relies on guest uptime starting near when the host releases the VM, so the 207 ms of host setup is an estimate good to about 10 ms. The source check is a reading of the code that matches the measurements; the patch test is still to do.

Update 2026-10-08: I patched it. With one lock per VM, 32 VMs start in 5 s instead of 20: One Lock Per VM.