The Score Rose One Line Too Late

In The Accusers Vouched for Each Other I simulated SWIM failure detection as HashiCorp's memberlist implements it, and found a way for healthy nodes to be declared dead. A node that's cut off from the network but keeps its timers running suspects its peers in silence. When several such nodes reconnect together, they count each other's suspicions as independent confirmations, the shrunken timeout is already overdue, and memberlist issues the death verdict on the spot, before the suspect can hear the accusation and refute it. Two small changes appeared to fix it. I ended that post saying all of it was a simulation and needed checking against the real library. This is that check.

Short version: the real library has the leak, at about the same outage lengths. One of my two fixes didn't work as written, for a reason the simulation couldn't show. A corrected version, together with the other fix, took false deaths to zero in every case I ran.

The setup

I ran 128 copies of memberlist v0.7.0 in one Go process, joined into one cluster, over an in-memory network I wrote for the test. Memberlist lets you plug in your own transport, so the library code is untouched: its probes, gossip, suspicion timers, TCP fallback ping and periodic full-state sync all run as shipped. The transport can cut nodes off. While a node is cut off, every UDP packet to or from it is held and delivered when the cut-off ends, and every TCP connection to or from it hangs until its timeout and fails. The node's own timers keep running. That's the "blackout" from the first post: think of a VM whose network is stalled, or a rack whose uplink drops.

Settings are memberlist's LAN defaults, except the suspicion multiplier, which I set to 5 (the default is 4) to match the Lifeguard paper and my simulation. Each run waits for the cluster to converge, cuts off 1, 8 or 32 nodes at once for a set time, then watches for 90 seconds after they reconnect. To fit more runs in, every interval and timeout runs 4× faster than real time, and outage lengths are reported in uncompressed seconds. I checked that compression doesn't matter where I could: with 10 nodes crashed, the median time for the first node to declare a crash was 11.9 to 12.5 seconds per run at real speed and 12.1 to 12.9 seconds at 4×. My simulation said 12.5. A run with an outage of practically zero length produced no false deaths.

The count I report is the same as before: healthy nodes declared dead by healthy nodes. Nodes that were actually cut off also get declared dead when the outage is long enough, which is correct, and I don't count it.

The leak is real

With memberlist as shipped, false deaths per outage (mean of 3 runs):

A single false death usually shows up as dozens in this count: when one healthy node is wrongly declared dead, most of the other healthy nodes hear about it and mark it dead too, until it refutes.

One node alone is safe until the outage passes Lifeguard's maximum suspicion timeout of 63 seconds, as designed, and as the simulation found (zero at 50 seconds, failing at 65). Nodes cut off together start failing much earlier: at 24 seconds for 8 of them and 16 seconds for 32. The simulation put those at 33 and 14 seconds. It ran outages back to back with 4 seconds between them, and these are single outages, so I'd compare where the failures start rather than the counts.

The mechanism is the one from the simulation. I logged every time a cut-off node's suspicion timer declared a healthy node dead. For outages of 50 seconds or less, with 8 or 32 nodes cut off, there were 374 such verdicts. Every one of them came after at least one confirmation from another node, and 373 fired within 2 seconds of the outage ending. That's the timer in memberlist's Confirm discovering it's already overdue and firing on the spot.

For comparison, turning Lifeguard off (fixed suspicion timeout, no probe slowdown) produces 568 false deaths per outage from a single node cut off for 16 seconds. Lifeguard is a big improvement; it just doesn't hold for correlated outages.

A fix that didn't work as written

The two changes from the first post were:

• Grace: when a confirmation arrives and the shrunken timeout has already passed, wait 2 more seconds instead of firing at once, so the suspect can hear the suspicion and refute it.
• Health-scaled timeout: multiply a node's suspicion timeout by its own health score + 1, the multiplier Lifeguard already applies to its probe timing. A cut-off node's probes all fail with no "I tried" replies from its helpers, so its score hits the maximum within a few probes and its suspicions wait up to 8 times as long. Healthy nodes have a score of 0 and are unaffected.

I patched both into a copy of the library, behind switches. Grace worked as expected: with 8 or 32 nodes cut off it removed nearly all false deaths up to 50-second outages (at most 3 per outage) and did nothing beyond 63 seconds, where suspicions run out during the outage itself. The health-scaled timeout barely helped. Across 54 outages from 24 to 90 seconds it cut false deaths from 43,233 to 19,016, and a single node cut off for 65 seconds still caused a false death, which the simulation said couldn't happen.

The reason is the order of two lines. When a probe fails, memberlist's probeNode marks the target as suspect, which creates the suspicion and its timer, and only afterwards, in a deferred call on the way out of the function, applies the health penalty for that failed probe. So the first suspicion a cut-off node raises is created while its score is still 0 and gets the ordinary 63-second timeout. The penalty lands a moment later. My simulation updated the score first, so it never saw this.

The fix is to apply the health scaling when the timer fires rather than when the suspicion is created. If the node is unhealthy at that moment and the scaled timeout hasn't passed, re-arm the timer for the difference:

s.timeoutFn = func() {
    if sc := s.scale.Load(); sc != nil {
        total := remainingSuspicionTime(s.n.Load(), s.k, 0, s.min, s.max)
        if extra := (*sc)(total) - time.Since(s.start); extra > 0 {
            s.timer.Reset(max(extra, PatchGrace))
            return
        }
    }
    fn(int(s.n.Load()))
}

Because both the timer and the instant-fire path in Confirm go through this function, both are covered.

Results

Mean of 3 runs per point. Dotted line: Lifeguard's 63-second maximum suspicion timeout. The green line sits at zero throughout. Health scaling set at creation was not run at 130 seconds.

54 outages: 1, 8 or 32 nodes cut off, 6 lengths, 3 runs each. Last column: mean per outage.

Checking health at firing on its own leaves a small leak with many nodes and long outages. I didn't trace those leftovers; the grace removed all of them. With both, no healthy node was declared dead in any of the 63 runs, out to 130-second outages.

The cost is in detection of real crashes, since the health score also rises when packets are lost. With 10 crashed nodes per run and 3 runs (30 crashes per cell), the median time for the first node to declare each crash:

A fraction of a second on the median, and up to 4 seconds on the slowest of 30. That's small but not zero, and 30 crashes per cell can't pin down the tail. Every crashed node was eventually seen as dead by all 118 surviving nodes in every run.

One result didn't carry over from the simulation, in the reassuring direction. The simulation had Lifeguard falsely killing 0.16 nodes per node-hour under 40% packet loss. The real library, over 10 minutes at 30% and 40% UDP loss (3 runs each, about 64 node-hours per setting), had none, with or without the changes. The difference is memberlist's TCP fallback ping, which I left out of the simulation and which my test network never drops. Real TCP on a link losing 40% of its packets would be slow but would mostly get through, so I believe the direction of this, but not the exact zero.

What I'd take from it

• The simulation got the problem right: correlated cut-offs make Lifeguard declare healthy nodes dead well before its 63-second ceiling, through confirmations that arrive overdue.
• It got one fix subtly wrong. Fixes depend on ordering that a model can quietly get "right" when the real code doesn't. Here the health penalty for a failed probe is applied a few lines after the suspicion it should have affected.
• If a decision is meant to depend on a node's own health, read the health when the decision is made, not when the timer was set.
• Testing against the real library took a transport, a harness and about 40 minutes of compute. That's cheap compared to trusting a model of it.

Caveats: one process on one machine, with an in-memory network and time compressed 4×, checked against real speed only on crash detection. A cut-off here holds packets and delivers them late; a network that drops them instead would deliver fewer of the stale suspicions, but the cut-off node would still gossip them after reconnecting. Three runs per point is enough to see where the failures start but not for tight error bars. With the switches off, the patched library passes memberlist's own suspicion and awareness tests.