A noisy judge breaks the 37% rule, and no cutoff fixes it
The secretary problem: n candidates arrive in random order, you see them one at a time, and you must accept or reject each on the spot. If you want the single best one, the classic answer is to reject the first 37% (1/e), then take the first candidate who beats everyone so far. That gets the best with probability about 1/e, whatever n is.
The proof assumes you rank candidates perfectly. Real judgements are noisy: an interview, a flat viewing, a tasting. I wanted to know how much noise costs, and whether a different cutoff wins some of it back.
The run
Each candidate has a true quality drawn from a standard normal. You see the quality plus normal noise with standard deviation ฯ, so ฯ = 1 means the noise is as large as the real spread between candidates. You apply the rule to what you see. I ran pools of 10, 30, 100, 300 and 1,000 candidates at ten noise levels from 0 to 3, with 1 to 4 million trials per setting, and scored every possible cutoff in each trial. Standard errors on the probabilities below are 0.0006 or less.
Two checks came out right. With no noise, the best cutoff for 10 candidates is 3 or 4 (39.9%), and for 1,000 it's 37% (36.7%), matching the textbook values. And at every noise level the rule picks the candidate who looked best 36.7โ37.1% of the time. Noise doesn't affect how the rule behaves, only whether the one who looked best really is.
The odds fall fast
Chance of ending up with the truly best candidate, using the 37% cutoff, then using the best cutoff for that noise level. The second column is the correlation between what you see and the truth.
| noise ฯ | correlation | 100 candidates, 37% | best cutoff | 1,000 candidates, 37% | best cutoff |
|---|---|---|---|---|---|
| 0 | 1.00 | 37.1% | 37.1% at 37% | 36.7% | 36.7% at 37% |
| 0.1 | 0.995 | 33.8% | 33.8% at 36% | 32.3% | 32.4% at 34% |
| 0.3 | 0.96 | 27.0% | 27.2% at 32% | 23.6% | 23.9% at 31% |
| 0.5 | 0.89 | 21.0% | 21.3% at 31% | 16.5% | 16.8% at 29% |
| 1 | 0.71 | 11.5% | 11.9% at 26% | 6.7% | 7.0% at 25% |
| 2 | 0.45 | 5.0% | 5.3% at 22% | 1.8% | 2.0% at 21% |
| 3 | 0.32 | 3.2% | 3.4% at 20% | 0.8% | 0.9% at 19% |
Noise you'd hardly notice costs a lot. A correlation of 0.96 is a very good judge, and it takes the odds from 37% to 27% at 100 candidates and 24% at 1,000. Bigger pools get hurt more, because there are more near-equals who can edge past the true best on the day.
The best cutoff does move earlier as noise grows, from 37% down to about 20%. But retuning gains almost nothing: at most 0.4 percentage points at 100 or more candidates, and 0.7 at 10. The curve of success against cutoff is flat near its top, so the old cutoff stays close to optimal even when the optimum has moved. If your judgement is noisy, a cleverer stopping rule won't help. Only better judgement will.
The goal matters more than the noise
"Best or nothing" is a strange goal anyway. Most people would rather end up with someone good than gamble on the very best. If you score a pick by its true quality instead, 37% is far too patient:
| what you score | candidates | at 37% cutoff | at best cutoff |
|---|---|---|---|
| mean quality, in standard deviations | 100 | 1.44 | 1.72 at 14% |
| mean quality, in standard deviations | 1,000 | 1.94 | 2.48 at 9.4% |
| mean percentile | 100 | 81% | 91% at 9% |
| mean percentile | 1,000 | 82% | 97% at 3.1% |
Waiting through 37% means that 37% of the time the best has already gone by, nobody beats them, and you're forced to take the last candidate, who is average. The percentile cutoff of 3.1% at 1,000 is about โn candidates, which matches Bearden's 2006 result that โn is the best cutoff when the payoff is the candidate's rank. Using 37% when you care about quality is a bigger mistake than any amount of noise up to ฯ = 0.5.
For mean quality, noise has an exact effect. With normal quality and normal noise, the best estimate of the true quality is the observed score divided by 1 + ฯยฒ, and the observed scores are just normal with a wider spread. So every cutoff's mean quality shrinks by the same factor, 1/โ(1 + ฯยฒ), and the best cutoff doesn't move at all. The simulation agrees with that to within 0.002 at every size and noise level, which is a good check on the code. The percentile score is different. Its best cutoff rises with noise, from 3.1% to 5.5% at ฯ = 1 for 1,000 candidates, because an early lucky-looking candidate is less likely to be really good.
So
The 37% rule holds up under noise in a narrow sense: no other cutoff does much better at getting the very best. But noise makes "the very best" nearly unreachable in a big pool, and the rule was never the right one if you'd take "very good". Look at fewer candidates before committing, roughly the square root of the pool, and spend the effort on judging them well.