Think, level 11
🎲 Probability and Base Rates
If a test is right most of the time, can you trust a positive result?
- If a test is right most of the time, can you trust a positive result? The surprising answer is that it depends on how common the thing you are testing for really is, and that background frequency is called the base rate. Imagine a toy factory where, on average, 1 toy in every 100 comes off the assembly line with a hidden crack. The factory purchases an electronic scanner that inspects every single toy, and whenever a toy really is cracked, the scanner detects it. Unfortunately, the scanner also makes mistakes with undamaged toys, because approximately 1 good toy in every 20 gets flagged as cracked even though it is perfectly fine. Mistakes like these are called false alarms, or false positives, and the scanner sounds quite accurate, so if it flags a particular toy, what is the probability that the toy is actually cracked?
- The clearest way to answer the question is to count, using round numbers instead of a formula. Picture 1,000 toys coming off the assembly line. Since 1 toy in 100 is cracked, approximately 10 of them are cracked, and the scanner flags all 10 of those. That leaves 990 undamaged toys, and the scanner mistakenly flags about 1 in 20 of them, which comes to roughly 50 good toys, because 990 divided by 20 is about 49.5. Altogether, then, the scanner flags about 60 toys, but only 10 of them are genuinely cracked, which means that a flagged toy has only about a 1 in 6 chance of being cracked, or roughly 17 percent. Even though the scanner never misses a cracked toy, most of the toys it flags are fine, because the undamaged toys vastly outnumber the cracked ones, and so the false alarms outnumber the real problems.
- Forgetting about the base rate is a common mistake, and it is sometimes called base rate neglect. People notice that a test is usually right and assume that a positive result must be right too. But when the thing being tested for is rare, even a small false alarm rate, applied to an enormous number of ordinary cases, can produce more false alarms than true discoveries. Mathematicians handle this situation with conditional probability, which is the probability of one event given that another event has already happened, and with a formula called Bayes's theorem. You do not need the formula to understand the central idea, though. Whenever you hear that a test or a warning system is accurate, ask two questions: how rare is the thing it is looking for, and how often does it raise a false alarm? Counting out of 1,000, as we did here, is often the simplest way to find the real answer.
Try it at home: Make up a rare event together, like 1 golden ticket in 100 boxes, and a checker that is wrong 1 time in 20, then count it out of 1,000.
Questions and answers
Probability and Base Rates
- What is a base rate? How common the thing being tested for is (The base rate is the background frequency, like 1 cracked toy in 100.)
- What is a false positive? A flag on something that is actually fine (A false positive is a false alarm.)
- 🧸 In the factory, how often does the scanner wrongly flag a good toy? About 1 in 20 (About 1 good toy in every 20 is flagged even though it is fine.)
- Of the 990 good toys, about how many does the scanner wrongly flag? About 50 (990 divided by 20 is about 49.5, so about 50.)
- Why are most of the flagged toys actually fine? Good toys vastly outnumber cracked ones (A small false alarm rate on many good toys adds up to more alarms than real cracks.)
- What is ignoring the base rate sometimes called? Base rate neglect (Forgetting how rare something is, and trusting the test alone, is base rate neglect.)