On this page
Probability has an excellent sense of humour. Usually at our expense.
Suppose you’re tested for a rare condition. The result comes back positive. The test has 99% sensitivity and 99% specificity. How worried should you be?
Most people hear those percentages and picture a 99% chance of having the condition. That would be a spectacularly bad interpretation.
Imagine the condition affects one in 10,000 people. Test 100,000 people, for simplicity. About 10 actually have it; a 99%-sensitive test correctly flags roughly 9.9 of them. Of the other 99,990 people, a 99%-specific test incorrectly flags roughly 999.9.
Among the roughly 1,010 positive results, only about 10 correspond to real cases. The positive predictive value is 9.9 / (9.9 + 999.9) ≈ 0.98%, before any further information about the individual is considered. These fractional counts are expectations in a hypothetical population, not a prediction that every real sample will contain exactly this mix.
Nothing is broken. The test performed exactly as specified. The mistake was ours: we forgot to ask how common the condition was in the first place.
“Before asking how good the test is, ask how common the thing is.” That’s base-rate reasoning, one of the most practical applications of conditional probability. It’s also a good introduction to a larger, occasionally humiliating idea: the numbers can be correct while the story we tell about them is wrong.
Probability isn’t the same as a verdict
The rare-condition example is a version of Bayes’ theorem: what you conclude from new evidence depends on what was plausible before it arrived, and on the chances of seeing that evidence under competing explanations.
It is also why scientific findings need context. If researchers test enormous numbers of highly speculative hypotheses, even statistically significant results may include many false positives. John Ioannidis explored these mechanisms in a much-discussed 2005 model. His conclusion that most published findings are false is not itself a universal theorem or an empirically established percentage; Goodman and Greenland questioned that stronger interpretation. But the basic base-rate mathematics survives the disagreement.
This leads to the advice “Never be certain of anything you could be wrong about.” Statisticians know it as Cromwell’s rule: don’t assign a contingent, learnable proposition a probability of exactly zero or one without extraordinary justification.
Why? If your prior probability for a hypothesis is exactly zero, ordinary Bayesian updating cannot lift it above zero, no matter how striking the evidence. A numerical certainty about the empirical world can become a refusal to learn. Logical impossibilities are a different matter; they don’t need a polite sliver of probability just in case.
Yet refusing certainty doesn’t mean treating every possibility as equally likely. It means leaving room to update in proportion to evidence.
And what if the evidence never appears?
You’ve probably heard “Absence of evidence is not evidence of absence.” Sometimes that’s right. Sometimes it’s a dreadful excuse.
If you’ve barely looked for your keys, failing to find them tells you little. If you’ve thoroughly searched every plausible place, the same failure becomes informative. Formally, a negative result counts against a hypothesis when that negative result is less likely if the hypothesis is true than if it’s false.
So a more useful version is “Absence of evidence can be evidence of absence—if you would have expected to find it.” The strength of the evidence depends on the search procedure, not on the rhetorical confidence of the person doing the searching.
Extreme days and miraculous coincidences
There’s another way our intuitions fail. We think events are meaningful because they are unusual, without accounting for how many chances there were for something unusual to happen.
Take “Extreme results are usually followed by ordinary ones.” Francis Galton described regression toward the mean while studying inherited height, in work published in 1886.
Imagine a salesperson whose performance is a mixture of ability and luck. On an extraordinarily good month, the result probably includes some unusually good luck. If luck does not repeat, the next month will tend to be closer to the person’s usual level, even if nothing about their ability or effort changes.
That’s why managers can convince themselves that scolding someone after a terrible month works, or that praising someone after an excellent month backfires. In both cases, ordinary statistical reversion may be doing much of the work.
Regression to the mean isn’t magic, and it doesn’t govern every time series. It arises when repeated measurements are imperfectly correlated. If a process has no random variation—or if an extreme outcome permanently changes the underlying system—the simple story may fail.
Now scale the number of opportunities in the opposite direction.
“With enough chances, the miraculous becomes ordinary.” Persi Diaconis and Frederick Mosteller studied the mathematics of coincidence in their 1989 paper. The law of truly large numbers is a persuasive probabilistic principle, not a promise that every imaginable event must occur.
Suppose an event has an independent one-in-a-million chance on each attempt. After a million attempts, the chance of at least one occurrence is about 63%. After ten million attempts, it’s about 99.995%.
The arithmetic is simply:
P(at least one) = 1 − (1 − p)^n
Of course independence is an assumption, and real-life coincidences rarely offer a neat fixed probability. But the mechanism matters. Millions of people live through millions of moments. Some will experience coincidences that feel astronomically unlikely when considered after they happen.
The trap is choosing the astonishing event first, then forgetting the enormous universe of other events that could have astonished us instead.
Your friends are not a random sample of people
Speaking of misleading perspectives, here’s an especially cruel mathematical observation: “Your friends have more friends than you do.”
Not necessarily each friend. And not necessarily you personally. But the average across a social network has a systematic bias.
Scott Feld showed this in his 1991 paper on the friendship paradox. People with many connections turn up in lots of other people’s friend lists. People with very few connections appear in fewer. So if you sample people through their friendships, highly connected people are overrepresented.
For an undirected network with positive average degree, choose a friendship uniformly at random, then choose one of its two endpoints with equal probability. The mean degree of the person sampled this way is:
E[degree of a friend] = E[degree²] / E[degree]
This is at least as large as the average degree of a uniformly selected person, and is strictly larger when degrees vary. Sampling a random person first and then one of their friends is a different procedure; the formula above does not generally describe it. That is a mathematical statement about sampling in a network; the impression that everyone else has a better social life is the psychological extrapolation.
The same size-biasing logic helps explain why a crowded place can seem more representative than it is, or why your online feed makes particular experiences seem universal. You’re not looking through a neutral window. You’re looking through a sampling mechanism.
And sometimes the mechanism doesn’t merely exaggerate a pattern. It can reverse one.
Two statistics can tell opposite stories without either lying
In the autumn of 1973, graduate admissions at the University of California, Berkeley appeared to show a striking pattern: a higher proportion of men were admitted than women.
Looking more closely at admissions by department told a different story. Women were disproportionately applying to departments with lower acceptance rates for everyone. Once departmental differences were considered, the university-wide figures did not support the simple conclusion that admissions committees were systematically discriminating against women.
This is the famous Berkeley case examined by Bickel, Hammel and O’Connell in 1975.
“The whole can tell the opposite story of its parts.” Simpson’s paradox describes situations in which an aggregate association changes, or even reverses, when data is partitioned into relevant groups.
This does not mean that the answer is always to break data into smaller buckets. Conditioning on the wrong variable can create its own distortions. And explaining a statistical pattern by department does not settle every question about earlier inequalities that influenced which departments people applied to.
It means the causal question comes first: What are we trying to compare, and what determined which group each observation entered?
There’s an even sneakier cousin.
Imagine a company recruits candidates who are exceptionally good at programming or exceptionally good at communication. In the general population, suppose those two skills are independent. But among candidates who pass that initial filter, someone weak at programming must usually be strong at communication to make the cut—and vice versa.
Suddenly the two skills appear negatively related, even though no such relationship existed in the population. You’ve created a correlation simply by selecting on a shared outcome.
This is “If you only see a filtered group, you’ll find links that don’t exist,” the intuition behind Berkson’s paradox. Joseph Berkson described the problem in hospital samples in 1946. In modern causal language, conditioning on a collider can induce an association between otherwise independent causes. The direction and size depend on how selection works; negative correlation isn’t guaranteed by every filter.
A dataset can be accurate, complete for the people it includes, and still be misleading about everyone outside it.
Ask how the numbers got there
All these ideas lead to a fairly humbling conclusion. A number does not come with its interpretation attached.
Before deciding what a probability, average, correlation, or significant result means, we need to know where the observations came from, which observations were missing, what population we’re talking about, and how the measurement was produced.
This is why I find the mathematics so comforting, in a strange sort of way. It doesn’t eliminate uncertainty or guarantee that I’ll reach the right conclusion. It shows me exactly where intuition is liable to take an expensive shortcut.
The next time a statistic seems to settle an argument, I’ll probably ask one more question before believing it.