On this page
Some answers only become available after you make a move.
Imagine I’m building a small software service. New users arrive, open the dashboard, and leave without finishing their first task. I suspect the introduction is too long. A colleague thinks it needs more explanation.
We can discuss this for an impressively long time. We can draw diagrams, inspect competitors, and find articles supporting both positions. Eventually we have two polished explanations and exactly the same decision to make.
I want to replace the introduction with a short, worked example. The familiar question is whether the change will help. But there’s another question hiding inside it: what would trying it teach us, and would that knowledge be worth the cost?
That’s where I want to finish this series. Once we’ve examined our theories, numbers, objectives, and assumptions, we still have to choose something. Sometimes the choice itself produces the evidence we’ll need next.
The familiar option has an unfair advantage
Suppose the current introduction has years of results behind it. The proposed example has a sketch and some encouraging comments. Choosing the established version looks sensible because its performance is easier to estimate.
But the best option you’ve tried isn’t necessarily the best option available. That’s my shorthand for a practical problem, not a theorem that novelty deserves a chance every time.
The mathematical version appears in multi-armed bandit problems. A decision maker repeatedly chooses between options with uncertain rewards. Exploitation uses what currently looks best; exploration gathers information that may improve later choices. Russo and colleagues’ tutorial on Thompson sampling explains both the trade-off and an algorithm for managing it.
An exploratory choice can have a lower expected immediate payoff and still be worthwhile because of what it teaches. That possibility depends on having future decisions in which the information matters. The time horizon, uncertainty, and costs all affect the answer.
Our software example resembles a bandit problem, but resemblance doesn’t give us its mathematical guarantees. Users change, effects may arrive late, and today’s introduction can affect tomorrow’s behaviour. We would need a model that captures those features before importing an algorithm’s formal claims.
Still, the question helps. If I keep rejecting the new introduction because I know less about it, I may preserve the very ignorance I’m using to justify the rejection.
A change is not yet an experiment
Let’s say I publish the worked example on Monday. Completions rise. Naturally, I congratulate myself.
Then I learn that a newsletter recommended the service on Tuesday. The new visitors already understood what it did. Perhaps the introduction helped. Perhaps the visitors needed less help. My before-and-after comparison cannot separate those explanations on its own.
An experiment needs a comparison that could have proved you wrong. Another formulation of mine; the precise issue underneath it is causal identification.
I’d like to know what would happen to comparable users under each introduction. I cannot observe both outcomes for the same person under the same circumstances. But I can randomly assign eligible new users to the existing introduction or the worked example during the same period.
Under a properly implemented random assignment, treatment assignment is independent of users’ potential outcomes. That supports estimating an average causal effect; it doesn’t guarantee identical groups in a finite sample. Hernán and Robins develop this reasoning in the opening chapters of Causal Inference: What If.
The details still matter. Users must receive the intended version, outcomes must be recorded reliably, and one person’s assigned introduction shouldn’t change another person’s result in ways the analysis ignores. Randomisation doesn’t repair a broken event counter or make a tiny sample precise.
Before starting, I’d specify what completion means, the period over which I’ll measure it, and an analysis plan appropriate to the sample and stopping rule. I’d also check for harms such as more support requests. These are practical design choices, not formal guarantees. Kohavi, Henne, and Sommerfield’s guide to online experiments makes clear how much trustworthy testing depends on implementation, sample size, and evaluation criteria.
Now the two explanations face a test that neither of us gets to rewrite after seeing the answer.
The answer needs somewhere to go
Even a well-designed experiment can answer a question that isn’t worth asking yet.
Suppose a complicated introduction redesign would cost €6,000. For a deliberately simple calculation, assume there are only two possibilities: it will produce €10,000 in additional benefit over the coming year, or none. I currently assign each possibility a probability of one half. Assume those amounts include every relevant consequence and that I maximise expected euros.
Building immediately has an expected net value of 0.5 × €10,000 − €6,000 = −€1,000. Doing nothing has a value of zero, so I would wait.
If I could learn the outcome perfectly before deciding, I’d build only in the favourable case. My expected net value would then be 0.5 × (€10,000 − €6,000) = €2,000.
Perfect information is therefore worth €2,000 in this toy decision. A realistic, imperfect test cannot be worth more as information alone, under those same assumptions. Its cost and delay can make it worth much less.
That’s the distinction I take from Ronald Howard’s Information Value Theory: the importance of uncertainty depends on its consequences for a decision, not simply on how much uncertainty exists.
My version is before buying an answer, name the decision it could change.
If every possible test result leaves the same action optimal, the test has no expected decision value within that model. It may still satisfy curiosity, reassure someone, or inform a later choice. Those are legitimate benefits; they belong in the objective if we care about them.
I like this because it interrupts a very respectable form of procrastination. Another survey, another dashboard, another analysis can feel productive while leaving the next action untouched.
Make the attempt small enough to learn from
Our imaginary team doesn’t need to build the €6,000 redesign to investigate whether examples help. It might first test a single worked example inside the existing introduction.
That smaller trial won’t answer every question about the full redesign. Its evidence concerns the intervention actually tested. But it could make the larger decision clearer at an acceptable cost.
Make the next attempt small enough to afford and clear enough to teach. That’s a heuristic. Some questions require expensive studies; some interventions cannot be meaningfully reduced. A cheap test of the wrong thing isn’t a bargain.
The title needs a qualification too. We learn plenty from other people’s experiments, observation, and reasoning. Nobody needs to touch every hot stove personally. The difficulty arises when the available evidence cannot distinguish the options in front of us, and waiting produces no useful new evidence.
At that point, I want to know what I can try, what the result would mean, and what I’d do differently afterwards.
The worked example might fail. The colleague who wanted more explanation might be right. Either result would give us something our polished arguments couldn’t: a reason to make the next decision differently.