All insights
GrowthAnalytics5 min read
The growth experiment that taught you nothing
An experiment that runs and produces a clean result might still teach you nothing — if the question it answered wasn't the question you were really asking.
You run an experiment. The result is clear: variant B beats variant A by 3%, p < 0.05. You ship variant B. A quarter later, the metric the experiment was supposed to move is flat. The experiment lied — or rather, it answered the wrong question.
Common ways experiments lie
- The experiment optimizes a proxy metric that doesn't drive the real metric.
- Novelty effect — the lift disappears when the variant becomes normal.
- The experiment was underpowered; the result was noise.
- The segment that was lifted wasn't the segment you cared about.
The discipline
Write down what you expect to learn before running the experiment. Be specific. Decide which result would update your beliefs how. Most experiments shipped today don't pass this bar — they're tests of "does this thing work" without a clear theory of what "working" would mean.
An experiment that doesn't change your beliefs taught you nothing — even if the p-value was beautiful.