All insights
GrowthAnalytics · 5 min read
The conversion lift that disappeared at scale
A 15% lift in a test cohort that became 0% at full rollout is one of the most common, and least understood, patterns in growth.
The test showed a 15% conversion lift. You ship to 100%. Three months later, the overall conversion rate is exactly where it was before the test. This pattern is so common it has a name, and most teams discover it the hard way.
What causes it
- Cannibalization, the lift came from users you’d have converted later anyway.
- Segment effects, the test cohort over-indexed on users for whom the variant worked.
- Selection bias, the experiment’s audience was different than the rollout’s.
- Operational drift, by the time of rollout, other factors had changed.
How to catch it earlier
Run the experiment longer than statistical power requires, observe the steady state, not just the peak. Hold out a control even after rollout, you’ll see whether the lift survives. Most teams disband the holdout because it feels wasteful; it’s actually the most valuable thing you can keep running.
If you don’t keep a holdout after rollout, you’ll never know which of your shipped wins were real.