All insights
GrowthAnalytics5 min read
The conversion lift that disappeared at scale
A 15% lift in a test cohort that became 0% at full rollout is one of the most common — and least understood — patterns in growth.
The test showed a 15% conversion lift. You ship to 100%. Three months later, the overall conversion rate is exactly where it was before the test. This pattern is so common it has a name — and most teams discover it the hard way.
What causes it
- Cannibalization — the lift came from users you'd have converted later anyway.
- Segment effects — the test cohort over-indexed on users for whom the variant worked.
- Selection bias — the experiment's audience was different than the rollout's.
- Operational drift — by the time of rollout, other factors had changed.
How to catch it earlier
Run the experiment longer than statistical power requires — observe the steady state, not just the peak. Hold out a control even after rollout — you'll see whether the lift survives. Most teams disband the holdout because it feels wasteful; it's actually the most valuable thing you can keep running.
If you don't keep a holdout after rollout, you'll never know which of your shipped wins were real.