All insights
EngineeringTesting5 min read
End-to-end tests at three different scales
End-to-end testing means wildly different things at 10, 100, and 1,000 tests. Most teams scale the approach that worked at one tier into the next, and pay for it.
At ten end-to-end tests, you can run all of them on every PR. They take twenty minutes, they catch a lot, and the team trusts them. At a hundred, you can't anymore — but most teams keep trying. At a thousand, end-to-end has become an unmaintainable graveyard.
What scales
- 10 tests — run all of them on every PR. Iterate freely.
- 100 tests — run a smoke subset on PRs, full suite nightly. Sharded execution.
- 1,000 tests — quarantine flaky tests aggressively. Treat the suite as a portfolio, not a list. Owner per area.
What doesn't scale
Running every end-to-end test on every PR. Pretending all end-to-ends are equally important. Letting tests proliferate without quotas. The number of teams that have tried these is exactly equal to the number of teams who've then dismantled their end-to-end suite.
End-to-end tests have to be managed differently at each order of magnitude. Most teams don't change the approach in time.