tomtunguz.com · 2016-05-16
· 3671d
Is A/B Testing A Good Idea for SaaS Startups?
A/B testing requires large user volumes to achieve statistical significance, making it impractical for early-stage SaaS companies that lack sufficient traffic. Early-stage startups should instead rely on qualitative research, user interviews, and expert intuition rather than statistical analysis to inform product, pricing, and design decisions. Statistical analysis becomes viable only at scale, where marginal improvements of 1-5% can justify the extended testing timelines.
Metrics in this report
A/B Test Sample Size Required
12000visitors per variation
To detect a 20% change in a target metric with 90% confidence when affecting 3% of visitors
A/B Test Total Sample Size
24000visitors
Required to test two variations with statistical significance