Standard Frequentist Sample Size Calculator
Calculate your estimated sample size and runtime based on the estimated lift percentage (the lift required to make a decision). Additional inputs are baseline conversion rate, traffic volume to your experience over the course of 30 days, the number of variations or treatments, the number of tails (1 or 2), confidence level, and power level. Outputs are sample size, runtime (if less than 30 days), and an error risk visualization.
Special thanks to Merritt Aho for his work on the updated version of this calculator.
This calculator is the standard Frequentist-based sample size calculator you’re likely familiar with around the web. If you’re not sure what to enter for the estimated lift %, but you DO know how much time you have to make a decision your you have a desired runtime, start with our Runtime-based calculator to get an estimated lift % first, then do a gut check to see if that seems reasonable based on what the natural fluctuations of that metric has seen in the past or how you’ve been able to influence it with an experiment historically. If it’s possible - go for it! If not, you may need to rethink your experimental design. Can you reduce the number of variations? Can you increase the traffic by moving up the funnel? (Note - do not buy more traffic! That traffic may not be representative of your normal audience!) Can you adjust your risk tolerance by adjusting your confidence levels? Can you adjust the experimental design by targeting a completely different metric closer to the intended treatment? If all of those things fail, it might be time to consider planning for a Sequential Design test - where you plan to peek and end your test early if you cross a decision barrier if you under (or over) perform to expectations.