Standard Frequentist Runtime-based Calculator

This calculator is a runtime-based sample size calculator. The output gives you the minimum lift you’d need in order to achieve the selected risk tolerance settings. If you think that the lift level is far-fetched, and you have flexibility in the runtime, you can increase the runtime. If it would be easy, adjust it down. If it’s possible - go for it! If not, you may need to rethink your experimental design. Can you reduce the number of variations? Can you increase the traffic by moving up the funnel? (Note - do not buy more traffic! That traffic may not be representative of your normal audience!) Can you adjust your risk tolerance by adjusting your confidence levels? Can you adjust the experimental design by targeting a completely different metric closer to the intended treatment? If all of those things fail, it might be time to consider planning for a Sequential Design test - where you plan to peek and end your test early if you cross a decision barrier if you under (or over) perform to expectations. You can also check out the more standard Sample size calculator that uses the lift calculated here as an input and gives the runtime as the output.