What to do next
Show the working, step by step
Where I used this
At Praava Health I built a data engine for product and growth decisions, behind about 57% compound annual growth in the consumer business.
Read the case: A data engine for product and growth decisions →Everything is calculated in your browser. Nothing you type is sent or stored.
Three ways A/B tests mislead
- Peeking. Checking every day and stopping the moment it looks significant inflates false wins. Decide the sample size first, then wait for it.
- Too-short tests. Run at least one full week, ideally two, so weekday and weekend behaviour are both in the data.
- Winning the wrong metric. A higher click rate that brings lower-value customers is not a win. Check revenue per visitor too.
Get new free tools by email. One short note when a new tool or essay goes live. No spam, unsubscribe any time.
Not sure your test result is real?
Bring the numbers to a free 30-minute call, and I will help you decide whether to ship, keep testing or stop.
Questions
What confidence level should I use?
95% is the common standard. Use 90% for low-risk changes you can easily reverse, and 99% for expensive or hard-to-undo decisions.
What does statistical significance mean here?
That a difference as large as the one you saw would be unlikely if A and B actually performed the same. It does not say the lift is big enough to matter.
How is the required sample size calculated?
For a two-sided test at your chosen confidence and 80% power, using the conversion rates you entered. Smaller lifts need much larger samples.