A/B Test Calculator

Evaluate an A/B test: significance, power and the Bayesian reading

Two versions of a page, two random halves of your visitors, two conversion rates at the end. The problem: two identical pages will also produce different numbers, purely by chance. This calculator separates real winners from lucky ones, in both ways statistics knows.

The frequentist view gives you the classic test protocol: conversion rates, relative uplift, z-score, p-value, observed power and the distribution curves of both variants. The Bayesian view answers the question you are actually asking: how likely is B truly better than A? It shows the likely uplifts as a distribution and, if you want, works out the business case on the spot: expected upside, expected risk, and the risk-weighted extra revenue of rolling out B.

Before the test: plan your sample size

Most A/B tests fail not on the idea but on the sample. If you want to detect a small uplift on a low-traffic page, you need more visitors than you will see in a quarter.

The planning mode does the sums: enter today's conversion rate and the improvement you expect, get the minimum sample per variant. With your weekly traffic that becomes a test duration, and if you only have a limited number of weeks, the calculator tells you the smallest improvement you could even detect in that time.

Frequently asked questions

What does statistically significant mean?

Significant at 95% confidence means: if there were truly no difference between A and B, a gap this size would show up in at most 5 of 100 tests. So the difference is very probably your change, not chance.

One-sided or two-sided?

One-sided asks only whether B beats A and is the usual choice for A/B tests. Two-sided asks whether B differs from A in either direction, and demands a bit more evidence. When in doubt: one-sided.

What does the Bayesian number tell me?

The p-value answers a roundabout question (how surprising would this data be if nothing had changed?). The Bayesian probability answers the direct one: how likely is B actually better? Many teams act above 95%.

Why don't the business case numbers wobble here?

Most calculators, including the well-known one at ABTestguide, simulate the business case with Monte Carlo: they roll thousands of random possible true conversion rates and average the revenue outcomes. Correct in principle, but every reload shows slightly different amounts, because the dice are rolled again. This calculator uses the exact formula for that same expected value (the partial expectation of the normal distribution). Same inputs, same number, every time. No simulation noise to second-guess.

Why shouldn't I peek daily and stop at the first green number?

Because the p-value wobbles along the way. Check every day and stop at the first significant reading, and you are fishing out your own false positives. Fix the sample size first, let the test run, judge once.