The A/B Testing Significance Calculator helps you assess the statistical significance of an A/B testing experiment on the conversion rate of two different landing pages (namely, 'Control' and 'Variation'). A/B testing is typically conducted by randomly dividing a sample group of users into two groups of similar size, with each group being shown a different version of the asset being tested (here, the Control and Variation pages). Overall, A/B testing is a valuable tool for businesses to optimize their marketing efforts and make data-driven decisions that lead to improved performance and growth. The ultimate goal is to understand whether the higher or lower conversion rate detected on the Variation page is truly attributable to changes that you made on the page, or it is just the effect of randomness. To use it, all you need to input is:
Total number of visitors to the Control page
Total number of conversions on the Control page
Total number of visitors to the Variation page
Total number of conversions to the Variation page
The calculator will compute:
The conversion rate on both pages (CRC and CRV) as the ratio between
conversions/visitorsThe standard error of both sample distributions, as
SQRT[CR*(1-CR)/Visitors]The statistical significance at 90%, 95 and 99%, namely checking whether the difference between CRC and CRV is significantly different from 0 at different levels of confidence.
From a statistical standpoint, the calculator assumes that converting on the page can be modeled as a discrete Bernoulli distribution ('convert' or 'not convert') with mean CR and standard deviation SQRT[CR*(1-CR)/n], where n is the size of the sample, here the number of visitors driven to the page.
Trying to investigate whether there is a (statistically) significant difference in the two Conversion Rates is equivalent to testing whether the mean of the difference between two Bernoulli experiments (with respective mean CRC and CRV) is (statistically) different from zero.
Using the Central Limit Theorem (CLT), which states that as the sample size increases, the distribution of the sample means approaches a normal distribution with mean μ, regardless of the distribution of the underlying population, the calculator generates three intervals to test the difference (CRC - CRV) against. Each interval has the following upper and lower bounds:
Upper bound =
(CRC- CRV) + z * SQRT[CR*(1-CR)/n]Lower bound =
(CRC- CRV) - z * SQRT[CR*(1-CR)/n]
where z represents the z-score corresponding to the confidence level chosen, i.e. the value of a normal distribution N(0,1) with mean 0 and standard deviation 1 that leaves 'outside the interval' (both right and left tails) respectively a probability of 10%, 5%, and 1%.
It's worth mentioning the fact that all the above results are reliable only if the two samples (the number of visitors to the Control and Variation pages) are not too far apart. In that case, a Sample Ratio Mismatch may arise, leading. Intuitively, a sample of 100 visitors to the Variation page vs 500 on the control page is way more subject to self-selection bias and thus might lead to false positive results.
As a rule of thumb, a difference higher than 5% should raise a concern about the way Visitors are split into the two pages and, consequently, be addressed with a more equal distribution.