A/B Testing Significance Calculator

Calculate the statistical significance of your A/B testing experiment.

The A/B Testing Significance Calculator helps you assess the statistical significance of an A/B testing experiment on the conversion rate of two different landing pages (namely, 'Control' and 'Variation'). A/B testing is typically conducted by randomly dividing a sample group of users into two groups of similar size, with each group being shown a different version of the asset being tested (here, the Control and Variation pages). Overall, A/B testing is a valuable tool for businesses to optimize their marketing efforts and make data-driven decisions that lead to improved performance and growth. The ultimate goal is to understand whether the higher or lower conversion rate detected on the Variation page is truly attributable to changes that you made on the page, or it is just the effect of randomness. To use it, all you need to input is:

  • Total number of visitors to the Control page

  • Total number of conversions on the Control page

  • Total number of visitors to the Variation page

  • Total number of conversions to the Variation page

The calculator will compute:

  • The conversion rate on both pages (CRC and CRV) as the ratio between conversions/visitors

  • The standard error of both sample distributions, as SQRT[CR*(1-CR)/Visitors]

  • The statistical significance at 90%, 95 and 99%, namely checking whether the difference between CRC and CRV is significantly different from 0 at different levels of confidence.

From a statistical standpoint, the calculator assumes that converting on the page can be modeled as a discrete Bernoulli distribution ('convert' or 'not convert') with mean CR and standard deviation SQRT[CR*(1-CR)/n], where n is the size of the sample, here the number of visitors driven to the page.

Trying to investigate whether there is a (statistically) significant difference in the two Conversion Rates is equivalent to testing whether the mean of the difference between two Bernoulli experiments (with respective mean CRC and CRV) is (statistically) different from zero.

Using the Central Limit Theorem (CLT), which states that as the sample size increases, the distribution of the sample means approaches a normal distribution with mean μ, regardless of the distribution of the underlying population, the calculator generates three intervals to test the difference (CRC - CRV) against. Each interval has the following upper and lower bounds:

  • Upper bound = (CRC - CRV) + z * SQRT[CR*(1-CR)/n]

  • Lower bound = (CRC - CRV) - z * SQRT[CR*(1-CR)/n]

where z represents the z-score corresponding to the confidence level chosen, i.e. the value of a normal distribution N(0,1) with mean 0 and standard deviation 1 that leaves 'outside the interval' (both right and left tails) respectively a probability of 10%, 5%, and 1%.

It's worth mentioning the fact that all the above results are reliable only if the two samples (the number of visitors to the Control and Variation pages) are not too far apart. In that case, a Sample Ratio Mismatch may arise, leading. Intuitively, a sample of 100 visitors to the Variation page vs 500 on the control page is way more subject to self-selection bias and thus might lead to false positive results.

As a rule of thumb, a difference higher than 5% should raise a concern about the way Visitors are split into the two pages and, consequently, be addressed with a more equal distribution.

More than an A/B Testing Significance Calculator

Rows is the easiest way to import, transform and share data in a spreadsheet.

Signup for free

Import data from anywhere

Unleash your data: import from files, marketing tools, databases, APIs, and other 3rd-party connectors.

Know more

Analyze with the power of AI

Unlock the power of AI on your data: ask the AI Analyst ✨ any question about your dataset and surface key insights, trends, and patterns.

Know more

Collaborate and Share

Seamlessly collaborate and share stunning reports with dynamic charts, embed options, and easy export features.

Know more