Skip to content

Statistical Significance Calculator

By Thuong Le · Reviewed

Enter visitors and conversions for A and B, and pick a confidence level. You get the rates, the uplift, the z-score, the p-value and a yes or no.

People who saw version A. Whole numbers only.

Visitors in A who converted.

People who saw version B. Whole numbers only.

Visitors in B who converted.

How sure you want to be before calling the result significant.

Significant?
Yes
p-value (two-sided)
0.0091
z-score
2.61
Relative uplift (B vs A)
24.00%
Conversion rate A
5.00%
Conversion rate B
6.20%

Compare scenariosSave up to three versions and compare them

Scenario A

Scenario B

Scenario C

How it works

This is a two-proportion z-test. It compares the conversion rates of A and B and asks one question: if both versions really converted at the same rate, how often would a gap this big or bigger show up by chance? That chance is the p-value. When it is below 1 minus your confidence level, such as below 0.05 at 95%, the result is called significant. The test uses a pooled standard error, so both groups share one estimate of the spread.

conversion rate = conversions ÷ visitors

uplift = (rate B − rate A) ÷ rate A

pooled rate = (conversions A + conversions B) ÷ (visitors A + visitors B)

standard error = sqrt(pooled rate × (1 − pooled rate) × (1 ÷ visitors A + 1 ÷ visitors B))

z = (rate B − rate A) ÷ standard error

p-value = 2 × (1 − normal CDF of |z|)

Worked example

Example numbers, not benchmarks: version A had 5,000 visitors and 250 conversions, version B had 5,000 visitors and 310 conversions, and the confidence level is 95%.

  1. 1Conversion rate A250 × 100 ÷ 5,000 = 5.00%
  2. 2Conversion rate B310 × 100 ÷ 5,000 = 6.20%
  3. 3Relative uplift(6.20% − 5.00%) ÷ 5.00% = 24.00%
  4. 4Pooled rate(250 + 310) × 100 ÷ (5,000 + 5,000) = 5.60%
  5. 5z-score(rate B − rate A) ÷ pooled standard error = 2.61
  6. 6p-value2 × (1 − normal CDF of |2.61|) = 0.0091
  7. 7Significant?0.0091 is below 0.05 = Yes

What does significant mean?

Yes means the p-value is below your cutoff, so a gap this large would be unusual if A and B truly converted at the same rate. No means the data do not rule out chance at that level. It does not show the two versions are equal. The test assumes each visitor is counted once and that you chose the sample size before looking. Checking again and again and stopping when the numbers look good breaks that assumption. The normal approximation is also less reliable when conversions or non-conversions are very few.

Questions

What is a p-value?

The chance of seeing a gap at least this large between A and B if the two versions really converted at the same rate. A small p-value means that would be unusual. It is not the chance that B is better.

What does the confidence level change?

It sets the cutoff. At 90% the p-value must be below 0.10, at 95% below 0.05 and at 99% below 0.01. A higher level asks for stronger evidence, so the same data can pass at 90% and fail at 95%.

Why does it say No when B has a higher rate?

A higher rate in one sample can happen by chance, especially with few visitors. No means the gap is not big enough, for the visitors you have, to pass the cutoff you chose.

What is relative uplift?

The change from A to B as a share of A's rate. With A at 5.00% and B at 6.20%, the uplift is 24.00%, while the gap is 1.20 percentage points.

Why are the z-score and p-value blank?

The test needs visitors above 0 in both groups, and at least one conversion and one non-conversion across them. If nobody converted, or everybody did, there is no spread to measure and the test is not defined.

More in Marketing & SEO

Often used together