Two-Proportion Test: Comparing Conversion Rates Between Variants
two-proportion test, A/B testing, conversion comparison, effect size, Cohen’s h
TL;DR: Variant A 12.0% vs B 9.5% (n=1,000 each), difference +2.5 pp [−0.2, +5.2 pp], χ²(1) = 3.26, p = .071 (not significant), Cohen’s h = 0.08 (below small). Decision: no clear winner yet. Run a larger test, about 2,410 visitors per group, to settle it.
Answer
Method: Two-proportion z-test.
Estimate: 12.0% vs 9.5%, difference 2.5 pp, CI −0.2, +5.2 pp.
Data: A/B test, variables variant, conversion, n = 2,000 (1,000 per variant).
Action: No significant difference at α = .05. Continue testing with a larger sample.
Case
You ran an A/B test on a product page. After two weeks, variant A had 120 conversions from 1,000 visitors (12.0%), while variant B had 95 conversions from 1,000 visitors (9.5%). The business question: Is the 2.5 percentage point difference statistically significant? Should you keep variant A or run a confirmatory test?
Dataset
Synthetic A/B experiment data (Schema C).
| Variable | Label | Value |
|---|---|---|
variant |
Test group | A or B |
converted |
Conversion event | 0 or 1 |
n |
Sample size | 2,000 rows |
x_A |
Conversions in A | 120 |
x_B |
Conversions in B | 95 |
n_A |
Visitors in A | 1,000 |
n_B |
Visitors in B | 1,000 |
Method
We use a two-proportion z test to compare independent proportions (Agresti 2019). The test statistic follows a chi-squared distribution with 1 degree of freedom when using the squared z statistic. We report the difference with a 95% confidence interval and Cohen’s h as an effect size measure.
The difference in proportions: \[ \hat{p}_A - \hat{p}_B = \frac{x_A}{n_A} - \frac{x_B}{n_B}. \]
Cohen’s h for effect size: \[ h = 2(\arcsin\sqrt{\hat{p}_A} - \arcsin\sqrt{\hat{p}_B}). \]
Calculation
Visualisation
Results and Interpretation
Variant A converted at 12.0% (120/1,000) and variant B at 9.5% (95/1,000). The estimated difference was 2.5 percentage points with a 95% confidence interval of [−0.2, 5.2] percentage points. A two-proportion z test found no statistically significant difference, χ²(1) = 3.26, p = .071 (Agresti 2019; R Core Team 2024).
The effect size, Cohen’s h = 0.08, sits below the conventional threshold for a small effect (small: 0.20, medium: 0.50, large: 0.80).
The 95% CI [−0.2, 5.2 pp] includes zero, so the data are consistent with variant A being slightly worse than B as well as up to 5.2 points better.
Decision framework. Variant A leads by 2.5 points, and the test was too small to tell that lead from chance. Keep both variants live and run a larger test before any rollout. Variant A is the one to watch.
Sample Size Planning
To detect a 2.5 percentage point difference with 80% power at α = 0.05, you need about 2,410 visitors per group (4,820 total) under the pooled normal approximation shown below, or 3,226 per group for 90% power. The current test, with 1,000 per group, had about 44% power to detect this effect, which explains the inconclusive result.
For future tests, use the formula: \[ n_{\text{per group}} = \frac{2\bar{p}(1-\bar{p})(z_{1-\alpha/2} + z_{1-\beta})^2}{\delta^2}, \] where \(\bar{p}\) is the average of the two proportions and \(\delta\) is the target difference. This is the pooled normal approximation. Exact methods, such as an arcsine or exact binomial calculation in power software, can return a slightly different figure.
Assumptions
The two-proportion test assumes:
- Independent samples: Each visitor appears in only one variant (no crossover or contamination between groups)
- Random assignment: Visitors were randomly allocated to A or B without systematic bias (e.g., no assignment based on time of day or device type)
- Large sample approximation: Each group has at least 5 expected successes and 5 expected failures for the normal approximation to hold (both conditions met: 120 successes and 880 failures in A, 95 successes and 905 failures in B, all above 5)
- Stable conversion rates: The true conversion rate for each variant remains constant during the test period (no external events or time trends affecting one group differently)
Limitations
This analysis treats all visitors alike, across device type, traffic source and time of day. Differences in these factors could influence conversion rates. Use a stratified analysis or a regression model if you suspect imbalances.
Use the below format to cite this page
Sharafuddin, M. A. (2024, June 17). Two-proportion test: Comparing conversion rates between variants. Flair Marketing Intelligence (FlairMI). https://flairmi.com/blog/posts/02-two-proportion-test.html
@online{sharafuddin2024-two-prop,
author = {Sharafuddin, Mohammed Ali},
title = {Two-Proportion Test: Comparing Conversion Rates Between Variants},
year = {2024},
date = {2024-06-17},
url = {https://flairmi.com/blog/posts/02-two-proportion-test.html},
langid = {en}
}
Try it in the browser
The A/B Test Calculator runs this two-proportion z-test and the sample size calculation in the browser, with confidence intervals drawn for both groups. It is one of 44 free tools in the Marketing Toolkit Hub.
References
Citation
@online{ali_sharafuddin2024,
author = {Ali Sharafuddin, Mohammed},
title = {Two-Proportion {Test:} {Comparing} {Conversion} {Rates}
{Between} {Variants}},
date = {2024-06-17},
url = {https://flairmi.com/blog/posts/02-two-proportion-test.html},
langid = {en}
}
Comments