Two-Proportion Test: Comparing Conversion Rates Between Variants

statistics
A/B testing
conversion optimization
web analytics
Compare add-to-cart conversion between variant A and variant B with a two-proportion z test and report the difference with a confidence interval and effect size.
Published

June 17, 2024

Keywords

two-proportion test, A/B testing, conversion comparison, effect size, Cohen’s h

TL;DR: Variant A 12.0% vs B 9.5% (n=1,000 each), difference +2.5 pp [−0.2, +5.2 pp], χ²(1) = 3.26, p = .071 (not significant), Cohen’s h = 0.08 (below small). Decision: no clear winner yet. Run a larger test, about 2,410 visitors per group, to settle it.

Answer
Method: Two-proportion z-test.
Estimate: 12.0% vs 9.5%, difference 2.5 pp, CI −0.2, +5.2 pp.
Data: A/B test, variables variant, conversion, n = 2,000 (1,000 per variant).
Action: No significant difference at α = .05. Continue testing with a larger sample.

Case

You ran an A/B test on a product page. After two weeks, variant A had 120 conversions from 1,000 visitors (12.0%), while variant B had 95 conversions from 1,000 visitors (9.5%). The business question: Is the 2.5 percentage point difference statistically significant? Should you keep variant A or run a confirmatory test?

Dataset

Synthetic A/B experiment data (Schema C).

Variable Label Value
variant Test group A or B
converted Conversion event 0 or 1
n Sample size 2,000 rows
x_A Conversions in A 120
x_B Conversions in B 95
n_A Visitors in A 1,000
n_B Visitors in B 1,000

Method

We use a two-proportion z test to compare independent proportions (Agresti 2019). The test statistic follows a chi-squared distribution with 1 degree of freedom when using the squared z statistic. We report the difference with a 95% confidence interval and Cohen’s h as an effect size measure.

The difference in proportions: \[ \hat{p}_A - \hat{p}_B = \frac{x_A}{n_A} - \frac{x_B}{n_B}. \]

Cohen’s h for effect size: \[ h = 2(\arcsin\sqrt{\hat{p}_A} - \arcsin\sqrt{\hat{p}_B}). \]

Calculation

Visualisation

Results and Interpretation

Variant A converted at 12.0% (120/1,000) and variant B at 9.5% (95/1,000). The estimated difference was 2.5 percentage points with a 95% confidence interval of [−0.2, 5.2] percentage points. A two-proportion z test found no statistically significant difference, χ²(1) = 3.26, p = .071 (Agresti 2019; R Core Team 2024).

The effect size, Cohen’s h = 0.08, sits below the conventional threshold for a small effect (small: 0.20, medium: 0.50, large: 0.80).

The 95% CI [−0.2, 5.2 pp] includes zero, so the data are consistent with variant A being slightly worse than B as well as up to 5.2 points better.

Decision framework. Variant A leads by 2.5 points, and the test was too small to tell that lead from chance. Keep both variants live and run a larger test before any rollout. Variant A is the one to watch.

Sample Size Planning

To detect a 2.5 percentage point difference with 80% power at α = 0.05, you need about 2,410 visitors per group (4,820 total) under the pooled normal approximation shown below, or 3,226 per group for 90% power. The current test, with 1,000 per group, had about 44% power to detect this effect, which explains the inconclusive result.

For future tests, use the formula: \[ n_{\text{per group}} = \frac{2\bar{p}(1-\bar{p})(z_{1-\alpha/2} + z_{1-\beta})^2}{\delta^2}, \] where \(\bar{p}\) is the average of the two proportions and \(\delta\) is the target difference. This is the pooled normal approximation. Exact methods, such as an arcsine or exact binomial calculation in power software, can return a slightly different figure.

Assumptions

The two-proportion test assumes:

  • Independent samples: Each visitor appears in only one variant (no crossover or contamination between groups)
  • Random assignment: Visitors were randomly allocated to A or B without systematic bias (e.g., no assignment based on time of day or device type)
  • Large sample approximation: Each group has at least 5 expected successes and 5 expected failures for the normal approximation to hold (both conditions met: 120 successes and 880 failures in A, 95 successes and 905 failures in B, all above 5)
  • Stable conversion rates: The true conversion rate for each variant remains constant during the test period (no external events or time trends affecting one group differently)

Limitations

This analysis treats all visitors alike, across device type, traffic source and time of day. Differences in these factors could influence conversion rates. Use a stratified analysis or a regression model if you suspect imbalances.


Use the below format to cite this page

Sharafuddin, M. A. (2024, June 17). Two-proportion test: Comparing conversion rates between variants. Flair Marketing Intelligence (FlairMI). https://flairmi.com/blog/posts/02-two-proportion-test.html
@online{sharafuddin2024-two-prop,
  author = {Sharafuddin, Mohammed Ali},
  title  = {Two-Proportion Test: Comparing Conversion Rates Between Variants},
  year   = {2024},
  date   = {2024-06-17},
  url    = {https://flairmi.com/blog/posts/02-two-proportion-test.html},
  langid = {en}
}

Try it in the browser

The A/B Test Calculator runs this two-proportion z-test and the sample size calculation in the browser, with confidence intervals drawn for both groups. It is one of 44 free tools in the Marketing Toolkit Hub.

Comments

References

Agresti, Alan. 2019. Statistical Methods for the Social Sciences. Boston, MA: Pearson.
R Core Team. 2024. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.r-project.org/.

Citation

BibTeX citation:
@online{ali_sharafuddin2024,
  author = {Ali Sharafuddin, Mohammed},
  title = {Two-Proportion {Test:} {Comparing} {Conversion} {Rates}
    {Between} {Variants}},
  date = {2024-06-17},
  url = {https://flairmi.com/blog/posts/02-two-proportion-test.html},
  langid = {en}
}
For attribution, please cite this work as:
Ali Sharafuddin, Mohammed. 2024. “Two-Proportion Test: Comparing Conversion Rates Between Variants.” June 17, 2024. https://flairmi.com/blog/posts/02-two-proportion-test.html.