Welch t-Test: Comparing Average Order Value Between Groups

statistics
A/B testing
revenue optimization
ecommerce
Compare average order value between control and treatment groups with a Welch t-test and report the mean difference with a confidence interval and effect size.
Published

June 24, 2024

Keywords

t-test, Welch t-test, average order value, effect size, Hedges g

TL;DR: Control ₹1,500 vs Treatment ₹1,649 (n=200 each), difference ₹149 [₹92, ₹205], t(397) = 5.17, p < .001, Hedges g = 0.52 (medium). Decision: with random allocation, the treatment raised AOV in this test, roll out if margin supports.

Answer
Method: Welch t-test for independent samples.
Estimate: ₹1,500 vs ₹1,649, difference ₹149, CI ₹92, ₹205.
Data: A/B test, variables group, order_value, n = 400 (200 per group).
Action: Provided allocation was random, the treatment raised AOV by about ₹149. Roll out if margin supports.

Case

You are analysing a merchandising experiment for an e-commerce store. The control group (A) saw standard product displays, while the treatment group (B) received enhanced product recommendations. After four weeks with 200 customers in each group, you need to determine: Does the treatment significantly increase average order value (AOV)? Should you roll out the new merchandising approach?

Dataset

Synthetic sample from e-commerce experiment (Schema A).

Variable Label Value
aov_a Control group AOV ₹ (rupees)
aov_b Treatment group AOV ₹ (rupees)
n_a Control sample size 200
n_b Treatment sample size 200
mean_a Control mean used to simulate ₹1,500
mean_b Treatment mean used to simulate ₹1,580

Method

We use a Welch t-test to compare two independent groups with potentially unequal variances (Welch 1947). This is more robust than Student’s t-test when variances differ. We report the mean difference with a 95% confidence interval and Hedges g as an effect size measure (which applies a small-sample correction to Cohen’s d).

The mean difference: \[ \bar{x}_B - \bar{x}_A = \frac{1}{n_B}\sum x_{B,i} - \frac{1}{n_A}\sum x_{A,i}. \]

Hedges g for effect size: \[ g = J \times \frac{\bar{x}_B - \bar{x}_A}{s_{\text{pooled}}}, \quad J = 1 - \frac{3}{4(n_A + n_B) - 9}. \]

Calculation

Visualisation

Results and Interpretation

The control group had a mean AOV of ₹1,499.8 (SD = ₹279.2), while the treatment group had a mean AOV of ₹1,648.6 (SD = ₹295.8). The estimated mean difference was ₹148.7 with a 95% confidence interval of [₹92.2, ₹205.3]. A Welch t-test found a statistically significant difference, t(396.7) = 5.17, p < .001 (Welch 1947; R Core Team 2024).

The effect size, Hedges g = 0.52, is medium by conventional standards (small: 0.20, medium: 0.50, large: 0.80). That label describes size relative to the spread of order values. Practical importance comes from the money: a lift of ₹92 to ₹205 per order, set against the margin and the cost of the change.

The 95% CI [₹92, ₹205] shows the true improvement could range from moderate to large. Even the lower bound, a ₹92 increase per order, adds up to real revenue at scale.

Decision framework. The treatment group shows a statistically significant increase in AOV of about ₹149 per order. If customers were randomly allocated and the merchandising change was implemented as designed, that difference can be attributed to the treatment. Whether it is worth rolling out depends on the margin each extra rupee carries and the cost of running the change. Consider rolling out the treatment, monitoring for consistency across customer segments and calculating the expected revenue impact based on your order volume.

Sample Size Planning

To detect an ₹80 difference in AOV, the lift the experiment was designed around, with 80% power at α = 0.05 (assuming SD ≈ ₹280), you need about 193 customers per group (386 total), or 258 per group for 90% power. The current test, with 200 per group, had about 81% power for that ₹80 difference.

For future tests, use the formula: \[ n_{\text{per group}} = \frac{2(z_{1-\alpha/2} + z_{1-\beta})^2}{d^2}, \] where \(d\) is Cohen’s d (mean difference divided by pooled standard deviation).

Assumptions

The Welch t-test assumes:

  • Independent observations: Each customer’s order is independent
  • Random assignment: Customers were randomly allocated to control or treatment
  • Approximate normality: AOV distributions are approximately normal, or sample sizes are large enough (n ≥ 30 per group) for the Central Limit Theorem to apply
  • Unequal variances allowed: The Welch t-test adjusts its degrees of freedom for unequal variances (SD control = ₹279, treatment = ₹296)

Limitations

This analysis leaves out:

  • Skewness: AOV distributions in retail are often right-skewed. Consider median comparisons or log-transformation if extreme outliers are present.
  • Segmentation: Results may vary by customer segment (new vs. returning, device type, traffic source)
  • Time effects: Seasonal patterns or time-of-week effects could influence AOV
  • Multiple testing: If running multiple simultaneous experiments, adjust significance levels accordingly

For highly skewed data, consider the Mann-Whitney U test (Wilcoxon rank-sum test) as a non-parametric alternative.


Use the below format to cite this page

Sharafuddin, M. A. (2024, June 24). Welch t-test: Comparing average order value between groups. Flair Marketing Intelligence (FlairMI). https://flairmi.com/blog/posts/03-t-test.html
@online{sharafuddin2024-t-test,
  author = {Sharafuddin, Mohammed Ali},
  title  = {Welch t-Test: Comparing Average Order Value Between Groups},
  year   = {2024},
  date   = {2024-06-24},
  url    = {https://flairmi.com/blog/posts/03-t-test.html},
  langid = {en}
}

Comments

References

R Core Team. 2024. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.r-project.org/.
Welch, Bernard Lewis. 1947. “The Generalization of Student’s Problem When Several Different Population Variances Are Involved.” Biometrika 34 (1-2): 28–35. https://doi.org/10.1093/biomet/34.1-2.28.

Citation

BibTeX citation:
@online{ali_sharafuddin2024,
  author = {Ali Sharafuddin, Mohammed},
  title = {Welch {t-Test:} {Comparing} {Average} {Order} {Value}
    {Between} {Groups}},
  date = {2024-06-24},
  url = {https://flairmi.com/blog/posts/03-t-test.html},
  langid = {en}
}
For attribution, please cite this work as:
Ali Sharafuddin, Mohammed. 2024. “Welch t-Test: Comparing Average Order Value Between Groups.” June 24, 2024. https://flairmi.com/blog/posts/03-t-test.html.