Welch t-Test: Comparing Average Order Value Between Groups
t-test, Welch t-test, average order value, effect size, Hedges g
TL;DR: Control ₹1,500 vs Treatment ₹1,649 (n=200 each), difference ₹149 [₹92, ₹205], t(397) = 5.17, p < .001, Hedges g = 0.52 (medium). Decision: with random allocation, the treatment raised AOV in this test, roll out if margin supports.
Answer
Method: Welch t-test for independent samples.
Estimate: ₹1,500 vs ₹1,649, difference ₹149, CI ₹92, ₹205.
Data: A/B test, variables group, order_value, n = 400 (200 per group).
Action: Provided allocation was random, the treatment raised AOV by about ₹149. Roll out if margin supports.
Case
You are analysing a merchandising experiment for an e-commerce store. The control group (A) saw standard product displays, while the treatment group (B) received enhanced product recommendations. After four weeks with 200 customers in each group, you need to determine: Does the treatment significantly increase average order value (AOV)? Should you roll out the new merchandising approach?
Dataset
Synthetic sample from e-commerce experiment (Schema A).
| Variable | Label | Value |
|---|---|---|
aov_a |
Control group AOV | ₹ (rupees) |
aov_b |
Treatment group AOV | ₹ (rupees) |
n_a |
Control sample size | 200 |
n_b |
Treatment sample size | 200 |
mean_a |
Control mean used to simulate | ₹1,500 |
mean_b |
Treatment mean used to simulate | ₹1,580 |
Method
We use a Welch t-test to compare two independent groups with potentially unequal variances (Welch 1947). This is more robust than Student’s t-test when variances differ. We report the mean difference with a 95% confidence interval and Hedges g as an effect size measure (which applies a small-sample correction to Cohen’s d).
The mean difference: \[ \bar{x}_B - \bar{x}_A = \frac{1}{n_B}\sum x_{B,i} - \frac{1}{n_A}\sum x_{A,i}. \]
Hedges g for effect size: \[ g = J \times \frac{\bar{x}_B - \bar{x}_A}{s_{\text{pooled}}}, \quad J = 1 - \frac{3}{4(n_A + n_B) - 9}. \]
Calculation
Visualisation
Results and Interpretation
The control group had a mean AOV of ₹1,499.8 (SD = ₹279.2), while the treatment group had a mean AOV of ₹1,648.6 (SD = ₹295.8). The estimated mean difference was ₹148.7 with a 95% confidence interval of [₹92.2, ₹205.3]. A Welch t-test found a statistically significant difference, t(396.7) = 5.17, p < .001 (Welch 1947; R Core Team 2024).
The effect size, Hedges g = 0.52, is medium by conventional standards (small: 0.20, medium: 0.50, large: 0.80). That label describes size relative to the spread of order values. Practical importance comes from the money: a lift of ₹92 to ₹205 per order, set against the margin and the cost of the change.
The 95% CI [₹92, ₹205] shows the true improvement could range from moderate to large. Even the lower bound, a ₹92 increase per order, adds up to real revenue at scale.
Decision framework. The treatment group shows a statistically significant increase in AOV of about ₹149 per order. If customers were randomly allocated and the merchandising change was implemented as designed, that difference can be attributed to the treatment. Whether it is worth rolling out depends on the margin each extra rupee carries and the cost of running the change. Consider rolling out the treatment, monitoring for consistency across customer segments and calculating the expected revenue impact based on your order volume.
Sample Size Planning
To detect an ₹80 difference in AOV, the lift the experiment was designed around, with 80% power at α = 0.05 (assuming SD ≈ ₹280), you need about 193 customers per group (386 total), or 258 per group for 90% power. The current test, with 200 per group, had about 81% power for that ₹80 difference.
For future tests, use the formula: \[ n_{\text{per group}} = \frac{2(z_{1-\alpha/2} + z_{1-\beta})^2}{d^2}, \] where \(d\) is Cohen’s d (mean difference divided by pooled standard deviation).
Assumptions
The Welch t-test assumes:
- Independent observations: Each customer’s order is independent
- Random assignment: Customers were randomly allocated to control or treatment
- Approximate normality: AOV distributions are approximately normal, or sample sizes are large enough (n ≥ 30 per group) for the Central Limit Theorem to apply
- Unequal variances allowed: The Welch t-test adjusts its degrees of freedom for unequal variances (SD control = ₹279, treatment = ₹296)
Limitations
This analysis leaves out:
- Skewness: AOV distributions in retail are often right-skewed. Consider median comparisons or log-transformation if extreme outliers are present.
- Segmentation: Results may vary by customer segment (new vs. returning, device type, traffic source)
- Time effects: Seasonal patterns or time-of-week effects could influence AOV
- Multiple testing: If running multiple simultaneous experiments, adjust significance levels accordingly
For highly skewed data, consider the Mann-Whitney U test (Wilcoxon rank-sum test) as a non-parametric alternative.
Use the below format to cite this page
Sharafuddin, M. A. (2024, June 24). Welch t-test: Comparing average order value between groups. Flair Marketing Intelligence (FlairMI). https://flairmi.com/blog/posts/03-t-test.html
@online{sharafuddin2024-t-test,
author = {Sharafuddin, Mohammed Ali},
title = {Welch t-Test: Comparing Average Order Value Between Groups},
year = {2024},
date = {2024-06-24},
url = {https://flairmi.com/blog/posts/03-t-test.html},
langid = {en}
}
References
Citation
@online{ali_sharafuddin2024,
author = {Ali Sharafuddin, Mohammed},
title = {Welch {t-Test:} {Comparing} {Average} {Order} {Value}
{Between} {Groups}},
date = {2024-06-24},
url = {https://flairmi.com/blog/posts/03-t-test.html},
langid = {en}
}
Comments