Welch t-Test vs Equal-Variance t-Test: Comparing Methods
Welch t-test, equal-variance t-test, variance ratio, robustness, AOV
TL;DR: Group A ₹1,204 (SD ₹151, n=120) vs B ₹1,282 (SD ₹291, n=150). Variance ratio 0.27. Welch t(234) = 2.85, p = .005, equal-variance t(268) = 2.67, p = .008, difference ₹78 [₹24, ₹132]. Decision: report Welch, and B’s higher AOV is statistically significant.
Answer
Method: Welch t-test vs equal-variance t-test comparison.
Estimate: ₹1,204 vs ₹1,282, difference ₹78, Welch CI ₹24, ₹132.
Data: 2 store locations, variables group, order_value, n = 270 (120 and 150).
Action: Use Welch for the independent-group comparison. Group B’s AOV is significantly higher.
Case
You are analysing average order value (AOV) from two store locations with different sample sizes (120 vs. 150 customers) and different variability in spending patterns. You need to decide: Should you use the standard equal-variance t-test (Student’s t-test) or the Welch t-test? What does the variance ratio tell you about the choice?
Dataset
Synthetic sample from e-commerce experiment (Schema A).
| Variable | Label | Value |
|---|---|---|
aov_a |
Group A AOV | ₹ (rupees) |
aov_b |
Group B AOV | ₹ (rupees) |
n_a |
Group A sample size | 120 |
n_b |
Group B sample size | 150 |
sd_a |
Group A SD used to simulate | ₹180 |
sd_b |
Group B SD used to simulate | ₹260 |
Method
We compare two approaches to testing differences in means:
- Equal-variance t-test (Student’s t-test): Assumes both groups have equal population variances and pools sample variances to estimate standard error
- Welch t-test: Allows the variances to differ and adjusts degrees of freedom to the variance ratio (Welch 1947)
The Welch t-test is generally more robust when variances differ, especially with unequal sample sizes. We calculate the variance ratio (smaller variance / larger variance) to assess heterogeneity. Use the Welch t-test for the independent-group comparison. It remains valid with unequal variances and avoids choosing a test from a preliminary variance rule. The ratio is reported here to describe the data. Picking the test from it would let the data choose the method, which itself distorts the error rate.
Test statistic for equal-variance t-test: \[ t = \frac{\bar{x}_A - \bar{x}_B}{s_p\sqrt{\frac{1}{n_A} + \frac{1}{n_B}}}, \quad s_p^2 = \frac{(n_A-1)s_A^2 + (n_B-1)s_B^2}{n_A + n_B - 2}. \]
Test statistic for Welch t-test: \[ t = \frac{\bar{x}_A - \bar{x}_B}{\sqrt{\frac{s_A^2}{n_A} + \frac{s_B^2}{n_B}}}, \quad df \approx \frac{\left(\frac{s_A^2}{n_A} + \frac{s_B^2}{n_B}\right)^2}{\frac{(s_A^2/n_A)^2}{n_A-1} + \frac{(s_B^2/n_B)^2}{n_B-1}}. \]
Calculation
Visualisation
Results and Interpretation
Group A had a mean AOV of ₹1,203.5 (SD = ₹151.4, n=120), while group B had a mean AOV of ₹1,281.7 (SD = ₹290.5, n=150). The variance ratio of 0.27 (22,919 / 84,416) shows group B’s variance is nearly 4 times group A’s, which strongly favours the Welch t-test.
Welch t-test results: The Welch t-test found a statistically significant difference, t(233.6) = 2.85, p = .005, with a 95% CI of [₹24.0, ₹132.2] for the mean difference (B − A) (Welch 1947; R Core Team 2024). The adjusted degrees of freedom (233.6 against the pooled 268) reflect the variance inequality.
Equal-variance t-test results: The equal-variance t-test gave t(268) = 2.67, p = .008, with a 95% CI of [₹20.5, ₹135.7]. The 2 tests agree here, but the equal-variance assumption is violated (variance ratio 0.27, below 0.5).
Method comparison: When variances differ substantially and sample sizes are unequal, the equal-variance t-test can give a misleading error rate, with the size and direction set by which group carries the larger variance. It is too liberal when the smaller group has the larger variance, and too conservative when, as here, the larger group has it. The Welch t-test keeps the error rate close to 5% either way. Both tests reach the same conclusion here (p < .05), and Welch is the result to report.
Decision framework. With a variance ratio of 0.27 and unequal sample sizes, use the Welch t-test as your primary result. The mean difference of ₹78 (95% CI [₹24, ₹132]) shows group B has the higher AOV, and the wide interval leaves the size of that gap uncertain. Check variance homogeneity before choosing your test, and default to Welch when in doubt, since it holds up even when the variances are equal.
Sample Size Planning
For future tests with similar variance structure, plan for adequate power using the more conservative Welch approach. To detect a ₹70 difference with 80% power at α = 0.05 (assuming the larger SD, about ₹260), you need about 217 per group (434 total), or 290 per group for 90% power. Planned this way, the current test, with 120 and 150 customers, had about 57% power for a difference of ₹68.
Note: When planning with unequal variances, use the larger standard deviation for conservative sample size estimates. Equal sample sizes are preferable as they maximise power and minimise Type I error inflation.
Assumptions
Both t-tests assume:
- Independent observations: Each customer’s order is independent
- Random sampling: Customers were randomly sampled from their respective populations
- Approximate normality: AOV distributions are approximately normal, or sample sizes are large enough (n ≥ 30 per group) for the Central Limit Theorem to apply
Additional assumption for equal-variance t-test: - Homogeneity of variance: Population variances are equal (violated here, with a ratio of 0.27)
Welch t-test advantage: Works with unequal variances, adjusting degrees of freedom automatically to the observed variance ratio.
Limitations
This analysis leaves out:
- Levene’s test or F-test: We used variance ratio as a rule of thumb (< 0.5 or > 2.0). Formal tests like Levene’s test can assess homogeneity statistically
- Non-normality: With smaller samples or extreme skewness, consider non-parametric alternatives (Mann-Whitney U test)
- Paired data: For paired measurements, use the paired t-test
- Multiple groups: For more than two groups, use ANOVA with appropriate variance adjustments (Welch ANOVA)
When variance ratio is between 0.5 and 2.0 with equal sample sizes, both tests typically give similar results. Always report which test you used and justify the choice based on variance diagnostics.
Use the below format to cite this page
Sharafuddin, M. A. (2025, October 25). Welch t-test vs equal-variance t-test: Comparing methods. Flair Marketing Intelligence (FlairMI). https://flairmi.com/blog/posts/04-welch-t.html
@online{sharafuddin2025-welch-comparison,
author = {Sharafuddin, Mohammed Ali},
title = {Welch t-Test vs Equal-Variance t-Test: Comparing Methods},
year = {2025},
date = {2025-10-25},
url = {https://flairmi.com/blog/posts/04-welch-t.html},
langid = {en}
}
References
Citation
@online{ali_sharafuddin2025,
author = {Ali Sharafuddin, Mohammed},
title = {Welch {t-Test} Vs {Equal-Variance} {t-Test:} {Comparing}
{Methods}},
date = {2025-10-25},
url = {https://flairmi.com/blog/posts/04-welch-t.html},
langid = {en}
}
Comments