Covariate Balance Diagnostics: When the Love Plot Says Balanced but the Groups Still Differ
Anas H. Alzahrani, MD PhD MPH
Department of Preventive Medicine and Public Health
Faculty of Medicine, King Abdulaziz University
Every propensity-score analysis ends with the same reassuring ritual. You match or weight, you compute a standardized mean difference for each covariate, you draw a Love plot, and you point to the tidy column of dots that have all slid under the 0.1 line. “Balance achieved.” The reviewers nod, the table goes into the supplement, and everyone proceeds as if the treated and control groups are now exchangeable.
The trouble is that a standardized mean difference does exactly one thing: it compares two means. Two groups can have identical means and still be profoundly different — different spread, different skew, different tails, different joint structure. A Love plot that is entirely green is consistent with distributions that barely overlap. And the other habit researchers reach for — running a t-test or a chi-squared test on each covariate and declaring balance when nothing is “significant” — is worse than uninformative: it rewards you for having a small sample. This guide is about reading balance the way it actually behaves, not the way the Love plot flatters it.
What Balance Actually Requires
The point of matching or weighting is to make the covariate distribution in the treated group look like the covariate distribution in the control group — so that comparing outcomes is not confounded by those covariates. Distribution, not average. Exchangeability is a statement about the whole conditional distribution of confounders, and a difference anywhere in that distribution — a fatter tail, a wider variance, an interaction that survived — is residual confounding waiting to bias a nonlinear outcome model.
Decision rule:
A standardized mean difference below 0.1 is a necessary check, not a sufficient one. Treat balance as achieved only when the means, the spread, and the shape of each covariate agree across groups — and check the balance of key interactions and squared terms, not just the main effects. If you only looked at SMDs, you only looked at the first moment.
The Love-Plot Blind Spot
The standardized mean difference is the difference in group means divided by a pooled standard deviation. It is scale-free and interpretable, which is why it deserves its place as the default balance metric. But its numerator is a difference of means, and a difference of means is blind to everything except the center of the distribution. Set the two group means exactly equal and the SMD is zero — no matter how different the two spreads are, no matter how much heavier one tail is.
This is not a hypothetical. Weighting methods in particular can equalize means while leaving variances mismatched, because the weights are chosen to balance the terms you put in the propensity model — usually the main effects — and nothing forces the higher moments to follow. That is why Austin and others recommend reporting the variance ratio (treated variance over control variance, which should sit near 1) alongside the SMD, and why looking at the empirical distributions directly — a QQ plot, an eCDF overlay, or a distributional distance like the Kolmogorov–Smirnov statistic — catches imbalance that any mean-based summary will miss.
See It Yourself
The explorer below holds the control group fixed and lets you move the treated group’s mean and spread on a single covariate, and set how many matched patients you have. Watch the four numbers people actually report. Start with The Love-plot blind spot: the means are identical, so the SMD reads 0.00 and glows green — while the KS distance and the variance ratio both scream that the curves do not overlap. Then try The small-n balance test and slide the sample size, and watch a fixed imbalance flip from “not significant” to “significant” without moving an inch.
Interactive covariate-balance explorer
One covariate, four balance metrics, three ways to be fooled
Control patients are fixed. Move the treated group’s mean and spread, and set how many matched patients you have. Then read the four numbers a balance check reports — and notice how often they disagree.
Shifts the treated center. This is the only thing SMD reacts to.
Widens or narrows the treated curve. Invisible to the SMD.
Only the balance p-value moves. SMD, VR and KS ignore n entirely.
Standardized mean difference
0.00
below 0.10 — "balanced"
Variance ratio (treated / control)
4.00
spreads differ
KS distance (whole distribution)
0.16
curves separate — shape/tail gap
Balance test p-value
1.00
"not significant" — depends on n
A green Love plot sitting on two distributions that do not overlap.
The standardized mean difference is below 0.10, so a Love plot would color this covariate "balanced" and move on. But the two curves differ in spread and tails — the KS distance is large and the variance ratio is far from 1. SMD compares only the first moment. Everything that lives in the variance and the tails is invisible to it, and for a nonlinear outcome model that is exactly where residual confounding hides.
Why You Should Not Significance-Test Balance
The most common balance mistake in clinical papers is not the Love-plot blind spot — it is the “Table 1 with p-values” that reports a hypothesis test for every covariate and reads a column of non-significant p-values as proof of balance. This is a category error, and it has been argued forcefully by Imai, King, and Stuart, and echoed in Austin’s guidance on propensity methods.
- Balance is a property of your sample, not a hypothesis about a population. In the matched or weighted dataset in front of you, the covariate means either are close or they are not. There is no superpopulation from which your matched sample is a draw, so there is no null hypothesis to test. The question “could this imbalance have arisen by chance?” is the wrong question; the sample is what it is.
- The p-value tracks your power, not your imbalance. A balance test conflates the size of the imbalance with the size of the sample. Matching discards unmatched patients, which shrinks n, which inflates every p-value — so the very act of matching can make covariates look “more balanced” on a test while doing nothing to the actual imbalance. A larger p-value from a smaller sample is not better balance; it is less power. The explorer above makes this concrete: fix the imbalance and the p-value swings across 0.05 on the strength of sample size alone.
- The standardized difference does not have this problem. Because it divides by a standard deviation rather than a standard error, the SMD does not depend on the sample size. That is precisely why it is the right summary and the balance test is not: two studies with the same imbalance report the same SMD whether they matched 40 patients or 4,000.
So report standardized differences and variance ratios before and after adjustment, show the distributions, and drop the balance p-values entirely. If a manuscript “demonstrates balance” with a row of non-significant tests, that is a red flag, not reassurance — especially when the matched sample is small.
What SMD Catches and What It Misses
| Kind of imbalance | Seen by SMD? | What to check instead |
|---|---|---|
| Different means | Yes — this is what it measures. | SMD (target < 0.1, ideally < 0.05). |
| Different variance / spread | No — invisible if means match. | Variance ratio (target near 1; roughly 0.5–2). |
| Different shape / skew / tails | No. | eCDF overlay, QQ plot, KS / other distributional distance. |
| Imbalanced interactions or non-linear terms | No — main-effect SMDs miss them entirely. | SMDs on products and squared terms; a prognostic score. |
| Chance imbalance in a small sample | SMD reflects it honestly; a p-value hides it as “non-significant.” | Read the SMD, not a balance test. |
A quick way to compress much of this: instead of — or in addition to — balancing every covariate one at a time, check balance on the prognostic score (a summary of how the covariates predict the outcome). If the groups are balanced on the thing that actually drives the outcome, main-effect imbalances that do not matter for the outcome are less alarming, and a residual imbalance on the prognostic score is a warning the individual SMDs can dilute.
Where This Goes Wrong in Real Studies
- Weighting that equalizes means but not variances. Inverse-probability weighting can drive main-effect SMDs to zero while leaving the weighted variances mismatched — a Love plot looks perfect and the tails still differ. Report weighted variance ratios, not just weighted SMDs.
- Balancing main effects and forgetting interactions. A propensity model with only main effects can leave age-by-sex or dose-by-comorbidity structure imbalanced. If the outcome model relies on those interactions, the main-effect Love plot is reassuring about the wrong thing.
- Post-match “Table 1 with p-values.” The most frequent version: a matched cohort of a few hundred, a Table 1 of non-significant p-values, and a sentence claiming balance. The small sample manufactured the non-significance.
- Reporting balance only after, never before. Without the before-adjustment column you cannot see how much confounding there was to remove, or whether adjustment helped or merely reshuffled.
Reviewer Red Flags
What a defensible study does
- Reports standardized differences before and after adjustment for every covariate.
- Adds variance ratios and looks at the distributions, not just the means.
- Checks balance on key interactions, non-linear terms, or a prognostic score.
- Uses standardized differences to judge balance — and reports no balance p-values.
What should make you nervous
- Balance argued from a column of non-significant p-values, especially with a small matched n.
- An all-green Love plot with no variance ratios and no look at the distributions.
- Only main-effect balance reported, when the analysis leans on interactions.
- Balance shown only after adjustment, with no before-adjustment comparison.
Decision Rules That Travel Well
- Judge balance with standardized differences and variance ratios, never with hypothesis tests.
- Treat SMD < 0.1 as necessary but not sufficient — confirm the spread and shape agree too.
- Look at the distributions directly (eCDF or QQ) or a distributional distance for the covariates that matter most.
- Balance the interactions and non-linear terms your outcome model actually uses, not just main effects.
- Report balance both before and after adjustment, so the reader can see what was removed.
How This Connects to the Rest of the Toolkit
Balance diagnostics are the audit step for the whole propensity-score family. They tell you whether matching or inverse-probability weighting did what it promised. They are distinct from — and prior to — positivity and overlap: overlap asks whether comparable patients exist in both groups at all, while balance asks whether, among the patients you kept, the covariate distributions were actually equalized. You can have adequate overlap and still finish with mismatched variances; you can have flawless means and still be missing a region of the covariate space. Both checks are separate obligations, and passing one is not passing the other.
Where Aqrab Fits
Balance is the moment a propensity-score analysis is most tempted to declare victory and least examined for it. A green Love plot is easy to produce and easy to trust; the harder questions — did the variances match, were the interactions balanced, was the “balance” really just a small sample hiding behind non-significant p-values — are the ones that decide whether the comparison is confounded. Aqrab reads a study’s balance evidence the way a careful methodologist would: checking whether balance was judged by standardized differences rather than tests, whether anything beyond the means was reported, and whether the diagnostics match what the outcome model relies on.
If you are appraising a propensity-score study — or writing one — bring more than a Love plot. Start with Aqrab Try and pressure-test the balance evidence before you let a column of green dots stand in for exchangeability.
Keep reading
Don't stop at one method.
Good methods judgment comes from contrast. Read the neighboring guides, see where the assumptions diverge, and avoid treating every observational problem like it needs the same hammer.
Simpson's Paradox: When Every Subgroup Says One Thing and the Total Says the Opposite
A practical guide to Simpson's paradox for clinical researchers. Covers why a treatment can help in every subgroup yet look harmful pooled, why 'always stratify' is wrong, and how the causal role of the stratifier — confounder, mediator, or collider — decides which table to trust.
The Table 2 Fallacy: When Every Adjusted Coefficient Looks Like a Cause
A practical guide to the Table 2 fallacy for clinical researchers. Covers why secondary coefficients in an adjusted model are not causal effects, mutual adjustment, mediator and confounder mismatch, and what reviewers should demand before reading a covariate row as a finding.
Causal Readiness: When a Huge Linked Dataset Still Cannot Identify an Effect
A practical guide to causal readiness in linked health and administrative data. Learn why scale and propensity-score overlap are not enough when treatment, need, comparators, or outcomes are poorly measured.