When More Covariates Break Positivity: Representation-Induced Overlap Failure in Clinical Text
Anas H. Alzahrani, MD PhD MPH
Department of Preventive Medicine and Public Health
Faculty of Medicine, King Abdulaziz University
A clinical research team adds thousands of words from medical notes to its causal model. The idea is sensible: richer text should reveal more of the patient history that drives treatment choice. But there is a second possibility. The text may also announce the treatment itself. When the representation learns who was treated, it can make treated and untreated patients look so different that the comparison becomes thin or disappears.
This is representation-induced positivity failure. It is not an argument against text, embeddings, or machine learning. It is a design warning: an adjustment representation must preserve information needed for confounding control without turning treatment assignment into a near-deterministic label.
Positivity Is About the Comparison, Not the Model’s Confidence
Positivity means that, for relevant patient profiles, each treatment strategy still has a meaningful chance of occurring. In plain language, the data must contain comparable people on both sides of the treatment question. A large sample does not create those comparisons if the treatment is nearly fixed within a region of the covariate space.
Reviewer translation
Before asking whether an embedding predicts treatment well, ask whether it predicts treatment so well that the counterfactual comparison is no longer credible.
This is different from ordinary confounding. Confounding asks whether important causes of treatment and outcome remain unmeasured. Positivity asks whether both treatment choices are represented at all. One can improve the first while damaging the second.
How Text Creates the Trap
Imagine comparing two medication strategies using notes recorded before treatment. The notes include symptoms, comorbidities, prior tests, clinician impressions, and the language used to justify a prescription. Some of that language is useful confounding information. Some may be a direct or near-direct marker of the treatment decision.
Original data
Patients may overlap in clinical history even when their notes are long and messy.
Representation
The embedding learns phrases, templates, or context that separate treatment groups.
Causal analysis
Weights or matches become extreme because some patients have no credible counterpart.
The danger is easy to miss because predictive success looks like methodological progress. A representation that classifies treatment accurately may be excellent for prediction and awkward for causal adjustment. The two tasks reward different things.
Interactive representation audit
Make the treatment signal visible before you adjust for it
Set how strongly the text representation encodes treatment, then try masking treatment-defining terms. The bars are an illustrative teaching proxy, not a replacement for study-level diagnostics.
Unmasked representation
28%
Illustrative overlap proxy
Treatment-separation proxy: 86%
After the proposed masking
28%
Illustrative overlap proxy
Treatment signal left: 80%
Quick read
Red zone: the representation is making treatment look nearly deterministic.
Teaching model only: real analyses should inspect representation-level treatment prediction, propensity-score tails, effective sample size, balance, and sensitivity to the representation choice.
What Masking Can—and Cannot—Repair
A natural response is to mask treatment-defining words before learning the adjustment representation. The goal is not to blind the analysis to all clinical information. It is to stop the representation from using obvious treatment signals as if they were pre-treatment confounders.
Masking is a sensitivity strategy, not a universal cure. If a masked term also carries legitimate information about prognosis or treatment indication, deleting it can create residual confounding. Replacement masking may preserve some context while hiding the exact token, but it introduces its own modeling choices. A credible analysis should show what was masked, why, how much information remained, and how the causal estimate changes.
Do not turn a repair into a new assumption
Better overlap after masking does not prove exchangeability. It only says the representation may be less dominated by treatment-defining language. Confounding control, outcome measurement, missingness, and the target estimand still need their own checks.
Failure Modes That Should Slow the Review
| Red flag | Why it is weak | Ask for instead |
|---|---|---|
| “The embedding predicts treatment well” | High treatment predictability may signal thin common support. | Overlap plots, propensity tails, effective sample size, and a declared target population. |
| Text window includes post-treatment notes | The representation can encode consequences of treatment or the treatment label itself. | A time-zero rule and a reproducible pre-treatment text window. |
| Masking is presented as “deconfounding” | Deleting tokens can remove real confounders and does not identify the effect by itself. | A masked/unmasked sensitivity analysis with a variable-level rationale. |
| Balance is shown only after weighting | A tidy balance table can hide extreme weights and an estimand changed by trimming. | Weight distributions, discarded patients, diagnostics before and after, and the resulting estimand. |
A Design-First Workflow for Text-Based Causal Studies
1. Freeze time zero
Define eligibility, treatment strategies, the outcome window, and the last permissible note before treatment.
2. Separate the roles of text
Label which features are intended to capture confounding, which may define treatment, and which could be downstream.
3. Compare representations
Show a clinically interpretable baseline, the unmasked representation, and any masked or restricted alternative. Compare overlap and the estimate, not just predictive metrics.
4. Declare the population you can support
If trimming or restriction removes patients, say whether the result is now an effect for the overlap population rather than the original cohort.
Decision Rules for Busy Reviewers
- If text comes from the treatment workflow, treat treatment leakage as a design question before treating it as a feature-engineering question.
- If adding text makes treatment prediction sharply better, inspect common support before celebrating the gain.
- If the representation has extreme tails, do not let a stable optimizer stand in for a credible counterfactual.
- If masking improves overlap, check which clinically meaningful confounders may have been removed.
- If the analysis trims patients, rewrite the estimand and report who remains before interpreting the effect.
Why This Matters for Aqrab
Text-based causal research can fail in a sophisticated way: the model appears to know more, while the study has fewer credible comparisons. The hard review question is not whether the representation is powerful. It is whether the representation still supports the treatment contrast the paper claims to estimate.
Aqrab is built for that judgment layer. Use Aqrab Try to pressure-check time zero, estimand, treatment leakage, overlap, and the difference between predictive performance and causal validity. Teams building repeatable review workflows can explore /developers.
Methods Anchor
The treatment-encoding failure mode and masking distinction follow the methodological preprint The Confounder Trap: Treatment-Encoding Representations in Causal Inference with Text. Its simulation results are useful motivation, not a substitute for checking overlap, measurement timing, and identification in a clinical dataset.
Keep reading
Don't stop at one method.
Good methods judgment comes from contrast. Read the neighboring guides, see where the assumptions diverge, and avoid treating every observational problem like it needs the same hammer.
Causal Readiness: When a Huge Linked Dataset Still Cannot Identify an Effect
A practical guide to causal readiness in linked health and administrative data. Learn why scale and propensity-score overlap are not enough when treatment, need, comparators, or outcomes are poorly measured.
Predicted Treatment Benefit: When a Risk Model Is Not a Treatment Recommendation
A practical guide to separating predicted outcome risk from predicted treatment benefit. Learn why a high-risk patient is not automatically a high-benefit patient, how risk modeling and effect modeling differ, and what reviewers should demand before trusting a personalized treatment claim.
Additive Interaction: When “No Interaction” Depends on the Scale
A practical guide to additive and multiplicative interaction in clinical research. Learn why a null product term can hide clinically important effect modification, how to read RERI, and what reviewers should demand before trusting a joint-exposure claim.