CALM Cuts Trial-Estimate Variance While Preserving Validity

Randomized trials identify causal effects through treatment assignment, but many remain too small to estimate subgroup effects precisely. Baseline covariate adjustment helps, yet standard estimators generally discard information in free-text surveys and the broader associations encoded in a language model. CALM, short for Causal Analysis leveraging Language Models, treats an LLM's outcome prediction as auxiliary prognostic information rather than as a causal oracle, then calibrates it inside a conventional randomized-experiment estimator.

The distinction is consequential. An LLM can make systematically wrong counterfactual predictions; simply inserting those predictions as covariates into augmented inverse-probability weighting (AIPW) can damage finite-sample coverage. The authors instead build a correction term whose weight is estimated from held-out trial data. Their claim is not that the model knows an unobserved potential outcome, but that a useful correlation with it can reduce uncertainty after calibration.

Core Contribution

CALM augments AIPW with a residualized LLM prediction of the potential outcome under treatment $t$. The correction is multiplied by a calibration weight $\omega_t(X_i)$ chosen to minimize asymptotic variance. The resulting estimator retains the trial's randomization-based identification: the prediction can be biased, provided the calibration and nuisance functions are fit on separate data folds.

The paper supplies an asymptotic argument rather than relying on empirical agreement with the LLM. In the oracle calculation, CALM's variance equals the AIPW variance less a nonnegative term determined by the conditional correlation between the observed outcome and the LLM-generated prediction. This makes the efficiency condition intelligible: an uninformative LLM should receive essentially no weight, while a prognostic one can help. The authors further argue that constrained calibration remains weakly more efficient than AIPW because setting the calibration weight to zero recovers the AIPW case.

Technical Approach

The implementation uses sample splitting and cross-fitting. One fold supplies demonstration examples for few-shot prompting, another fits nuisance functions and calibration weights, and a third evaluates the estimator; the roles rotate before aggregation. This separation prevents a unit's own outcome from contaminating its prediction. A reduced two-fold procedure is also described for parsimonious nuisance-function classes, but the three-way split is the safer option when flexible models are used.

Few-shot CALM adds a second source of dependence: predictions change with the sampled demonstrations. The paper averages predictions over $B$ randomly drawn demonstration sets, producing a U-statistic-like predictor. The supplementary proof gives an approximation error of order $1/B + 1/n$ for this resampling-based prediction, so aggregation is intended to reduce prompt-selection noise as well as restore the independence structure needed for inference.

The BRIGHTEN case study illustrates what information enters the prompts. The underlying smartphone depression trial includes structured demographics, phone activity, app-use behavior, PHQ-9, functional-status and anxiety measures, plus open-ended responses about app satisfaction and motivation. Figure 1 shows that the text fields have comparatively high cosine similarity with the primary mental-health outcome, which gives the LLM prediction a plausible prognostic role rather than making text an incidental feature.

Results and Analysis

The empirical evidence is a simulation, calibrated to BRIGHTEN rather than a reanalysis of observed trial effects. The authors construct a synthetic population of $N=20{,}000$: sentence-transformer embeddings of text are compressed to 32 principal components, a CTGAN generates covariates, causal forests generate conditional means and variances, and text is reconstructed through nearest-neighbor retrieval from phrase banks. Each simulated trial randomizes treatment with probability 0.5.

Across 300 Monte Carlo repetitions and sample sizes from $n=400$ to $2{,}000$, CALM variants have near-zero absolute bias and roughly nominal 95% interval coverage. The plots show lower $\sqrt{n}$-scaled standard deviation as the few-shot demonstration count rises from $m=0$ to $m=14$; the authors identify $m=14$ as the lowest-variance CALM setting. In contrast, the AIPW variants that append LLM predictions as ordinary covariates show visibly lower coverage in the few-shot settings. The coverage table makes that contrast concrete: at $n=400$, AIPW with a few-shot covariate records 0.92, 0.90, and 0.89 coverage for $m=6$, 10, and 14, while CALM records 0.94, 0.96, and 0.95.

That is credible evidence for the estimator's intended safeguard in this constructed setting. It is not yet evidence that the approach improves power in a new clinical trial: the data-generating process, outcome discretization into five depression-severity levels, and prompt task are all tailored to BRIGHTEN. The paper also compares GPT-4o-mini with alternative models in a later case study, but its central benchmark remains a single synthetic population. The method is most useful where free text is genuinely predictive, analysts can maintain strict pre-treatment separation, and the added prompting and cross-fitting work is justified by a narrower confidence interval.