Bias-Corrected LLM Choices Improve Economic Inference

LLMs can generate cheap counterfactual choices, but raw synthetic choices are not human data. The central problem in this paper is econometric: can those synthetic choices be corrected well enough to recover demand parameters or treatment effects when only a small amount of human evidence is available? Agentic Economic Modeling, or AEM, treats the LLM as a noisy choice generator rather than a replacement for respondents, then learns how to map task features and LLM decisions back toward human-aligned choices.

Core Contribution

The paper’s main contribution is a missing-data framing for synthetic economic experiments. A small primary set contains task features, human choices, and LLM choices; a larger auxiliary set contains task features and LLM choices but no human outcomes. AEM uses the primary set to estimate the LLM-to-human bias, then fills in human-aligned choices for the auxiliary set so standard econometric estimators can run on a larger effective sample.

Figure 1 captures that design: the LLM supplies coverage, while the human sample supplies calibration rather than being discarded or averaged with synthetic responses.

That distinction matters. The paper reports that uncorrected LLM choices can move estimates in the wrong direction or inflate error. The value of the framework is not that the model “acts like” consumers out of the box; it is that systematic model bias can be learned from a limited overlap sample and then used for inference.

Technical Approach

AEM has two stages. First, the authors prompt an LLM with task information and ask it to make choices in the same format as human participants. In the conjoint setting, those choices concern product configurations. In the regional delivery experiment, each prompt combines an order, a persona, and a delivery-option menu; the appendix reports 638,484 such prompts, with confidential fields replaced by placeholders in the paper.

Second, AEM estimates a correction model. The paper studies a mixture model and an integrated neural baseline. The mixture model concatenates order and persona features, passes them through a linear layer with hidden dimension 64, and uses an attention-layer hidden dimension of 32; optimization uses Adam with learning rate $10^{-5}$. The integrated baseline uses three linear layers with ReLU activations and includes the LLM’s choice among its inputs. The authors also test a prompting-only correction that injects prior delivery-option shares into the LLM prompt.

Figure 2 shows why correction is needed in the conjoint experiment: partworth estimates based on different choice sources do not line up automatically, so the econometric target is sensitive to synthetic-choice bias.

Results and Analysis

The conjoint study is the first check on whether the method helps rather than merely adds machinery. Using only 10% of the original human data to fit the correction model lowers the error of demand-parameter estimates, while raw LLM choices increase the errors. The paper’s interpretation is credible in a narrow sense: LLM-generated choices contain usable signal, but only after a supervised correction step anchored in human responses.

The regional field experiment is the stronger test because it targets a treatment effect. The full human experiment estimates a -60±8 bps effect on the share of Same-Day delivery. AEM’s mixture model, calibrated on 10% of geographic regions, estimates -65±10 bps on hold-out regions. That is close enough to make the framework practically interesting for experiments where full rollout is expensive, though the result is still one domain and one intervention.

The appendix adds useful stress tests. Table 5 reports share-prediction errors for four SSD and Non-SSD treatment/control groups. Prompting-based correction performs poorly, with errors from 0.5094 to 0.6817, apparently because one delivery option accounts for 97% of choices and the prompt pushes the model toward the safer prior. The integrated model is better, with errors from 0.0928 to 0.1414. The mixture model is best in three of four groups, including 0.0518 for SSD treatment and 0.0768 for Non-SSD control, but worse than the integrated model for Non-SSD treatment.

Figure 5 and Table 6 make the temporal limitation clear. With one week of human data, the human-only estimate is -37 bps; raw LLM choices give -350 bps; prompting correction gives +4 bps; integrated correction gives -38 bps; and the mixture model gives -41 bps. These corrected estimates move toward the full-experiment -60 bps effect, but the paper says no confidence interval is available for the time-wise setting because there is no time-wise bootstrap analogue and first days cannot be resampled as exchangeable units.

Limitations

The evidence supports AEM as a calibration framework, not as a general substitute for randomized experiments. The regional experiment still depends on a real human overlap sample, and temporal prediction is explicitly hard because customer behavior, holidays, new users, and other contextual shifts can change over time. The strongest claim is therefore efficiency: in these two settings, corrected LLM choices recover more useful economic signal than raw LLM choices or prompt-only correction.