LLM Survey Surrogates Flatten Human Cultural Taste

Synthetic survey panels are moving from research curiosity to commercial product, but cultural taste is a difficult target for imitation. Taste is not just a marginal probability that someone likes jazz, opera, or hip-hop; it is a structured set of relations between activities, identities, and social position. This paper tests whether large language models can reproduce that structure using the Survey of Public Participation in the Arts as the human reference point. The answer is mostly negative: the models produce recognizable facsimiles of survey respondents, but their tastes are too positive, too stylized, and poorly aligned with the social patterning present in the real data.

Core Contribution

The paper’s main contribution is to move silicon sampling from broad attitude imitation into cultural consumption, where the target includes both individual preferences and the relational structure among tastes. The authors generate 277,470 synthetic respondents: 30 LLM completions for each of 9,249 SPPA respondents, using models from OpenAI, Anthropic, and DeepSeek. They then compare model-produced cultural preferences with the original survey data at several levels: aggregate liking rates, individual-level bias, taste-to-taste associations, and associations between tastes and demographic or social variables.

The paper’s strongest point is that it does not treat close aggregate averages as sufficient. A model could match the population share that likes classical music while still failing to reproduce who likes it, what other tastes travel with it, and how those tastes vary by age, gender, race, or class. The authors make that distinction explicit, and the results show why it matters.

Technical Approach

The study uses SPPA respondents as profile anchors and asks LLMs to act as survey surrogates. For each human respondent, each model produces repeated responses to cultural taste questions, creating multiple synthetic samples per person. The evaluation then asks whether these samples preserve the human survey’s ecological estimates and its internal dependence structure.

Cramer’s V is one of the paper’s central diagnostics. It measures association strength between categorical taste variables, allowing the authors to compare whether, for example, liking hip-hop is related to liking country, electronic music, jazz, or opera in the same way in human and synthetic data. The extracted figures summarize this comparison directly: Figure 1 compares Cramer’s V estimates from silicon samples against SPPA data, while Figure 2 maps deviations from the human benchmark.

This is the right test for the paper’s claim. If LLM survey surrogates are useful only as marginal-probability machines, they may still be poor substitutes for real respondents in cultural sociology, market segmentation, recommender evaluation, or any setting where the correlation structure is the object of interest.

Results and Analysis

The paper reports three main failures. First, silicon samples show a systematic positive bias for liking cultural categories, inflating aggregate estimates of taste. The abstract states that this individual-level bias is not well explained by the standard WEIRD-bias account often used to explain LLM distortions. That matters because it points away from a simple demographic correction and toward a more model-specific bias: the surrogate tends to infer cultural approval too readily.

Second, the relational structure among tastes is badly distorted. The visible appendix tables make this concrete. In the DeepSeek Cramer’s V table, hip-hop and country have an association of 0.64, while the bootstrap human-reference table shows 0.09 for the same pair. Reggae and hip-hop are 0.68 under DeepSeek versus 0.37 in the bootstrap table. Bluegrass and country reach 0.80 under DeepSeek versus 0.33 in the bootstrap table. These are not small calibration errors; they imply a different map of cultural affinity.

The reverse problem also appears. Some human associations are weakened or reshaped. The bootstrap table shows broad midrange links among genres such as jazz, blues, classical, bluegrass, and Broadway, with many values around 0.30–0.55. The DeepSeek table has sharper, less human-looking contrasts, including near-zero values for some pairs that retain nontrivial association in the bootstrap reference. Figure 2 is useful here because the issue is not one genre pair; it is the pattern of deviations across the matrix.

Third, the models fail to preserve cultural alignment with social space. According to the abstract, they attenuate age-taste associations, revive outdated class-taste associations, and exaggerate gender- and race-taste associations. That is a serious limitation for anyone using LLMs as survey substitutes: the model may appear demographically aware while encoding the wrong relationships between identity and preference.

The evidence supports a cautious conclusion. These LLMs can produce survey-shaped answers at scale, but the paper shows that survey-shaped is not the same as survey-valid. The practical beneficiary is less the synthetic-panel vendor than the evaluator: the study offers a set of diagnostics for deciding when silicon samples are unsuitable for replacing human cultural data.