Cultural Riddles Expose LLM Blind Spots Across Communities

Language coverage is not the same as cultural understanding. A model can translate a riddle or retrieve facts about its setting yet still miss the social associations, wordplay, and abductive reasoning that make its answer intelligible to people in the originating community. CulturalRiddles targets that distinction with human-authored riddles from 62 cultures across 45 countries and 51 languages.

The paper’s central result is not a single average score. Performance varies sharply by community: even frontier models reach at most 22% accuracy in some settings. That makes the benchmark useful less as another aggregate multilingual leaderboard than as a diagnostic for where broad language-model competence fails to transfer.

Core Contribution

CulturalRiddles is built through participatory collection rather than translating an English benchmark outward. The authors assemble original riddles rooted in local communities, then evaluate whether models can infer answers that depend on culturally specific knowledge and associations. The visual coverage analysis reports 6,092 English translations across 61 communities, while the paper describes the broader benchmark as spanning 62 cultures.

That design matters because riddles require more than fact recall. A plausible response may depend on identifying an object, practice, relationship, or linguistic cue from an indirect description. Exact string matching is therefore insufficient: it can reject an answer that native speakers would accept, while an apparently fluent answer may fail to resolve the riddle’s intended association.

Technical Approach

The benchmark combines the riddle collection with a layered evaluation pipeline. Models produce answers, automatic rules handle cases that can be resolved mechanically, and remaining responses are routed for human and LLM-based assessment. Native-speaker review is central rather than incidental: Figure 5 shows that native speakers score more than half of all responses entering the pipeline. A jury assigns production scores only when two judges agree, leaving disagreements and missing verdicts unscored.

Figure 1 situates this pipeline alongside the benchmark’s main findings. The paper also tests the reliability of LLM judging against human ratings. Its agreement analysis distinguishes binary accuracy, a weighted 0–3 score, and 13 response types, with 95% intervals obtained from 1,000 riddle-bootstrap resamples. Agreement approaches the human–human range for accuracy and weighted scores, but the paper reports weaker alignment on the fine-grained response taxonomy. That is a sensible boundary: an automated judge can support large-scale scoring without being treated as a replacement for cultural adjudication.

Results and Analysis

The authors benchmark 28 LLM configurations. The clearest finding is heterogeneity concealed by pooled accuracy. Figure 7 separates exact-match answers from additional answers accepted by native speakers or the jury, showing that exact match misses a nontrivial share of ultimately correct responses. It also displays the number of communities below 50% accuracy, making community-level failure visible even when a model’s aggregate score appears acceptable.

The result is especially consequential for deployment claims based on multilingual breadth. A model that succeeds in high-resource settings may still be unusable for riddles from communities whose cultural references were poorly represented in training or evaluation. Figure 8 ranks communities by mean accuracy and shows that even the strongest models struggle in the hardest settings. Figure 9 further argues that Common Crawl resource levels do not fully explain these differences: language-resource availability is correlated with performance, but it is not a complete account of cultural variation.

The experimental design supports the narrower claim that present models have uneven culturally grounded riddle-solving ability. It does not establish a complete measure of cultural competence. Riddles are intentionally indirect, and their accepted answers depend on community judgment; this is a demanding but specific form of reasoning. Still, the combination of 28 configurations, native-speaker assessment, and explicit treatment of scoring ambiguity is stronger evidence than a translated multiple-choice test would provide.

Limits for Practice

Coverage remains finite despite the collection’s scale: 62 cultures and 51 languages cannot represent the full diversity within countries, languages, or diasporas. The evaluation also relies partly on judge agreement and routes only a subset of responses beyond automatic rules, so unresolved disagreement is part of the measured system rather than noise that disappears. Finally, model comparisons should be read at the community level, not reduced to a universal ranking: the paper’s own plots show that mean performance can hide severe local failures.