CogGym Measures the Gap Between Models and Human Judgments

A unified evaluation of 50 language models across 258 cognitive experiments finds improving model--human fit, but a substantial gap from human reliability remains.

Editorial Desk·September 22, 2026·3 min readstrong

Underlying Paper

CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognition

Understanding and modeling human intelligence are parallel goals shared by artificial intelligence (AI) and cognitive science. As AI systems grow increasingly capable, in what ways do model responses resemble human responses, and where do they systematically diverge? The sheer breadth and diversity of the tasks humans can perform and think about pose a challenge for scalable and rigorous comparison between humans and models. We introduce CogGym, a scalable, unified framework grounded in cognitive science for systematically comparing model and human behavior on matched experimental trials. CogGym uses a semi-automated, human-in-the-loop pipeline to standardize diverse experimental paradigms into a task-agnostic Experiment Markup Language (EML), enabling reproducible and faithful comparison at scale. For initial release, we curate and standardize 258 cognitive experiments from 100 papers that focuses on human commonsense reasoning, and evaluate 50 large language models against human responses. We find a clear scaling trend where larger and more recent AI models better reproduce human judgments. Yet AI models' improvement on such common reasoning tasks is considerably slower than the gains observed on formal-reasoning benchmarks like math and coding, and model--human fit remains well below human splithalf reliability ($R^2 = 0.93$ on text, $0.95$ on image, and $0.92$ on video) with the best models achieving $R^2 = 0.59$ on text, $0.58$ on image, and $0.43$ on video experiments. We intend for CogGym to provide a living evaluation framework that continually incorporates new cognitive science experiments to characterize where model behavior resembles human behavior, where it systematically diverges, and how those patterns change as models and experiments evolve.

arXiv:2609.21259Submitted: Sep 22, 2026v1

arXiv Abstract PDF

AI benchmarks often assess whether a model reaches a correct answer. CogGym instead asks how closely model responses resemble the judgments people make on matched cognitive-science experiments. The framework standardizes 258 experiments from 100 papers and evaluates 50 language models against human responses.

Core Contribution

CogGym is an evaluation framework rather than a new model. Its central component is Experiment Markup Language (EML), a task-agnostic representation that standardizes experiments originally built in diverse formats. An EML specification captures experiment instructions, trial stimuli, response queries, and human data, then supports corresponding human studies and model prompts.

This shared representation is meant to make human and model trials more directly comparable. The workflow combines AI-assisted conversion with human-in-the-loop supervision, followed by administration to people and models and joint analysis of the resulting responses.

Figure 1. CogGym workflow. CogGym proceeds in three stages. First, original experiments implemented in diverse frameworks are converted into a standardized Experiment Markup Language (EML) representation through an AI agent with human-in-the-loop supervision. Second, the standardized experiments are rendered and administered to human participants and AI models. Lastly, human and model responses are analyzed jointly to evaluate experiment validity and model--human alignment.

Results and Analysis

The initial release spans commonsense-reasoning topics including physical reasoning, causal learning, concept learning, social cognition, moral judgment, probabilistic inference, and pragmatic language use. It includes text, image, and video experiments.

Across these tasks, larger and more recent models tend to reproduce human judgments more closely. But the strongest reported model--human fit remains below human split-half reliability: the best models reach $R^2$ values of 0.59 on text experiments, 0.58 on image experiments, and 0.43 on video experiments, compared with human reliability values of 0.93, 0.95, and 0.92 respectively.

The paper also compares progress on CogGym with formal-reasoning benchmarks. Its results indicate that gains on benchmarks such as GPQA Diamond, MATH, and HumanEval have risen more steeply than model--human alignment on these commonsense tasks. This does not imply that models cannot solve individual tasks; it indicates that matching aggregate human response patterns remains harder than improving formal benchmark performance.

Figure 4. Model performance on CogGym tasks and formal reasoning benchmarks plotted by model release date. Models are color-coded by size, and selected models are labeled. Top: model--human fit R^2 across experimental tasks. Middle: normalized distributional divergence between AI models and humans. Bottom: AI models' performance on GPQA Diamond, MATH, and HumanEval performance. The plots show that models' improvement on formal reasoning benchmarks is much steeper than models' improvement on their alignment with humans on commonsense reasoning tasks. Error bars show 1 standard error across experiments.

A per-experiment comparison further illustrates the gap. For Gemini 3.1 Pro, 94% of experiments fall below parity when model--human fit is compared with human split-half reliability. The result suggests that, across most evaluated experiments, agreement with people remains lower than agreement measured between groups of people.

Figure 6. Gemini 3.1 Pro vs. human split-half, per experiment. Each point is one experiment. (Top) R^2: Human split-half R^2 vs. model--human fit R^2; 94% of experiments fall below the parity diagonal. (Bottom) Distributional Divergence: human split-half normalized divergence (reliability floor) vs. model--human normalized divergence; Error bars in both panels are 1 SE from 1,000-sample item bootstraps.

Limits in Practice

CogGym is a large initial collection, not a complete account of human cognition. Its experiments are drawn from published commonsense-reasoning research, and the framework is intended to evolve as additional experiments are standardized. Aggregate alignment metrics are useful for comparison, but individual cognitive tasks can still reveal different kinds of model--human divergence.

CogGym is therefore best read as measurement infrastructure: a way to make matched comparisons at scale and identify where models resemble human judgments, where they differ, and how those patterns change over time.

Evidence Box

strong

Key Claims

•EML enables matched comparisons between human studies and model prompts
•Larger and newer models better reproduce human response patterns
•Model--human fit remains below human split-half reliability
•Formal-reasoning gains outpace gains in commonsense alignment

Key Results

•258 experiments from 100 papers were evaluated across 50 language models
•Best model--human R²: 0.59 text, 0.58 image, and 0.43 video
•Human split-half R²: 0.93 text, 0.95 image, and 0.92 video
•94% of Gemini 3.1 Pro experiments fall below human-reliability parity

Limitations & Caveats

•The initial release focuses on cognitive experiments drawn from published commonsense-reasoning work
•The benchmark does not provide complete coverage of human cognition
•Aggregate alignment metrics can obscure differences between individual experiments
•Continued expansion depends on standardizing additional cognitive-science experiments

DeepSeek Cuts KV Cache Footprint Fourfold

A causal encoder-decoder MoE pairs cross-layer sparse-attention reuse with FP4 caching to reduce global context memory to 890 bytes per token.

cs.CL

Sep 19, 20264 minmoderate

Listwise Generation Raises Short-Video Engagement in Existing Rankers

LIGE-GR adds causal list context, a list-level value function, and RL beam decoding to existing rankers, raising Reels time spent by 1.14%.

cs.IR cs.LG

Sep 17, 20264 minstrong

MinerU-Chem Reports 93.02% Exact Molecular Recognition

A chemistry-focused extension to MinerU converts document regions into molecule and reaction records, outperforming the evaluated GPT-5.6-Sol comparison on a SMILES benchmark subset.

cs.CV

Sep 14, 20262 minmoderate