Million-Clip Dataset Tests Video Reasoning Beyond Visual Quality

Video generation has improved quickly on perceptual fidelity, but attractive clips do not establish that a model understands continuity, interaction, causality, or the consequences of actions over time. Existing video-reasoning resources make that distinction difficult to study at scale: their task coverage and training volume are too narrow to separate memorization from transferable reasoning. The authors introduce the Very Big Video Reasoning suite, comprising a training dataset and an evaluation benchmark designed around verifiable task outcomes rather than model-based judging.

Core Contribution

The central contribution is scale paired with task structure. VBVR Dataset contains more than one million clips across 200 curated reasoning tasks, which the paper describes as roughly three orders of magnitude larger than prior datasets. Rather than treating video reasoning as a single visual-question-answering score, the suite organizes tasks through a taxonomy intended to cover spatiotemporal structure: persistence and continuity, interactions between entities, causal consequences, and related forms of visual reasoning that are hard to specify faithfully in text alone.

That framing matters because a video model can produce locally plausible motion while failing a task whose answer depends on what happened earlier, which object caused a change, or whether a sequence remains physically and temporally consistent. The paper's claim is not simply that more video data improves generation. It is that a sufficiently broad, controlled collection can expose and train the reasoning behaviors that visual-quality metrics leave unmeasured.

Technical Approach

VBVR-Bench is the evaluation half of the suite. Its distinguishing design choice is verifiable scoring. For tasks with objectively checkable outcomes, the benchmark uses rule-based scorers instead of asking another model to judge a generated video. This makes a failure diagnosable: a system can be evaluated on the relevant reasoning dimension rather than receive an opaque preference score from a language or vision-language judge.

The authors also assess whether the benchmark tracks human perception. Figure 3 plots, for individual video-generation models, human-preference win ratios against VBVR-Bench win ratios across the benchmark splits and reasoning dimensions. The fitted relationships and reported Spearman correlations are meant to test an important property of an automatic evaluator: agreement with people without making human comparison the sole scoring mechanism.

The paper then uses the suite for a scaling study. Models are trained and evaluated over a broad task collection, including held-out reasoning tasks. This setup is more demanding than testing on examples drawn from the same narrowly defined task distribution: evidence of improvement on unseen tasks is the relevant signal if the stated goal is generalizable video reasoning.

Results and Analysis

The strongest directly supported result is the resource scale: 200 curated tasks and over 1 million clips. That is a meaningful change in experimental capacity, not merely a larger benchmark leaderboard. It permits comparisons across task families and creates enough variation to investigate whether training on one collection of visual rules transfers to another.

The paper reports that VBVR-Bench evaluations closely match human perceptions across all of its splits, using Spearman correlation between the human and benchmark win ratios. The accompanying analysis supports the benchmark's intended role as an interpretable proxy for human judgments, although the provided material does not establish that it replaces human evaluation in every setting. Its rule-based design is especially useful where the desired outcome can be specified precisely; it is less informative for open-ended qualities that lack a checkable target.

The scaling experiments show early signs of generalization to unseen reasoning tasks. That wording is appropriately cautious. It indicates a positive transfer trend rather than a settled demonstration of broad video-world understanding. Still, the combination of controlled task construction, rule-based diagnosis, and held-out-task evaluation gives researchers a more useful instrument for testing that question than clip realism alone.

Caveats in Practice

The suite evaluates a defined taxonomy of 200 tasks, so its coverage is not equivalent to the open-ended distribution of real-world video. Rule-based scoring also depends on tasks admitting explicit success criteria, which may favor structured environments and specified goals. Finally, “early signs” of transfer leave open how far the observed generalization extends across longer horizons, richer scenes, and task families outside the benchmark taxonomy.