Gemma 4 Narrows Open Model Gap With Efficient Multimodality

Gemma 4 is Google DeepMind’s next open-weight model family, and the report is unusually explicit about the engineering trade-offs behind that label. The suite spans effective 2.3B, 4.5B, 12B, 31B dense models and a 26B total-parameter MoE with 3.8B active parameters, all aimed at making multimodal and reasoning-heavy use cases run on varied hardware. The main claim is not just higher benchmark scores over Gemma 3; it is that the gains come from architecture and inference choices that reduce memory pressure while adding vision, audio, long context, and a thinking mode.

Core Contribution

The contribution is a model-family design rather than a single algorithm. Gemma 4 keeps a decoder-only Transformer backbone, but changes several pressure points: local-to-global attention ratios for long contexts, p-RoPE on global layers, KV cache sharing in global attention, quantization-aware training, an autoregressive multi-token prediction drafter, and multimodal encoders or encoder-free projections depending on model size.

The most distinctive architectural move is the 12B model. Instead of using frozen external vision and audio encoders, Gemma 4 12B ingests image patches and raw audio chunks through lightweight projection modules. Images are represented as 48×48×3 RGB patches and projected by a single large matrix with about 35M parameters. Audio is segmented into 40ms chunks at 16kHz, producing 640-dimensional vectors per chunk. That does not make the 12B model universally better than the larger variants, but it tests whether a unified encoder-free route can reduce fragmentation in multimodal model design.

Technical Approach

For long context, the paper uses local sliding-window attention mixed with global attention: 4:1 local-to-global for E2B and 5:1 for the remaining models. It also reuses keys as values in global attention layers, except for E2B and E4B, and reports that the combination reduces the global KV cache footprint by up to 37.5%. The report gives the training infrastructure as TPUv6e or TPUv4, with 4,096 chips for E2B, 6,144 for E4B and 26B-A4B, 12,288 TPUv4 chips for 12B, and 10,240 TPUv6e chips for 31B.

Figure 1 shows the speculative decoding path. The autoregressive MTP drafter receives activations and KV cache from the main model, then predicts future tokens through a separate embedder and 4-layer Transformer block. For E2B and E4B, the authors reduce decoding overhead by replacing a full-vocabulary projection over 262,000 entries with a top-k operation over clusters of tokens, reducing the matrix multiplication from $d \times 262{,}000$ to $d \times 4096$ while preserving a similar acceptance rate.

The multimodal stack is pragmatic. E2B and E4B use a 150M vision encoder and 305M audio encoder; 26B-A4B and 31B use a 550M vision encoder. Figure 2 details the aspect-ratio preserving image resizing scheme: the system chooses a pooled patch layout under a target maximum token count, runs a ViT-style encoder on 16×16 patches, pools 3×3 encoder representations, and sends the resulting soft tokens into the LLM backbone.

Results and Analysis

The strongest evidence is broad benchmark coverage, though almost all of it is self-reported static or leaderboard evaluation. On Arena Text as of June 19, 2026, Gemma 4 31B reaches 1451 Elo, ranked 43, versus Gemma 3 27B at 1366, ranked 157. The 26B-A4B MoE reaches 1438 Elo, ranked 61. The report’s interpretation that 31B is the leading dense open model in that snapshot is supported by the table shown, but the comparison is time-sensitive and leaderboard composition matters.

The static benchmark gains are large. Gemma 4 31B scores 85.2 on MMLU Pro versus 67.6 for Gemma 3 27B, 89.2 on AIME 2026 without tools versus 20.8, 80.0 on LiveCodeBench v6 versus 29.1, and 2150 Codeforces Elo versus 110. The smaller E2B model roughly matches Gemma 3 27B on some measures despite far fewer effective parameters: MMLU Pro is 60.0 versus 67.6, but AIME 2026 is 37.5 versus 20.8 and LiveCodeBench is 44.0 versus 29.1. That supports the paper’s efficiency claim more than the raw 31B scores do.

Vision and long-context results also back the design choices. At maximal supported resolution, Gemma 4 31B scores 85.6 on MATH-Vision, 92.0 on InfographicVQA, and 0.131 on OmniDocBench 1.5 where lower is better, compared with 46.0, 70.6, and 0.365 for Gemma 3 27B. On RULER at 128k context without thinking, Gemma 4 31B scores 96.4 and E4B scores 86.6, both above Gemma 3 27B at 66.0. The weakest E2B long-context result, 70.4 on the same setting, still clears the older 27B baseline.

Caveats

The report is closer to a systems and benchmark release than a controlled ablation paper. It gives useful implementation details, but it does not isolate how much each design choice contributes across the full suite. The model family looks practically strong, especially for open-weight deployment under memory limits, but the evidence should be read as a release evaluation rather than a reproducible scientific decomposition.