Mini Activations Narrow Frontier Model Gaps

Long-horizon agent use changes what language models are asked to optimize. The paper argues that production workloads now involve repository repair, web research, spreadsheet work, slide generation, and multi-step tool use rather than only single-turn QA. MiniMax-M2 targets that setting with a sparse Mixture-of-Experts design: the flagship model has 229.9B total parameters but activates only 9.8B per token, then spends the saved per-token budget on long-context training, agent data, and reinforcement learning.

Figure 1 gives the paper's headline comparison: MiniMax-M2.7 is not uniformly ahead of larger closed systems, but it is close on several agentic benchmarks despite the much smaller activated footprint.

Core Contribution

The central claim is architectural and procedural rather than just scale-based. MiniMax-M2 keeps active compute near a 10B-parameter dense model while scaling total capacity through 256 fine-grained experts, 8 activated experts per token, sigmoid routing, and learned expert bias terms. Around that base model, the authors build a post-training stack aimed at verifiable agent trajectories: software tasks with runnable Docker environments and test-based rewards, application-development tasks judged through execution and UI interaction, open-web research grounded in retrieved evidence, and office tasks judged against produced artifacts.

The new part is the coupling of that sparse backbone with an agent-native RL system named Forge. Instead of treating a tool-using agent as a special wrapper around a language model, Forge models the LLM output as the policy action and treats context management, tools, memory, and harness behavior as environment dynamics. That lets the training system collect trajectories from both white-box agents, where context transitions are visible, and black-box agents, where only request-output observations are available.

Technical Approach

The pre-training model is a 62-layer decoder-only Transformer with hidden dimension 3,072, vocabulary size 200,064, GQA with 48 query heads and 8 key-value heads, RoPE throughout, full attention, and a Multi-Token Prediction module. The authors report ablations at a smaller 17.8B-total, 2B-activated scale: fine-grained experts improve MATH from 19.6 to 24.1 and HumanEval from 29.7 to 32.5, while MTP improves MATH from 19.6 to 21.3 and KorBench from 14.1 to 15.0. They also report that hybrid sliding-window attention looked acceptable on some short-context tasks but degraded long-context retrieval and translation: RULER 128K CWE falls from 90.0 to 72.0 and MTOB e-k BLEURT from 60.0 to 45.0. That explains the choice to keep full attention despite its cost.

Pre-training totals 29.2T tokens, with 19.9T in the constant phase and 9.3T in decay, while context length is extended from 8K through 32K to 192K tokens. Post-training then adds supervised fine-tuning for interleaved thinking and Forge-based RL. The RL formulation uses CISPO, composite rewards that mix process feedback, completion-time incentives, and task performance, plus mixed-domain training across reasoning, coding, agent, and general data. The infrastructure details matter: windowed-FIFO scheduling addresses rollout times ranging from seconds to hours, and prefix-tree merging avoids recomputing shared long prefixes, with the paper claiming up to 40× training speedup for that component.

Results and Analysis

The strongest evidence is the breadth of evaluation. On coding, M2.7 scores 56.2 on SWE-bench Pro, 76.5 on SWE-bench Multilingual, 52.7 on Multi-SWE-bench, 57.0 on Terminal-Bench 2.0, and 66.6 on MLE Bench Lite. It trails GPT 5.4 on SWE-bench Pro and Terminal-Bench 2.0, but beats the listed closed baselines on SWE-bench Multilingual, Multi-SWE-bench, and MLE Bench Lite. The MLE Bench Lite case is the cleanest self-evolution evidence: across 22 competitions and three 24-hour trials, M2.7 averages a 66.6% medal rate, and the best run produces 9 gold, 5 silver, and 1 bronze medal.

The cowork and reasoning results are mixed but useful. M2.7 reaches 77.8 on BrowseComp, 75.2 on Wide Search, 50.0 on GDPval-AA, 46.3 on Toolathlon, 63.3 on MEWC v2, and 57.0 on Finance Modeling Pro. In reasoning, it scores 94.2 on AIME 2026, 89.8 on GPQA-Diamond, 76.0 on IFBench, 72.0 on AA-LCR, 47.0 on SciCode, 28.0 on HLE, and 81.8 on MMLU-Pro. Those numbers place it in the same band as closed frontier systems on some math and agent tasks, but not across the board: Gemini 3.1 Pro leads GPQA-Diamond at 94.1 and MMLU-Pro at 91.2, while GPT 5.4 leads AIME 2026 at 97.0 and several office-tool benchmarks.

Caveats in Practice

The paper is best read as a systems report from a model developer, not as a fully reproducible academic benchmark. Many evaluations use internal benchmarks or shared scaffolds controlled by the authors, and the paper does not provide enough public release detail to independently verify the training data, RL infrastructure, or model behavior. Still, the ablations and broad benchmark table support a narrower conclusion: sparse activation can reduce active parameter count without preventing competitive agentic performance, provided the training stack is built around executable tasks and artifact-grounded rewards.