Solar Open 2 Extends Open MoE Models to Agentic Context

Long-horizon agent tasks stress language models in two ways that ordinary chat benchmarks understate: trajectories can exceed conventional context lengths, and training signals arrive from multi-step tool use rather than isolated answers. Solar Open 2 is Upstage’s attempt to build an open-weight model around those constraints rather than treating them as after-the-fact fine-tuning problems. The report presents a 250B-parameter Mixture-of-Experts model with 15B active parameters, scaled from Solar Open 1, and evaluates it across English reasoning, agentic, and Korean benchmark groups.

The central claim is not just that the model is larger. The authors argue that a long-context architecture, a compute-conscious pretraining recipe, and a post-training system built around specialist teachers are all needed for agentic performance at this scale.

Core Contribution

Solar Open 2 combines three design choices that usually appear separately. First, it uses a 48-layer hybrid attention stack with one softmax-attention layer followed by three linear-attention layers, repeated twelve times, and removes positional encoding entirely. Second, it initializes the reusable 5.69B-parameter shared skeleton from Solar Open 1 while learning the parts changed by the new architecture. Third, it trains agent skills through twelve domain specialists and merges them with Multi-Teacher On-Policy Distillation, so the final model learns from teacher distributions on the student’s own trajectories rather than from static teacher traces.

Figure 3 is the most useful architectural view: it shows the repeated softmax-plus-linear pattern, the MoE block with 320 routed experts and one shared expert, GQA with an output gate, and the KDA linear-attention layer with the negative-eigenvalue extension $\beta = 2\sigma(\cdot) \in (0,2)$.

Technical Approach

The architecture is built for a 1M-token window. The report’s hybrid stack keeps occasional softmax layers while using linear attention for most layers, a trade-off meant to preserve global mixing without paying full quadratic cost throughout the network. The no-positional-encoding design is part of that long-context story: the model is not relying on a fixed positional scheme that must be extrapolated at deployment length.

Pretraining is framed as value per token under a fixed compute budget. The authors start from a 20T-token pool, then use quality- and rarity-aware curation plus mixture-ratio optimization to produce a 10T-token training mixture. The selective transfer from Solar Open 1 reduces the cost of moving from the earlier 100B model to the new hybrid MoE design, but the report is careful that only the shared skeleton transfers directly; the new components are trained through full pretraining.

The post-training section is more system-specific. For reinforcement learning, the authors describe fully asynchronous rollouts to avoid idle devices when agent trajectories have long-tailed completion times. They add a fresh-token-fraction gateway: tokens record the policy version that generated them, stale prefixes can be masked, and trajectories are dropped when the fresh-token ratio falls below a threshold. They also use length- and staleness-aware sampling, direct double-sided importance sampling at token level, and filtering for environment failures such as file-system errors or container crashes.

MOPD then consolidates twelve specialists spanning reasoning, agents and tools, and preference and alignment. Each prompt is routed to one specialist teacher, and the student minimizes per-position reverse KL against the routed teacher over the full vocabulary. The report gives concrete systems constraints here: the vocabulary is 196,608 tokens, a naive full distribution would be about 26 GB per microbatch in fp32, and the implementation avoids materializing the full matrix by sending pre-lm_head hidden states and reducing logits in 1,024-token tiles.

Results and Analysis

The benchmark picture supports a narrower but still meaningful claim: Solar Open 2 is competitive with the strongest open-weight models below 320B total parameters, and it is especially aimed at Korean and agentic work. The evaluation compares against DeepSeek-V4-Flash, MiMo-V2.5, Command A+, Mistral Medium 3.5, and Solar Open 1, with Korean comparisons also including Claude Haiku 4.5 and GPT-5.4 mini fast-tier APIs.

On English knowledge and reasoning, the report highlights the best compared score on MMLU-Pro at 86.2 and says Solar Open 2 leads on LiveCodeBench v6. On agentic evaluation, it reports leading performance on the APEX-Agents suite and includes GDPval-AA v2 as an ELO-style measure. On Korean evaluation, the authors report the highest average among all compared models and emphasize Ko-GDPval, an in-house Korean officework-agent benchmark. The tokenizer analysis supports part of that result: on Ko-GDPval task prompts, Solar Open 2’s unchanged Solar Open 1 tokenizer ranks first among twelve tokenizers and is about 24% above the best global model tokenizer, MiniMax-M3 at 3.54 bytes per token.

The strongest evidence is the breadth of evaluation and the detailed training-system account. The main caveat is reproducibility: several of the most distinctive claims depend on internal benchmarks, internal harness settings, and a technical report rather than independently reproduced measurements. The paper makes a plausible case that hybrid long-context architecture and on-policy specialist distillation matter for officework agents, but the public evidence is strongest as a comparative model report, not as an isolated ablation proving each component’s contribution.