Mobius Separates Knowledge From Reasoning for Faster Inference

Large language models entangle factual storage and token-to-token computation inside every transformer block. That coupling makes a model expensive to run even when a query requires only a small subset of its learned knowledge. Intern-S2-Mobius proposes a different division of labor: globally shared feed-forward memory stores knowledge vectors, while multiple self-attention reasoners retrieve and compose those vectors through hidden states that act as a cache and carrier.

Core Contribution

The paper’s central claim is architectural rather than a new training objective. Mobius-v0 shares one memory component across the model and assigns the iterative work of composition to separate reasoning operators. The intended effect is twofold: parameterized knowledge need not be repeatedly replicated across layers, and attention can focus on transforming retrieved information rather than serving as the sole site of both storage and inference.

This distinction matters because it targets a practical trade-off in large models. A system that preserves downstream quality with less training data or lower serving latency is useful only if the new routing arrangement does not erase the benefits of depth. The authors report that a 7B Mobius model trained from scratch reaches a similar downstream score to a 7B Transformer baseline after using 62.6% of the baseline’s training data. For a converted and continually pretrained model, they report similar downstream performance to Qwen3.5-35B with nearly 4× end-to-end inference speedup.

Technical Approach

Mobius uses hidden states as the interface between its two roles. Reasoners repeatedly query the shared FFN memory for relevant knowledge vectors, then pass the resulting representations back into subsequent reasoning operations. In the paper’s framing, this turns the network into a loop: memory supplies content and attention-based reasoners determine how that content should be combined.

The architecture also makes routing behavior observable. Figure 6 compares expert-selection frequency by layer for a 7B model trained from scratch and a 35B model obtained through architecture conversion plus continual pre-training. The converted model’s experts are reordered specifically to expose inherited routing structure. That visualization supports the narrower point that conversion retains organized expert-use patterns; it does not by itself establish that the pattern causes the reported quality or speed results.

The model is also paired with multi-token prediction behavior. Figure 8 applies a layerwise prediction lens under the same teacher-forced context to Mobius and a Qwen3.5 baseline. Mobius produces more target-aligned intermediate predictions and a five-token draft accepted in full, whereas the baseline produces a two-token accepted prefix. This is a useful mechanistic example of why speculative-style decoding can benefit from better intermediate drafts, though it is an illustrative case rather than an aggregate benchmark.

Results and Analysis

The reported efficiency result is a nearly 4× end-to-end inference speedup for continually pretrained Intern-S2-Mobius-35B relative to Qwen3.5-35B while maintaining a similar downstream score. Alongside the from-scratch 7B comparison, this places the architecture in both a native-training and an architecture-conversion setting.

The speed claim is consequential, but its interpretation depends on evaluation conditions. The supplied material does not expose the hardware configuration, latency distribution, prompt lengths, batch sizes, or exact downstream benchmark values. Likewise, the 62.6% training-data comparison is promising, yet “similar” leaves the size and variance of any quality gap unspecified here.

Evidence and Scope

The paper presents both a from-scratch 7B experiment and a 35B architecture-conversion setting, which is stronger than a single demonstration. It also includes routing and prediction-lens diagnostics rather than relying only on final scores. The evidence supports treating Mobius as a credible efficiency-oriented architecture proposal. It does not yet show whether the same trade-off holds across long-context tasks, diverse reasoning domains, or production serving conditions.