Hybrid Mamba MoE Trades Dense Scale for Throughput

European language-model releases often face the same trade-off: they aim for open access and local control, but they usually trail larger global baselines or inherit dense-transformer serving costs. This paper presents Soofi S 30B-A3B Base, a German-English foundation model trained on German Industrial AI Cloud infrastructure. The claim is not that it beats the largest proprietary systems. The narrower claim is that a fully open base model, tuned for German and English, can sit near larger dense open models on benchmark quality while being much cheaper to serve at long context.

Core Contribution

The main contribution is a 30B-parameter sparse model that activates about 3B parameters per token. Soofi S combines Mixture-of-Experts routing with a hybrid Mamba-Transformer stack, then pairs that architecture with a deliberately German-heavy data curriculum. That combination matters because the paper is trying to optimize two quantities at once: benchmark quality in English and German, and sustained decode throughput when many long-context requests share a GPU.

Figure 1 captures the paper's central engineering trade-off: aggregate English and German scores are plotted against measured decode throughput at a 40K-token context, rather than against parameter count alone.

Technical Approach

The architecture is built to keep the attention cache small as context length grows. The paper reports that only 6 of 52 layers maintain a KV cache, with 2 KV heads each, while 23 Mamba-2 layers carry a fixed-size recurrent state. The result is an incremental attention-cache footprint of about 6 KB per token per sequence, which the authors report as 11–53× smaller than the dense comparison models.

Training uses roughly 27T consumed tokens, with a pretraining-to-annealing transition at 20T tokens. The effective-token accounting is larger because some sources are repeated by epoch multipliers: Phase 1 contains 23,051.13B effective tokens, Phase 2 contains 6,303.0B, and the long-context extension phase contains 188B. The mixture is organized into seven categories: English Web, Academic & Wiki, SFT, Reasoning, Code, Math, and German.

Figure 3 is useful because it shows that the German result is not an incidental byproduct of multilingual web crawl. German rises from 7.2% of Phase 1 effective tokens to 15.3% of Phase 2, while the mixture shifts toward skill-dense code, reasoning, SFT, and German data during annealing.

The openness story is careful rather than absolute. The authors say Soofi S satisfies OSAID 1.0 through release of weights, intermediate checkpoints, training and evaluation code, hyperparameters, and exact per-source data accounting. Under a stricter open-data standard, they identify one exception: the commercially licensed Genios corpus, which is 1.3% of Phase 1 effective tokens and is documented through aggregate statistics rather than redistributed raw text.

Results and Analysis

The evaluation compares Soofi S against 15 open-source and open-weight base models under a common lm-evaluation-harness setup. Against the open-source group of Alia 40B, EuroLLM 22B, Apertus 70B, and Olmo 3 32B, Soofi S reports the highest English aggregate, +1.5 over Olmo 3 32B, and the highest German aggregate, +5.9 over Apertus 70B. The paper also compares to open-weight models with similar or larger active parameter counts, including Nemotron 3 Nano 30B-A3B, Qwen3.5 35B-A3B, Ministral 3 14B, and Gemma 3 27B.

Figure 4 gives the broad comparison across English, German, code, math, knowledge, and reasoning suites.

The strongest measured advantages are concentrated in German and code. In the open-source comparison, Soofi S leads four of five code benchmarks: +10.8 points on HumanEval, +7.4 on MBPP, +30.0 on HumanEval-DE, and +13.4 on MBPP-DE over the next-best open-source baseline. LBPP is the exception, where Olmo 3 32B scores 32.1 versus 31.0 for Soofi S. Mathematics shows a similar pattern: the paper reports margins of +9.7 on GSM8K-Platinum-DE, +24.2 on Minerva-500, +27.0 on Minerva Math-EN, and +7.5 on Minerva MATH-DE within the open-source subset.

The ablation against architecture-identical Nemotron 3 Nano helps separate architecture from data. The German-English recipe improves GLP-DE by +15.1, HellaSwag-DE by +7.4, the German aggregate by +4.6, and the English aggregate by +0.6. It also has small losses on GSM8K, NaturalQuestions, HellaSwag, and Math-DE. That makes the recipe look like a targeted redistribution of capability, not a free gain everywhere.

Serving results are the other major piece of evidence. At 40K context and batch size 32, Soofi S sustains 4.82k aggregate decode TPS/GPU, reported as 9.2× higher than Ministral 3 14B. From 4K to 256K context, its endpoint aggregate decode rate changes from 4.29k to 4.30k TPS/GPU, with no point more than 34% below the 4K value. Dense baselines fall off as KV-cache reads dominate. This is the clearest support for the architectural choice.

Caveats

The paper is unusually explicit about contamination. GPQA was ingested through a split-selection error because the published file used a label that the pipeline treated as training data. The authors remove GPQA-Diamond and GPQA-Diamond-DE from the reported tables and withdraw the held-out suite aggregates, then describe a revised split-selection and n-gram screening process. That correction improves trust in the report, but it also limits how much weight to put on earlier held-out claims.

The evaluation is still a base-model evaluation. Instruction tuning, preference optimization, and reasoning-specific reinforcement learning are outside the paper. The authors also identify residual gaps on LBPP, SocialIQA, SQuAD, DROP, and NaturalQuestions. For deployment, the throughput results are persuasive for the tested single-B200, TP=1, batch-32 protocol, but production behavior will depend on serving stack, batching policy, and request mix.