MemT Internalizes History While Cutting KV Storage

Long-context language models usually retain prior interaction through an ever-growing KV cache or retrieve external text at query time. Both approaches keep history outside the model’s recurrent computation: the former makes per-session storage grow with every token, while the latter depends on retrieval quality and adds another system component. MemT-1.0 instead treats accumulated interaction state as part of inference itself. Its central bet is that a fixed-size learned memory can retain enough procedural and factual information to answer later queries without replaying the full transcript.

Core Contribution

MemT-1.0 is a family of memory foundation models built on Metis, with MemT-1.0-Nano and MemT-1.0-Flash as the reported variants. The paper’s distinguishing move is to update native memory at token granularity within each turn rather than append a separate textual memory or retrieve a document on demand. The authors combine Gated DeltaNet-2 with a stable update rule and Mixture-of-Storage (MoS), which routes memory computation across experts to expand capacity.

That design changes the storage curve. For a fixed configuration, each trajectory owns dynamic memory states while model weights remain shared; the number of state elements is set by the configured memory layers and capacity rather than the length of the interaction. The paper reports $O(1)$ state with respect to history length, in contrast to the $O(N)$ growth of retained representations. This is a meaningful systems property, although it necessarily creates a compression problem: a bounded state must decide which historical detail survives.

Technical Approach

Training proceeds in two stages. During continued pre-training, sequences are split into consecutive, non-overlapping segments. The backbone attends only to the current segment, while state from earlier segments is passed through native memory; memory is reset only at the start of an independent sequence. During post-training, a full-context teacher produces responses over historical context, and the memory model is trained under teacher forcing with cross-entropy and KL distillation on response tokens. Historical tokens can shape memory without receiving direct token-level loss.

MoS makes the memory block less monolithic by using storage and utilization routers. The former modulate expert erase, write, and decay behavior; the latter mix gated expert readouts. The infrastructure is as consequential as the cell design: memory writes must respect trajectory order even while attention and projections are batched across a training pack. The authors use chunked updates and snapshot reads, then execute MoS in complete-step windows to limit the lifetime of expert projections and their backward activations.

Figure 1 summarizes the intended serving outcome: a persistent trajectory state replaces the full historical KV cache after history preparation.

Results and Analysis

The systems measurements make the storage argument concrete. On a single-GPU MoS block with a 32K-token pack, a 4,096-token window reduced peak memory from 47.83 GiB to 9.49 GiB, an 80.1% reduction, while median forward-plus-backward time rose from 463.0 ms to 470.6 ms, or 1.6%. At 8K tokens, the corresponding windowed overhead was 15.0%, so the trade-off is workload-dependent rather than free. In the online two-GPU, single-request BEAM experiment, query latency remained nearly constant as history grew; at 64K history, Nano used about 33× less logical per-trajectory state than full context. The headline 128K result is 1.86× end-to-end inference speedup and 66× lower persistent storage relative to a full-context KV cache.

The agent experiments offer more direct evidence that compression can retain actionable information. On SWE-bench Verified’s 500 tasks, MemT-1.0-Nano with native memory reached 69.73% average success in the harness-triggered setting, above 66.07% with no experience and 69.13% with rank-1 textual experience; its pass^3 was 62.80%, versus 54.00% without experience. On the 300-task SWE-bench Multilingual set, the same setting reached 63.56% average success and 55.33% pass^3, compared with 60.00% and 45.33% without experience.

Those results support reuse of externally constructed procedural experience, especially in repeated-run consistency. They do not establish autonomous lifelong learning. The case studies show factual recall at 4K tokens, abstention when a 16K history lacks the requested fact, chained rule application at 32K, and correct tracking of a settled update across 65,587 tokens. They are useful behavioral checks, but curated examples cannot demonstrate reliable memory fidelity across open-ended interactions.

Limits in Practice

The authors explicitly identify the fixed-size state as the main constraint: as interactions accumulate, new information is folded into an already compressed representation, forcing trade-offs between old detail and recent updates. Agent gains also rely on externally constructed experience, making performance contingent on experience quality and retrieval. Finally, memory updates themselves add computation, so reducing textual context does not guarantee lower total execution cost.