Generative Latents Push Video Compression Below 0.005 Bpp

Video codecs are usually asked to preserve signal fidelity while spending fewer bits. At extremely low rates, that objective becomes brittle: motion, texture, and semantic content compete for the same small budget, and a distortion-optimized codec can produce frames that are numerically defensible but perceptually poor. This paper reframes the target as perceptual reconstruction under severe coding constraints. The authors introduce Group-of-Latents, a latent-space compression scheme that sends only selected anchor information and asks a generative model to reconstruct the rest.

The central claim is not that generative video compression should replace conventional codecs across the rate curve. It is narrower: below about 0.005 bpp, the method can allocate bits to structural anchors while using a pretrained diffusion-transformer prior to restore textures and motion that would otherwise be impossible to transmit directly.

Core Contribution

Group-of-Latents separates two jobs that conventional low-rate video compression often entangles. The transmitted stream preserves sparse spatial anchors, called I-latents, while the missing inter latents, or P-latents, are regenerated during decoding. The paper’s visual comparison frames this as a resource-allocation problem: existing methods spend the same coding budget on texture and motion, while GoL reserves bits for perceptual anchors and delegates temporal synthesis to a generative prior.

Figure 1 shows that design contrast: low-bitrate codecs compress motion and texture together, while the proposed method explicitly splits anchor preservation from motion generation.

That split is the genuine novelty. The method is not just adding diffusion post-processing after a codec. It changes what is coded: selected latent groups, structural masks, and a text condition are transmitted, while untransmitted latent positions are treated as noise and denoised jointly with the anchors.

Technical Approach

The pipeline has three main components. First, a causal tokenizer maps video into a latent sequence suitable for autoregressive temporal structure. Second, a Group-of-Latents mask partitions the sequence into I-latents and P-latents. Third, an I-latent Deep Compression Module, or I-DCM, compresses the selected I-latents into the bitstream. The encoder also extracts a text description, which becomes an additional decoding condition.

At the decoder, reconstructed I-latents are interleaved with pure noise according to the GoL mask. A DiT-based Unified Latent Denoising Module, U-LDM, then produces the full latent sequence under two conditions: the sparse structural latent input and the text prompt. In practical terms, the transmitted I-latents define where the generated video must stay anchored, while U-LDM fills in both intra-frame perceptual detail and inter-frame dynamics.

Figure 2 is the clearest schematic of the method: the left side shows the compression and decoding path, and the right side expands U-LDM’s masked latent denoising architecture.

This is a sensible architecture for the target regime. If there are too few bits to preserve all frame details, spending them on semantically useful anchors and using a video prior for the rest is more coherent than asking quantization alone to carry appearance and motion.

Results and Analysis

The paper evaluates the method on MCL-JCV and UVG, with rate-distortion and rate-perception plots over the extreme-low-bitrate region. Figure 4 marks the sub-0.005 bpp band and reports both distortion-oriented and perception-oriented behavior, including KID computed on the combined test set for stability. The authors report that their method operates in this band while preserving richer spatial detail and more consistent motion than traditional and neural codecs at comparable rates.

Figure 4 summarizes the main quantitative evidence: the curves compare rate-distortion and rate-perception behavior on MCL-JCV and UVG, with bpp shown on a logarithmic scale.

The supporting visual comparisons matter because this is a perceptual codec. Figure 6 compares methods at a matched bitrate, while Figure 7 varies the GoL configuration to show how the choice of latent grouping changes reconstruction behavior. The ablation figure also indicates that U-LDM is doing more than cosmetic enhancement: removing or changing the masked latent generation setup visibly affects spatial and temporal quality.

The user-facing evidence includes a preference study, shown in Figure 8, where the bars report pairwise preference rates for the proposed method. That does not prove frame-level faithfulness, but it supports the paper’s perceptual framing: at these bitrates, subjective preference is part of the evaluation, not just distortion metrics.

Limitations

The trade-off is fidelity versus perceptual plausibility. Because P-latents are synthesized rather than transmitted, the decoder can generate details that look realistic without being exact. That is acceptable for some human-viewing applications, but risky for archival, scientific, surveillance, or forensic use. The method also depends on a pretrained DiT prior, a causal tokenizer, and a text-conditioning path, so deployment cost and failure modes differ from ordinary codecs. The paper shows evidence for extreme-rate perceptual compression, not a general replacement for VVC, AV1, or learned codecs across all bitrates.