Adaptive Distillation Stabilizes One-Step Generative Video Compression

Generative video compression is useful at bitrates where conventional codecs and reconstruction losses tend to erase texture. Recent diffusion-based codecs have improved perceptual quality, but many adapt large text-to-video or image-generation backbones and then pay for multi-step sampling at decode time. This paper introduces GenVC, a video codec whose diffusion model is trained from scratch for compression and then distilled to a one-step decoder.

The central claim is narrower than “diffusion helps compression.” The authors argue that a compression-specific pixel-space video diffusion model can become a standalone codec, and that one-step distillation fails unless the distillation signal is filtered when it pushes reconstructions away from the source video.

Core Contribution

GenVC combines a contextual video codec with a pixel-domain diffusion reconstruction model. The codec transmits a compressed representation, and the diffusion branch reconstructs the decoded GOP conditioned on that representation. Unlike prior generative codecs that inherit billion-parameter pretrained models, the paper reports a 478.0M-parameter diffusion model trained jointly with the codec.

The genuinely new part is Adaptive Score Distillation. Standard distribution matching distillation is meant to compress a multi-step teacher into a one-step student. In this setting, the authors find that direct DMD can produce motion-stalled videos: texture may look plausible, but temporal motion freezes or forms discontinuous staircase artifacts. Their diagnosis is teacher-side guidance failure. When the student’s reconstruction drifts outside the teacher’s training region, the frozen teacher can return a plausible but lower-motion correction, and the DMD update then reinforces the drift.

Technical Approach

The compression model operates on groups of pictures split into consecutive 8-frame chunks. A contextual encoder produces a compact representation, which is quantized and entropy coded; a contextual decoder provides the condition for the pixel video diffusion model. The diffusion model is trained with flow matching in pixel space, predicting the clean GOP from noisy frames and codec conditions.

The architecture uses a global-to-local hierarchy. A coarse spatio-temporal DiT branch models long-range structure on a downsampled token grid, while local Patch Refine blocks recover finer spatial and temporal detail. Patch Modulation injects global features into the local grid, Spatial-Temporal Convs add local mixing, and a Frame Refine head maps features back to RGB pixels. The ablation suggests this hierarchy matters more than parameter count alone: simply enlarging the DiT baseline from 416M to 900M parameters reduces BD-LPIPS by 0.024 and BD-FID by 4.75, while the full 478M hierarchy reaches reductions of 0.087 and 25.51 relative to the DiT-only baseline.

Adaptive Score Distillation adds a per-sample gate to the DMD gradient. For each training sample, the method measures whether the DMD update is aligned with the ground-truth direction from the current reconstruction toward the source video. Aligned updates are kept; poorly aligned updates are downweighted through a smooth sigmoid gate. The final one-step objective combines the gated DMD loss with source-referenced pixel, LPIPS, and optical-flow distortion losses.

Results and Analysis

The evaluation covers HEVC Class B, UVG, and MCL-JCV, with rate-quality curves over LPIPS, DISTS, FloLPIPS, FID, MS-SSIM, and PSNR. Against GLVC, GenVC reports bitrate savings of 59.6%, 69.0%, and 59.0% at matched LPIPS on HEVC Class B, UVG, and MCL-JCV, averaging 62.5%. At matched FID, the corresponding savings are 69.6%, 69.5%, and 74.9%, averaging 71.3%.

The trade-off is visible in the metric mix. GenVC is strongest on perceptual metrics such as LPIPS, DISTS, FloLPIPS, and FID. The paper also says it remains competitive on PSNR and MS-SSIM, but its main advantage is perceptual rate-quality rather than distortion-only reconstruction. That is the right target for generative compression, provided the application tolerates plausible detail rather than exact signal preservation.

Throughput is a major part of the result. The multi-step teacher uses 20-step CFG sampling and decodes 1080p at 0.40 fps on an A100. The distilled one-step GenVC decodes at 15.1 fps on the same device, a 37.8× speedup, while keeping the same encoding throughput. Table I also reports the combined codec plus diffusion size as 578.4M parameters, far below GLVC’s 2,273.3M parameters and GNVC-VD’s 2,334.5M, with the best listed MCL-JCV BD-rate among the compared diffusion codecs on A100.

The ablations support the authors’ mechanism. Removing DMD makes the one-step codec worse than the full model on all reported metrics. Keeping DMD but removing the adaptive gate raises motion error and degrades BD-ME, consistent with the motion-stalling diagnosis. Using gated DMD without source-referenced distortion improves motion error but worsens DISTS, which explains why the final method combines both terms. The evidence is convincing for perceptual low-bitrate reconstruction on the tested datasets; it does not establish that the generated details are acceptable for every downstream use of compressed video.