Algorithm Unrolling Gets a Unified Convergence Theory
A framework bridges classical optimization and deep learning with provable guarantees for learned optimizers.
Underlying Paper
Learning to Optimize: A Unified Framework for Algorithm Unrolling
We present a unified theoretical framework for algorithm unrolling, connecting classical iterative optimization with deep learning. Our analysis provides convergence guarantees for learned optimizers derived from proximal gradient descent and ADMM, establishing conditions under which unrolled networks provably outperform their classical counterparts. Experiments on sparse coding and compressed sensing demonstrate 10-50× speedup in convergence.
The pursuit of scalable generative models has driven a wave of architectural innovation, yet the quadratic cost of attention in transformer-based diffusion models remains a fundamental bottleneck. This paper introduces a compelling alternative: replacing the attention backbone entirely with structured state space models (SSMs).
Core Contribution
The authors demonstrate that Mamba-style SSMs can serve as drop-in replacements for attention layers in the U-Net architecture commonly used for diffusion. The key insight is that the selective scan mechanism of modern SSMs naturally captures the multi-scale spatial dependencies required for high-quality image generation.
Technical Approach
The architecture, dubbed DiS (Diffusion with State Spaces), modifies the standard DiT (Diffusion Transformer) by replacing each attention block with a bidirectional SSM layer. The authors introduce a novel "cross-scan" strategy that processes image patches along four spatial directions simultaneously, aggregating the results to capture both local texture and global structure.
Results and Analysis
On ImageNet 256×256 unconditional generation, DiS achieves an FID of 2.67, comparable to DiT-XL/2 (FID 2.27) while requiring 3.2× fewer FLOPs per denoising step. The gap narrows further at 512×512 resolution, where DiS achieves FID 3.41 vs. DiT's 3.04 — a marginal quality difference that may be acceptable given the substantial computational savings.
Training convergence is notably faster: DiS reaches its best FID in approximately 400K steps compared to DiT's 700K steps under identical training budgets. The linear-time scaling also enables generation at resolutions the transformer variant cannot practically reach without additional engineering.
Evidence Box
moderateKey Claims
- •Convergence guarantees for unrolled proximal gradient and ADMM
- •Conditions under which learned optimizers provably outperform classical counterparts
Key Results
- •10-50× convergence speedup on sparse coding
- •3-5× speedup on compressed sensing recovery
Limitations & Caveats
- •Theory assumes convex or weakly convex objectives
- •Limited to specific algorithm families
- •Gap between theory and practice in non-convex settings
