Daniel Y. Fu

Sparse Window Attention Extends Context to 128K Tokens

A hierarchical sparse decomposition achieves sub-quadratic long-range attention with minimal quality loss.