Flash Linear Attention (`fla`) is the reference implementation library for sub-quadratic sequence modeling, and at roughly 5,400 stars it is smaller than the frameworks that depend on it — which understates its position. When Qwen3-Next shipped Gated DeltaNet, the kernels came from here. When Kimi Linear published KDA, the implementation landed here. The library's actual product is the gap between a published attention-variant paper and a kernel you can train on, and it has spent two and a half years compressing that gap to weeks. ## What It Actually Ships The scope is broader than the name suggests. `fla` covers linear attention, sparse attention, state space models, and hybrid architectures under one interface, with more than thirty named architectures implemented against their original papers — RetNet, GLA, DeltaNet, Mamba2, Mamba3, RWKV-6 and RWKV-7, GSA, MLA, NSA, MoBA, Samba, YOCO, Gated DeltaNet and its GDN-2 successor, KDA, Log-Linear Attention, and MesaNet among them. Each entry links the paper and the code path, which makes the repository usable as an index of the field as much as a dependency. Every implementation is written to be platform-agnostic and is verified on NVIDIA, AMD, and Intel hardware, with an H100 CI badge on the README backing the first of those claims. Install is a single command with a backend extra — `pip install flash-linear-attention[cuda]`, or a two-step path for ROCm, XPU, NPU, and CPU so that `torch` and the matching Triton flavor come from the PyTorch wheel index rather than PyPI. That packaging detail is small but tells you who the users are: people training models on whatever accelerator they were allocated. ## Kernels, Layers, and the Backend Question Three layers of abstraction are on offer. Raw ops in `fla.ops` are the Triton kernels themselves. Layers in `fla.layers` are drop-in token mixers you can substitute into an existing block. Full models in `fla.models` come with HuggingFace `AutoModelForCausalLM` configs, so a GLA model instantiates in three lines and generates through the standard `transformers` APIs. Pretrained checkpoints live under the `fla-hub` organization on HuggingFace. Alongside the mixers, `fla.modules` provides fused training utilities that matter more than they sound: rotary embeddings, RMSNorm and LayerNorm variants with linear fusion to avoid materializing intermediates, norm layers fused with sigmoid or swish gating for RetNet and GLA, a Triton cross-entropy, and a fused linear cross-entropy that skips materializing the full logits tensor. That last one is disabled by default, and the README says why — it trades numerical precision for memory, and can destabilize training. Documenting the tradeoff instead of quietly defaulting it on is characteristic of the project. Recent work has been on backends rather than new architectures: a Gluon backend for AttnRes, FlashQLA support for Gated DeltaNet, TileLang for selected kernels, FlashMoBA for MoBA, and GPT-OSS-style attention sink support in the attention kernels. Context Parallel for KDA and GDN enables distributed training across the sequence dimension. The pattern is one implementation surface with multiple swappable kernel backends underneath. ## Hybrids and Training Hybrid models — interleaving linear attention layers with standard softmax attention — get first-class support rather than being left to the user, which reflects where the research has actually landed. Pure linear attention trades recall for throughput, and most competitive models built on these primitives are hybrids. For end-to-end training, the sibling `flame` project provides a `torchtitan`-based framework, keeping `fla` itself focused on kernels and layers. ## Pros and Cons The strengths are reach and currency. Thirty-plus architectures with paper links make this the most complete single collection of sub-quadratic sequence modeling code available, and new papers land in weeks. Multi-vendor verification is rare in Triton-kernel projects. MIT licensing with 115 contributors and integration into Qwen3-Next means it is load-bearing infrastructure, not a research artifact. The caveats are real. Triton kernels are hard to debug when they break, and the abstraction is thin enough that failures surface as kernel errors rather than framework errors. Coverage breadth means variance in maturity — a 2026 architecture added last month has had far less use than GLA. The 74 open issues against 115 contributors suggests a maintenance load typical of a fast-moving kernel library, and API churn is visible in the changelog, including a head-first to seq-first input switch. Numerical precision tradeoffs in the fused paths need to be understood rather than assumed. ## Outlook The bet behind `fla` is that attention will not stay quadratic, and the evidence is accumulating in its favor: Qwen3-Next, Kimi Linear, and MiniMax have all shipped production models using linear or hybrid attention. As long as that continues, a library that turns each new paper into a tested kernel across three hardware vendors occupies a position that is hard to displace. ## Conclusion Flash Linear Attention is worth adopting for anyone training sequence models who wants to try an architecture other than standard transformer attention without writing Triton. It is also the fastest way to read the sub-quadratic attention literature, since every entry pairs the paper with running code. Teams that need a stable API surface over a long project should pin a version and read the changelog before upgrading.