RedKnot is a long-context inference acceleration layer built on top of [SGLang](https://github.com/sgl-project/sglang), released by the RedNote Machine Learning Framework Team (小红书机器学习引擎框架团队). Opened in June 2026 under Apache-2.0, it has gathered roughly 1,517 stars and 600 forks, and it ships with an arXiv paper rather than just a README. Its central claim is narrow and testable: long-context prefill spends most of its FLOPs on work that does not change the answer, and a serving system can identify and skip that work. ## The Core Argument The project's thesis fits in one sentence from its own documentation — **not every attention head needs the full KV, and not every token needs to go through the full FFN**. Rather than treating a long prompt as an undifferentiated block of tokens, RedKnot classifies each `(layer, kv_head)` pair into one of four behavioural classes — `global`, `local`, `retrieval`, or `dense` — and lets the KV storage and reuse policy differ per class. A head that only ever attends locally does not need a KV cache spanning 32K tokens. Four mechanisms implement this. **Head classification** assigns the four-class strategy from a JSON config, with a profiler to generate it. **Offline KV reuse with RoPE relocation** stores KV for reusable segments ahead of time and recomputes only the tokens that actually matter at serving time, with RoPE relocation handling the positional realignment so the numerics stay aligned. **Elastic Sparsity / Sparse FFN** skips feed-forward computation for tokens with low attention importance. And **SegPagedAttention** provides the runtime substrate: a per-head page table over a segmented KV store, so different head classes can genuinely hold different visible windows instead of sharing one uniform paging scheme. Because it is an extension under `python/sglang/srt/layers/attention/redknot/` rather than a rewrite, the repository retains SGLang's serving machinery — RadixAttention, the zero-overhead scheduler, PD disaggregation, continuous batching, and quantization all still apply. ## What the Benchmarks Show The headline figures are a **50–70% reduction in long-context prefill FLOPs** and a **1.35x–2.2x TTFT speedup**, with the gains growing as context length increases. The per-model tables are more informative than the summary. On **Qwen3-32B / HotpotQA**, the results are the cleanest in the repository — accuracy does not merely survive, it improves: | Context | base F1 | RedKnot F1 | base TTFT | RedKnot TTFT | speedup | FLOPs saved | |---|---|---|---|---|---|---| | 16K | 0.750 | 1.000 | 3.24s | 2.33s | 1.39x | 69.2% | | 24K | 1.000 | 1.000 | 5.25s | 2.96s | 1.77x | 70.9% | | 32K | 0.750 | 1.000 | 7.74s | 4.02s | 1.93x | 72.2% | FLOPs savings hold near 70% while the speedup scales from 1.39x to 1.93x — the pattern you would expect if the skipped work is genuinely proportional to context length. On **Qwen3.5-35B-A3B (MoE) / LongBench**, the speedups are larger but the accuracy story changes. TTFT improves 1.87x at 16K to 2.16x at 64K, and the run is lossless at 16K — but F1 falls from 0.792 to 0.576 at 32K on `multifieldqa_en`, and from 0.875 to 0.750 at 64K on `triviaqa`. The documentation attributes this to the compounding of linear attention and MoE sparsity, and states it plainly rather than omitting the rows. On **Mistral-7B-Instruct-v0.3**, the system metrics stay stable at ~1.35x and ~50% FLOPs saved, but the F1 column is visibly noisy in both directions because the 7B baseline is itself weak at long-context RAG. The project also publishes a **Known Issues** section: on **Llama-3.3-70B-Instruct** the baseline runs fine but the RedKnot decode path degrades into repeated tokens under long-context LongBench, single-GPU INT4 readily OOMs, and multi-GPU bf16 triggers cross-device errors. Naming a model your own method breaks on is not standard practice in this category of release. ## Reproduction and Scope Benchmarks live in `test/srt/redknot/` with a one-command entry point (`bash run_all_rag.sh`), per-model scripts for the five model families, and environment variables covering sample count, context lengths, dtype, and `torch.compile`. Logs are written to `rag_logs/`. The comparison baseline is described as an honest dense baseline — one full FlashAttention-2 prefill — rather than a weakened reference. The scope limits are stated up front. Only the base version is open-sourced for now; **DeepSeek-V4 and the full Qwen 3.5 series are deferred to a next release**. `Qwen3.5-*` MoE and DeepSeek-V4 require transformers 5.x, so the repository ships a separate `.venv_tf5` environment because system transformers 4.57 cannot load them. Huawei Cloud is adapting the work for Ascend NPUs, but that is explicitly work in progress with no published results. ## Pros and Cons The strengths are real. Head-level heterogeneity is a more surgical lever than blanket KV quantization or fixed-window truncation, and SegPagedAttention gives it a runtime that can actually express per-head windows. Building on SGLang instead of forking a new engine means the acceleration composes with a production serving stack rather than competing with one. The Qwen3-32B numbers pair ~70% FLOPs savings with F1 at or above the dense baseline, which is the combination the approach needs to justify itself. Publishing degraded rows, a broken model, and an arXiv paper together suggests the remaining numbers were not cherry-picked. Apache-2.0 with clear attribution to SGLang and vLLM is clean. The caveats matter just as much. The benchmark hardware note reads **4 samples per model** — small enough that individual F1 values, particularly the jumps to a perfect 1.000, should be read as directional rather than precise. The repository has **no tagged release**, three contributors, and an organization created a day before the repo with a single project and few followers, so the maintenance track record is essentially nonexistent. The MoE accuracy degradation at 32K and beyond is a genuine cost, not a rounding error, and the method's benefit depends on head-class configs (`head_class/*.json`) that must exist and be correct for your model — the Llama-3.3 failure is attributed partly to config quality. Everything is NVIDIA-measured on L20Y hardware, results are prefill/TTFT-centric rather than sustained-throughput, and adopting it means tracking a specific SGLang integration point rather than an upstream feature. ## Outlook The interesting question is whether head classification becomes a general serving primitive or stays a per-model tuning exercise. If profiling reliably produces good head configs across architectures, this is a broadly reusable optimization; if each model needs hand-checked configs — as the Llama-3.3 note hints — the operational cost may outweigh the FLOPs saved for smaller deployments. Upstream adoption into SGLang proper, and the promised DeepSeek-V4 and Qwen 3.5 support, are the signals worth watching. ## Conclusion RedKnot is worth evaluating by teams already serving long-context RAG workloads on SGLang and NVIDIA hardware, where prefill latency at 16K–64K tokens is the binding constraint. The Qwen3-32B results are strong enough to justify a trial, and the paper plus reproduction scripts make verification straightforward. Treat the accuracy numbers as preliminary given the four-sample evaluation, budget time for head-class configuration on your own model, and do not expect the MoE path to be lossless at long context. As an early public artifact from a corporate ML systems team, it is unusually candid about where it does not work.