**Colibrì** runs frontier Mixture-of-Experts models — 744B to 2.8T parameters — on hardware you already own, by refusing to treat "fits in memory" as a precondition. The engine is a single C file per model family with no BLAS, no Python at runtime and no required GPU, and it treats VRAM, RAM and NVMe as one placement hierarchy rather than three separate capacity limits. The repository was created on **1 July 2026** and has reached roughly **22,900 stars and 2,450 forks** in five weeks, with **v1.5.0** released on 5 August and 30 contributors. It is Apache-2.0 licensed. ## The Sparsity Argument A 744B-parameter MoE model activates only about 40B parameters per token, and only ~11 GB of those change between tokens — the routed experts. Colibrì builds its whole design on that gap. The dense part (attention, shared experts, embeddings, ~17B params) stays resident in RAM at int4, about 9.9 GB. The 19,456 routed experts — 75 MoE layers × 256, roughly 19 MB each — live on disk at ~370 GB and are streamed on demand. The README's framing is that this is **a JIT, but for weights**. A compiler JIT never compiles the whole program; it watches what actually runs and compiles hot paths. Colibrì makes the same bet on a 744B parameter space: parameters are data to be staged across a hierarchy exactly when the router proves they are needed. Measured routing heat drives a per-layer LRU cache and a learned pinned hot-store recorded in `.coli_usage`, so the engine gets faster the more a given workload is run. The design guarantee is drawn sharply: **placement decides speed, never semantics**. Insufficient fast memory may make a run slow; the project states it must not quietly redefine the model's precision or router behaviour. ## The Streaming Path | Technique | What it does | |---|---| | Batched expert unions | Each expert's three matrices are stored adjacent and read in one `pread` | | Async I/O pool (`PIPE=1`) | Loads missing experts while resident ones compute | | Router lookahead (`PILOT=1`) | Prefetches the next layer's experts — routing is measured **71.6% predictable one layer ahead** | | `O_DIRECT` | Bypasses the page cache; +34% decode measured on one Blackwell/Windows box | | Dual-SSD striping | A second full copy on another drive; experts hash-routed by measured bandwidth, aggregate ≈ sum of both | | MLA compressed KV | 576 floats/token instead of 32,768 — **57× smaller** — persisted across restarts for warm reopens | Backends span CPU, CUDA, Metal and Vulkan. The Vulkan path is notable for reaching GPUs the vendor stacks have dropped, including the RX 580, and the project reports it as competitive with ROCm on RDNA4. Five model families run today on the same `coli chat` / `coli serve` / `coli web` front end: GLM-5.2 (744B), Inkling (975B), Kimi K3 (2.8T), DeepSeek V4 Flash (284B) and OLMoE (7B). Kimi K3 needs no conversion — its QAT-trained MXFP4 experts stream straight from the original Hugging Face shards. ## Measured Results The benchmark ladder is published with hardware attached rather than as headline numbers: | Hardware | Decode | |---|---| | 6× RTX 5090, full expert residency | 5.8–6.8 tok/s, TTFT ~13 s | | 128 GB CPU-only desktop | ~1.8 tok/s warm | | Single RTX 5070 Ti laptop-class box | 1.07 tok/s | | 25 GB dev box, cold | 0.05–0.1 tok/s | That last row is the project's own "honest baseline" — the floor where it started, kept in the table rather than dropped from it. The forward pass is validated against a `transformers` oracle at typically 30-32/32 on teacher-forcing, and DSA sparse attention is verified by forcing full-key selection to reproduce dense attention exactly. ## Caveats **Speed is set by your disk, and it is slow.** A fraction of a token per second on a slow drive is the realistic expectation for the large models. Colibrì makes frontier models *runnable*, not *interactive* — anyone reading the star count as "ChatGPT at home" will be disappointed by the tok/s column. **The storage requirement replaces the memory requirement.** GLM-5.2 needs ~372 GB of disk, Inkling ~469 GB, Kimi K3 ~1.6 TB. The hardware barrier moves from scarce VRAM to a large fast SSD; it does not disappear. **Container choice materially affects quality.** The project documents that older per-row int4 mirrors measure ~9pp worse on quality and caused never-terminating generations, and that the MTP speculation head must be int8 — an int4 head collapses draft acceptance to 0–4%. These are recorded as fixed issues, but they show how much correctness rides on picking the right checkpoint. **Most optimizations are explicitly unproven.** The README carries a table of open hypotheses with "experiment still needed" columns, and notes MTP measuring a 32% *loss* around 85% expert hit rate. This is unusually candid for a repository at 22k stars — but it does mean the default configuration is not a settled answer for any given machine. **86 open issues at five weeks old.** Release cadence is fast and the surface is wide (four backends, five model families, three operating systems). Expect to be measuring, not just installing. ## Verdict The contribution here is a reframing rather than a kernel trick: MoE sparsity is usually spent on *compute* savings, and Colibrì spends it on *placement* instead. Treating the parameter space as a staging problem with a learned cache — rather than a residency requirement — is what lets a 744B model answer on a desktop at all. The honesty is worth as much as the engine. Benchmarks come with hardware, commit and command; negative results stay in the README; the quality cost of quantization is linked rather than elided. Systems engineers curious about the storage-bound end of inference should read the hypothesis table first — it is a better map of what is actually known than the star count is. Anyone who needs throughput should keep renting GPUs.