Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
**vllm.cpp** is a from-scratch **C++20** inference engine from the **LocalAI** team that aims to reproduce vLLM's serving behaviour token-for-token while shipping as a single binary. It is **Apache-2.0**, has **316 stars and 47 forks** since its **2 July 2026** creation, and is explicit that it is an **independent, unofficial community port** — not affiliated with, endorsed by, or sponsored by the vLLM project, the PyTorch Foundation or the Linux Foundation. The headline claim is a deployment one. Measured on the same NVIDIA GB10, the engine is a **66 MiB binary against a 9.1 GiB vLLM install** — roughly **140x less to deploy** — serving the same model in **24.88 GiB of peak host memory versus vLLM's 28.18 GiB**, with no interpreter in the process and zero bytes of bundled CUDA userspace. Cold start to the first `/health` is reported at **36.5 s against vLLM's 221.5 s**, a figure the project itself marks provisional. Throughput is where the project's unusual measurement discipline shows. On **Qwen3.6-27B (NVFP4)** against the vLLM oracle in its *production graphed* configuration — not `--enforce-eager` — output is **token-for-token identical at every concurrency**, and vllm.cpp is nominally ahead at all six points (86.05 vs 82.32 tok/s at c1, 1095.01 vs 1076.25 at c32). The README then argues against its own numbers: only the c1 result, at 4.5%, sits clearly outside the 0.5% run-to-run noise band, so **c2 through c32 should be read as ties**. The same candour applies to the llama.cpp comparison — prefill 223.8 vs 177.3 tok/s from the same GGUF file, decode a tie at 24.7 vs 25.4 — which is annotated **SUPERSEDED** because the denominators came from a local fork 65 performance commits deep, with a re-take owed under issue #1003. Against **MLX-LM on an Apple M4**, prefill TTFT is 1.5% ahead at 524.5 ms but warm total lands at 97.6% of MLX-LM, a gap the project calls real rather than noise. Architecturally it borrows deliberately rather than reimplementing one engine. vLLM supplies the serving core — continuous batching, block-paged KV, automatic prefix caching, speculative decoding. **SGLang** supplies scheduling ideas, exposed as documented opt-in toggles: **RadixAttention** prefix caching, LPM cache-aware admission, jump-forward decoding and custom logits processors, each defaulting to current behaviour so an engine that sets none of them is byte-identical to one built without them. **llama.cpp** supplies the deployment story: one library behind a flat, exception-free C ABI (`include/vllm.h`, ABI v23, versioned and checkable at runtime), **GGUF straight off the shelf**, and CPU compute **directly on the compressed blocks** for Q4_0, Q8_0, Q3_K, Q4_K, Q5_K and Q6_K with no BF16 expansion — byte-identical to llama.cpp's greedy decode. MLX's GEMM is used where it wins on Apple Silicon. Coverage is broad for a project seven weeks old: **40 registered architectures**, 38 tool-parser families, structured output including GBNF, three speculative decoders (MTP, block-diffusion DFlash and draft-free ngram through the same `--speculative-config` JSON vLLM takes), image, video and audio input, music generation, external KV offload and Prometheus metrics. MTP is token-identical to vLLM's MTP and about 4% faster at c1; DFlash roughly doubles throughput over spec-off but stays below vLLM, an open bf16 acceptance floor the record tracks. CUDA, CPU, Metal and Vulkan ship from one source tree, with ROCm and Tenstorrent growing. Release **v0.0.2** on **11 August 2026** ships eight server archives across CPU, CUDA, Vulkan, Metal and MLX builds. The caveats are stated by the project rather than found by a reviewer: it is **pre-release and under heavy development**, internals, CLI flags and server behaviour can change between commits, and the C ABI is the one thing held stable for embedders. Correctness is gated token-for-token across **27 gated architectures**; speed is proven on one GPU (GB10, sm_121a) plus the CPU path. Every capability in `docs/STATUS.md` is labelled correctness-complete, speed-pending, build-only or hardware-blocked.