vllm.cpp is a from-scratch C++20 inference engine from the LocalAI team that aims to reproduce vLLM's serving behaviour token-for-token while shipping as a single binary. It is Apache-2.0, has 316 stars and 47 forks since its 2 July 2026 creation, and is explicit that it is an independent, unofficial community port — not affiliated with, endorsed by, or sponsored by the vLLM project, the PyTorch Foundation or the Linux Foundation.

The headline claim is a deployment one. Measured on the same NVIDIA GB10, the engine is a 66 MiB binary against a 9.1 GiB vLLM install — roughly 140x less to deploy — serving the same model in 24.88 GiB of peak host memory versus vLLM's 28.18 GiB, with no interpreter in the process and zero bytes of bundled CUDA userspace. Cold start to the first /health is reported at 36.5 s against vLLM's 221.5 s, a figure the project itself marks provisional.

Throughput is where the project's unusual measurement discipline shows. On Qwen3.6-27B (NVFP4) against the vLLM oracle in its production graphed configuration — not --enforce-eager — output is token-for-token identical at every concurrency, and vllm.cpp is nominally ahead at all six points (86.05 vs 82.32 tok/s at c1, 1095.01 vs 1076.25 at c32). The README then argues against its own numbers: only the c1 result, at 4.5%, sits clearly outside the 0.5% run-to-run noise band, so c2 through c32 should be read as ties. The same candour applies to the llama.cpp comparison — prefill 223.8 vs 177.3 tok/s from the same GGUF file, decode a tie at 24.7 vs 25.4 — which is annotated SUPERSEDED because the denominators came from a local fork 65 performance commits deep, with a re-take owed under issue #1003. Against MLX-LM on an Apple M4, prefill TTFT is 1.5% ahead at 524.5 ms but warm total lands at 97.6% of MLX-LM, a gap the project calls real rather than noise.

Architecturally it borrows deliberately rather than reimplementing one engine. vLLM supplies the serving core — continuous batching, block-paged KV, automatic prefix caching, speculative decoding. SGLang supplies scheduling ideas, exposed as documented opt-in toggles: RadixAttention prefix caching, LPM cache-aware admission, jump-forward decoding and custom logits processors, each defaulting to current behaviour so an engine that sets none of them is byte-identical to one built without them. llama.cpp supplies the deployment story: one library behind a flat, exception-free C ABI (include/vllm.h, ABI v23, versioned and checkable at runtime), GGUF straight off the shelf, and CPU compute directly on the compressed blocks for Q4_0, Q8_0, Q3_K, Q4_K, Q5_K and Q6_K with no BF16 expansion — byte-identical to llama.cpp's greedy decode. MLX's GEMM is used where it wins on Apple Silicon.

Coverage is broad for a project seven weeks old: 40 registered architectures, 38 tool-parser families, structured output including GBNF, three speculative decoders (MTP, block-diffusion DFlash and draft-free ngram through the same --speculative-config JSON vLLM takes), image, video and audio input, music generation, external KV offload and Prometheus metrics. MTP is token-identical to vLLM's MTP and about 4% faster at c1; DFlash roughly doubles throughput over spec-off but stays below vLLM, an open bf16 acceptance floor the record tracks. CUDA, CPU, Metal and Vulkan ship from one source tree, with ROCm and Tenstorrent growing.

Release v0.0.2 on 11 August 2026 ships eight server archives across CPU, CUDA, Vulkan, Metal and MLX builds. The caveats are stated by the project rather than found by a reviewer: it is pre-release and under heavy development, internals, CLI flags and server behaviour can change between commits, and the C ABI is the one thing held stable for embedders. Correctness is gated token-for-token across 27 gated architectures; speed is proven on one GPU (GB10, sm_121a) plus the CPU path. Every capability in docs/STATUS.md is labelled correctness-complete, speed-pending, build-only or hardware-blocked.