openinfer is an LLM inference engine written entirely in Rust and CUDA, with no PyTorch, no ONNX, and no model-framework runtime underneath it — every kernel and scheduler is hand-written. Apache-2.0 licensed and started in February 2026, it has reached roughly 612 stars and 91 forks, cut its v0.1.0 release on 2026-06-13, and is being pushed to daily. It serves models from Qwen3-4B up to the trillion-parameter Kimi-K2, and the numbers it posts against vLLM are specific enough to be worth taking seriously. ## What Removing the Framework Buys Most inference engines are built on PyTorch. That brings enormous capability but also a heavy runtime: a large resident footprint, a compile step, and a process tree. openinfer's argument is that for serving, you do not need any of it, and the measurements are where that argument lives. On a single RTX 5090 running Qwen3-4B in BF16, openinfer idles at **771 MB resident** against vLLM 0.22.1's 3,814 MB — roughly 5× smaller, with openinfer as one process where vLLM's figure is summed across a tree. Cold start to HTTP-ready is **3.0 seconds versus 70.0**. Even against vLLM's warm `torch.compile` cache at 32.7 seconds, openinfer is still about 11× faster to serve traffic. For anything that scales replicas up and down, or for a developer restarting a server repeatedly, that gap is the headline result. ## Where It Wins and Where It Doesn't The benchmark reporting here is unusually honest, and it does not claim a clean sweep. Under Poisson-arrival serving load with 1k-token prompts and 128-token outputs, throughput tracks vLLM step for step through the knee and edges ahead at saturation — 1,794 vs 1,692 tok/s, about 14.0 vs 13.2 req/s at QPS 16. But vLLM keeps a per-token decode advantage at mid load, QPS 8 through 12, and both engines knee at roughly QPS 10–12 where queueing takes over. The clearest win is the warm prefix-cache path, which is the chat and agent hot path — most of a multi-turn prompt is a cache hit. There openinfer's time to first token stays close to flat as context grows: about **9 ms at 1k tokens and 26 ms at 16k, against vLLM's ~96 ms** — a 3.6× gap — with p99 within roughly 1 ms of p50 at every length. That tail-latency tightness is arguably more valuable than the mean. Cold, uncached prefill is at parity, around 1.1 s at 16k. With `--kv-offload`, prefixes evicted from HBM are restored from host DRAM rather than recomputed, turning that 1.14 s cold prefill at 16k into a 126 ms host-tier restore — 9.1×. The resulting ladder is explicit: HBM hit ~26 ms, host restore ~126 ms, cold prefill ~1.14 s. On Qwen3.5-4B against the newer vLLM 0.23.0, the picture flips in places. openinfer leads TTFT at 1024/256 (55.3 vs 66.3 ms), but vLLM leads decode TPOT and output throughput (151.9 vs 137.0 tok/s) and holds the high-concurrency HTTP rows. The project states this plainly rather than selecting the favorable rows, which is a good sign about the rest of its numbers. ## Architecture and Model Coverage The layout is a Rust workspace: `openinfer-server` fronts an OpenAI-compatible endpoint, a runtime contract sits beneath it, and each model line gets its own engine crate — `openinfer-qwen3`, `openinfer-qwen35`, `openinfer-deepseek-v2-lite`, `openinfer-kimi-k2` — owning its config, weights, scheduler, tests, and kernel plan. Shared crates handle kernels, KV cache, and KV offload, with `kvbm-logical` ported from NVIDIA Dynamo. Backends are CUDA and cuBLAS, Triton AOT for Qwen3.5 compatibility kernels, TileLang for GLM5.2 sparse-MLA kernel generation, FlashInfer for paged attention and sampling, and NCCL plus a DeepEP shim for multi-GPU expert parallelism. CUDA Graph is used on the Qwen decode paths. Model support is real but gated behind cargo features, and only `qwen3` is on by default: | Model | Architecture | Params | Gate | |-------|-------------|--------|------| | Qwen3-4B / 8B | Full attention (GQA 4:1) | 4B / 8B | default, zero Python | | Qwen3.5-4B / 9B / 27B | Hybrid Gated DeltaNet + full attention | 4B / 9B / 27B | `--features qwen35`, build-time Triton | | DeepSeek-V2-Lite | MoE + expert parallel | 15.7B total / 2.4B active | `--features deepseek-v2-lite`, 2-GPU EP2 | | Kimi-K2-Instruct | MLA + MoE + Marlin INT4 | 1T total / 32B active | `--features kimi-k2`, 8-GPU EP | The stock build genuinely needs no Python at all — `cargo run --release` after a `huggingface-cli download` is the whole path. Model type is auto-detected from `config.json`. ## Pros and Cons The strengths are coherent. A 771 MB, 3-second-start single process is a materially different deployment object than a PyTorch server, and it makes autoscaling and local development cheaper in ways throughput charts do not capture. Flat, tight-tailed warm-cache TTFT targets exactly the workload agents generate. Reaching Kimi-K2 at trillion scale with hand-written kernels is a real engineering result for a five-month-old project. The per-model crate boundary keeps model-specific hacks from leaking into shared code. Benchmark reports publish flags, commits, and losing rows, and the docs include engineering deep-dives rather than marketing. Apache-2.0 with proper NOTICE attribution for the ported Dynamo components is clean. The limitations are equally clear. This is NVIDIA-only by construction — CUDA, cuBLAS, NCCL, driver R545 or newer — so there is no CPU, Apple, or AMD story. The feature-gate design means **switching model lines requires recompiling the server**, which is awkward for anyone serving more than one model. Only five model families are supported, against the dozens vLLM covers, so it is a fast engine for a narrow menu. General-purpose quantization for the Qwen lines is explicitly not implemented; INT4 and FP8 exist only as model-specific paths for Kimi-K2 and GLM5.2, meaning the BF16 Qwen models cannot be shrunk. The API surface is `/v1/completions` only — the raw text-completion endpoint — so chat templating, tool calling, and the `/v1/chat/completions` shape most clients now assume are not covered, which limits drop-in replacement for agent stacks despite the OpenAI-compatible label. Sampling and logprob support varies by model. Larger models need multi-GPU expert-parallel setups (2 GPUs for DeepSeek-V2-Lite, 8 for Kimi-K2, NCCL 2.27+), so the impressive scale numbers are not reachable on one card. And with 134 open issues against 612 stars, one tagged release, and very few watchers, this is early-stage software with a small community behind it. ## Outlook The framework-free approach is a bet that the deployment characteristics of inference — footprint, start time, process count, tail latency — matter enough to justify rewriting the stack from kernels up. For serverless and edge-adjacent serving, where a 70-second cold start is disqualifying, that bet looks strong. Whether it generalizes depends on breadth: model coverage and a chat-completions endpoint are the two gaps standing between a fast engine and a usable default. The published TPOT losses to vLLM also suggest the mature engine's scheduling still has an edge under sustained load. ## Conclusion openinfer is worth evaluating for teams serving Qwen3 or Qwen3.5 on NVIDIA hardware where cold-start time, memory footprint, or warm-cache tail latency is the binding constraint — scale-to-zero deployments and agent workloads with heavy prefix reuse are the natural fit. It is not yet a vLLM replacement for general production: check that your model is on the supported list, that `/v1/completions` is enough for your clients, and that you can accept a rebuild to switch models. As a demonstration that a Rust and CUDA engine can reach frontier scale and hold its own on latency, it is one of the more credible systems projects to appear this year.