openinfer is an LLM inference engine written entirely in Rust and CUDA, with no PyTorch, no ONNX, and no model-framework runtime underneath it — every kernel and scheduler is hand-written. Apache-2.0 licensed and started in February 2026, it has reached roughly 612 stars and 91 forks, cut its v0.1.0 release on 2026-06-13, and is being pushed to daily. It serves models from Qwen3-4B up to the trillion-parameter Kimi-K2, and the numbers it posts against vLLM are specific enough to be worth taking seriously.

What Removing the Framework Buys

Most inference engines are built on PyTorch. That brings enormous capability but also a heavy runtime: a large resident footprint, a compile step, and a process tree. openinfer's argument is that for serving, you do not need any of it, and the measurements are where that argument lives.

On a single RTX 5090 running Qwen3-4B in BF16, openinfer idles at 771 MB resident against vLLM 0.22.1's 3,814 MB — roughly 5× smaller, with openinfer as one process where vLLM's figure is summed across a tree. Cold start to HTTP-ready is 3.0 seconds versus 70.0. Even against vLLM's warm torch.compile cache at 32.7 seconds, openinfer is still about 11× faster to serve traffic. For anything that scales replicas up and down, or for a developer restarting a server repeatedly, that gap is the headline result.

Where It Wins and Where It Doesn't

The benchmark reporting here is unusually honest, and it does not claim a clean sweep.

Under Poisson-arrival serving load with 1k-token prompts and 128-token outputs, throughput tracks vLLM step for step through the knee and edges ahead at saturation — 1,794 vs 1,692 tok/s, about 14.0 vs 13.2 req/s at QPS 16. But vLLM keeps a per-token decode advantage at mid load, QPS 8 through 12, and both engines knee at roughly QPS 10–12 where queueing takes over.

The clearest win is the warm prefix-cache path, which is the chat and agent hot path — most of a multi-turn prompt is a cache hit. There openinfer's time to first token stays close to flat as context grows: about 9 ms at 1k tokens and 26 ms at 16k, against vLLM's ~96 ms — a 3.6× gap — with p99 within roughly 1 ms of p50 at every length. That tail-latency tightness is arguably more valuable than the mean. Cold, uncached prefill is at parity, around 1.1 s at 16k.

With --kv-offload, prefixes evicted from HBM are restored from host DRAM rather than recomputed, turning that 1.14 s cold prefill at 16k into a 126 ms host-tier restore — 9.1×. The resulting ladder is explicit: HBM hit ~26 ms, host restore ~126 ms, cold prefill ~1.14 s.

On Qwen3.5-4B against the newer vLLM 0.23.0, the picture flips in places. openinfer leads TTFT at 1024/256 (55.3 vs 66.3 ms), but vLLM leads decode TPOT and output throughput (151.9 vs 137.0 tok/s) and holds the high-concurrency HTTP rows. The project states this plainly rather than selecting the favorable rows, which is a good sign about the rest of its numbers.

Architecture and Model Coverage

The layout is a Rust workspace: openinfer-server fronts an OpenAI-compatible endpoint, a runtime contract sits beneath it, and each model line gets its own engine crate — openinfer-qwen3, openinfer-qwen35, openinfer-deepseek-v2-lite, openinfer-kimi-k2 — owning its config, weights, scheduler, tests, and kernel plan. Shared crates handle kernels, KV cache, and KV offload, with kvbm-logical ported from NVIDIA Dynamo. Backends are CUDA and cuBLAS, Triton AOT for Qwen3.5 compatibility kernels, TileLang for GLM5.2 sparse-MLA kernel generation, FlashInfer for paged attention and sampling, and NCCL plus a DeepEP shim for multi-GPU expert parallelism. CUDA Graph is used on the Qwen decode paths.

Model support is real but gated behind cargo features, and only qwen3 is on by default:

The stock build genuinely needs no Python at all — cargo run --release after a huggingface-cli download is the whole path. Model type is auto-detected from config.json.

Pros and Cons

The strengths are coherent. A 771 MB, 3-second-start single process is a materially different deployment object than a PyTorch server, and it makes autoscaling and local development cheaper in ways throughput charts do not capture. Flat, tight-tailed warm-cache TTFT targets exactly the workload agents generate. Reaching Kimi-K2 at trillion scale with hand-written kernels is a real engineering result for a five-month-old project. The per-model crate boundary keeps model-specific hacks from leaking into shared code. Benchmark reports publish flags, commits, and losing rows, and the docs include engineering deep-dives rather than marketing. Apache-2.0 with proper NOTICE attribution for the ported Dynamo components is clean.

The limitations are equally clear. This is NVIDIA-only by construction — CUDA, cuBLAS, NCCL, driver R545 or newer — so there is no CPU, Apple, or AMD story. The feature-gate design means switching model lines requires recompiling the server, which is awkward for anyone serving more than one model. Only five model families are supported, against the dozens vLLM covers, so it is a fast engine for a narrow menu. General-purpose quantization for the Qwen lines is explicitly not implemented; INT4 and FP8 exist only as model-specific paths for Kimi-K2 and GLM5.2, meaning the BF16 Qwen models cannot be shrunk. The API surface is /v1/completions only — the raw text-completion endpoint — so chat templating, tool calling, and the /v1/chat/completions shape most clients now assume are not covered, which limits drop-in replacement for agent stacks despite the OpenAI-compatible label. Sampling and logprob support varies by model. Larger models need multi-GPU expert-parallel setups (2 GPUs for DeepSeek-V2-Lite, 8 for Kimi-K2, NCCL 2.27+), so the impressive scale numbers are not reachable on one card. And with 134 open issues against 612 stars, one tagged release, and very few watchers, this is early-stage software with a small community behind it.

Outlook

The framework-free approach is a bet that the deployment characteristics of inference — footprint, start time, process count, tail latency — matter enough to justify rewriting the stack from kernels up. For serverless and edge-adjacent serving, where a 70-second cold start is disqualifying, that bet looks strong. Whether it generalizes depends on breadth: model coverage and a chat-completions endpoint are the two gaps standing between a fast engine and a usable default. The published TPOT losses to vLLM also suggest the mature engine's scheduling still has an edge under sustained load.

Conclusion

openinfer is worth evaluating for teams serving Qwen3 or Qwen3.5 on NVIDIA hardware where cold-start time, memory footprint, or warm-cache tail latency is the binding constraint — scale-to-zero deployments and agent workloads with heavy prefix reuse are the natural fit. It is not yet a vLLM replacement for general production: check that your model is on the supported list, that /v1/completions is enough for your clients, and that you can accept a rebuild to switch models. As a demonstration that a Rust and CUDA engine can reach frontier scale and hold its own on latency, it is one of the more credible systems projects to appear this year.