Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
NInfer is a from-scratch C++/CUDA inference engine built for Qwen3.5-family Dense and MoE architectures on a single NVIDIA GeForce RTX 5090, serving text, image, and video prompts through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs; the runtime is scoped to one GPU, one resident model, and a startup-fixed capacity of one to eight active requests. The maintainer publishes five official v3 .ninfer artifacts on Hugging Face under the neroued account: Qwen3.6-27B and Qwen3.8-27B each in groupwise-int and nvfp4 formats, plus Qwen3.6-35B-A3B in groupwise-int; existing v2 downloads can be upgraded locally without re-downloading weights. Building from source requires 64-bit Linux, an RTX 5090, a CUDA toolkit supporting sm_120a (the build rejects other CUDA architectures), CMake 3.28+, and a C++20 compiler; there is no install target or packaged binary, so NInfer runs from its source build tree, though a Docker image can also be built. The README's example ninfer-serve configuration for Qwen3.8-27B NVFP4 sets a 240,000-token max-context and kv-capacity, max-concurrency of 2, fp8 KV cache, and MTP speculative decoding with 3 draft tokens. All performance figures are maintainer-published measurements on an RTX 5090 with INT8 group-64 KV and CUDA Graphs; in the saturated concurrent-decode benchmark (MTP3, one 8,192-token generation per active request), Qwen3.6-27B nvfp4 reaches 202.4 tok/s at concurrency 1 (69.3% acceptance) and 1,146.9 tok/s at concurrency 8 (68.6% acceptance, a 5.67x scaling factor). The Evaluation table reports scores for Qwen checkpoints served through NInfer's OpenAI-compatible route (thinking enabled, MTP3, EvalScope 1.9.0, 0-shot), not a measure of NInfer itself: Qwen3.8-27B NVFP4 scores 96.67% on AIME 2025 and 90.40% on GPQA-Diamond. The product boundary stays narrow: no multi-GPU or weight offload, no request preemption, one shared startup-fixed KV pool, and while NInfer parses and returns tool calls to the client it does not execute them. Published artifacts derive from Qwen/Qwen3.6-27B, Qwen/Qwen3.8-27B, and Qwen/Qwen3.6-35B-A3B, with the NVFP4 builds using fixed weights from rdtand and unsloth. NInfer is a personal project under the Apache License 2.0.