Xinference (Xorbits Inference) is an open-source model-serving framework that lets you run open-source large language, speech, embedding, reranking, and multimodal models behind a single, production-ready API — on your laptop, on-premise, or in the cloud. Its tagline, "Swap GPT for any LLM by changing a single line of code," captures the core promise: expose an OpenAI-compatible endpoint so existing applications can point at a locally hosted model without rewriting their integration. With more than 9,000 GitHub stars and active daily development from the Xorbits team, it has become a popular way to self-host models without gluing together a different serving stack for every model family. ## One API, Many Backends Rather than being a single inference engine, Xinference is an orchestration layer that sits on top of several. The same server can launch models through vLLM for high-throughput GPU serving, llama.cpp/GGML for quantized CPU and consumer-GPU inference, SGLang, MLX for Apple Silicon, or plain PyTorch/Transformers. Because the framework exposes an OpenAI-compatible REST interface (plus Python and command-line clients), the choice of backend becomes an implementation detail — you register a model and get a stable endpoint regardless of what runs underneath. ## Beyond Text Generation What distinguishes Xinference from many LLM-only serving tools is the breadth of model types it manages from one place. In addition to chat and completion LLMs, it serves embedding models, rerankers, image-generation models, and speech models such as Whisper — the building blocks of a retrieval-augmented-generation or multimodal pipeline. Teams can therefore host the LLM, the embedding model, and the reranker used by a RAG stack on a single control plane instead of operating three separate services. ## Deployment and Scaling Xinference is built to grow from a single machine to a cluster. A local install can serve models with one command for prototyping, while its distributed mode schedules models across multiple workers and GPUs for production traffic. A built-in web UI and model registry make it straightforward to launch, monitor, and swap models, and Docker images plus a managed enterprise edition exist for teams that want packaged deployment. The project is Apache-2.0 licensed. ## Trade-offs and Limitations Because Xinference wraps other engines, its performance and hardware support ultimately depend on the backend you select, and keeping the abstraction current with fast-moving engines like vLLM adds an extra layer to debug when something breaks. The convenience of a unified control plane also means operators must understand which backend a given model uses to tune memory, quantization, and throughput correctly. For a team standardized on a single engine, running that engine directly may be simpler than adopting the orchestration layer. ## Who Should Use This Xinference is a strong fit for developers and platform teams that need to self-host a mix of model types — LLMs, embeddings, rerankers, and speech — behind one OpenAI-compatible API, and want the freedom to switch engines or hardware without changing application code. Groups building on-premise RAG systems, or replacing hosted API calls with private models for cost or data-control reasons, get the most value; teams committed to a single engine on fixed hardware may not need the extra abstraction.