Overview

LiteRT-LM is Google AI Edge's official open-source inference framework for deploying Large Language Models directly on edge devices. Announced and open-sourced in April 2026, LiteRT-LM brings production-grade LLM inference to Android, iOS, Web browsers, desktop computers, and IoT devices like Raspberry Pi — completely eliminating the need for cloud round-trips. The framework already powers on-device AI in Google's own products including Chrome, Chromebook Plus, and Pixel Watch, demonstrating battle-tested production readiness at scale.

With 4,028 stars and active development (updated daily), LiteRT-LM is rapidly establishing itself as the definitive standard for on-device LLM inference.

Key Features

• Cross-Platform Deployment: Single framework targets Android (Kotlin/Java), iOS (Swift), Web (WASM), Desktop (Python/C++), and embedded Linux including Raspberry Pi
• Hardware Acceleration: Automatically leverages GPU and NPU processors on each target platform for optimal throughput and energy efficiency
• Multimodal Support: Handles text, vision, and audio inputs, enabling on-device multimodal AI applications
• Function Calling for Agents: Built-in support for structured output and function calling enables fully on-device agentic workflows without cloud dependency
• Broad Model Compatibility: Works with Gemma, Llama, Phi-4, Qwen, and architecturally similar open-weight models
• Constrained Decoding: Advanced output control ensures reliability in production agentic scenarios

Use Cases

LiteRT-LM is ideal for privacy-sensitive applications where data must not leave the device, offline-capable mobile and desktop AI assistants, low-latency interactive experiences that cannot tolerate cloud round-trip times, embedded systems and IoT deployments with limited or no connectivity, and developers building consumer apps who want to avoid per-query API costs.

Technical Details

The core is written in C++ for maximum performance, with stable language bindings for Kotlin (Android production), Python (prototyping and desktop), and direct C++ for native applications. The framework uses platform-native acceleration APIs: NNAPI and GPU on Android, Metal on iOS, and WebGPU in browsers. Model weights are quantized and optimized at load time.

Getting Started

# Install CLI via uv (no coding required)
uv tool install litert-lm

# Run a model immediately
litert-lm run --model gemma-3-1b-it

# Python API
pip install litert-lm
from litert_lm import LlmInference
model = LlmInference.create_from_model("gemma-3-1b-it-cpu.task")
print(model.generate_response("Hello, world!"))