Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
lithos-metal is an open-source inference engine from lithos-ai that generates Metal megakernels for Apple silicon, aimed at low-latency local LLM inference and local serving for coding agents and other applications. It combines layer-wise mixer megakernels, GPU-resident generation control and DSpark speculative decoding. Its compiler decomposes operators into matrix tiles, attention partitions and recurrent-state slices, records their dependencies and generates specialized Metal code run by persistent threadgroup workers. Fusion is layer-wise rather than one whole-model megakernel: in selected configurations the Gated DeltaNet or full-attention mixer of a layer becomes a single megakernel while the feed-forward network stays separate, and megakernels and conventional kernels are linked through a pre-encoded Metal indirect command buffer. DSpark pairs a small target-conditioned block drafter with a sequential Markov head; by default it proposes seven tokens that the target verifies together with one anchor token, and --no-draft switches to target-only generation. The server exposes Chat Completions, Responses and Anthropic Messages adapters with tool calls and SSE streaming, and runs one generation at a time. It requires an Apple silicon Mac with macOS 26+, a registered chip backend and enough unified memory for the target, draft and context. The Qwen3.8-27B and Qwen3.6-35B-A3B configurations have been exercised on a 40-core M5 Max with 48 GB; the 35B hybrid MoE integration is experimental, and M3 Max, M4 Pro and the 32-core M5 Max use unmeasured native fallbacks without automatic megakernel fusion. For the Qwen3.8-27B-NVFP4 case study on that machine, a target-compute projection reaches approximately 168 tokens/s at 128-token context and 127 tokens/s at 32K context; these assume six accepted tokens per eight-row target pass, exclude drafting, acceptance processing and rewind, and are not end-to-end generation rates. The code is under Apache 2.0; the Qwen3.8-27B DSpark head on Hugging Face carries a separate 'other' weight license, while the 35B-A3B head is Apache-2.0.