Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
BigMoeOnEdge is a streaming inference layer built on top of stock llama.cpp's public API, not a fork, that lets Mixture-of-Experts (MoE) models several times larger than a device's RAM generate text on CPU-only hardware, with Android as the primary target and desktop as a secondary case. Instead of loading a model's full weight set, the engine hooks llama.cpp's evaluation callback: when a layer selects the experts a token routes to, it reads just those slices from flash storage at the moment they are needed, optionally caching hot experts in RAM and overlapping reads with compute. No llama.cpp source is changed for the core path; the one exception is an optional ~25-line hook needed only for the --overlap flag, kept on a separate fork branch since no public API exposes that wait point. The project states this default streaming path is lossless, byte-identical to running the same model fully resident, backed by ctest gates run in CI against a synthetic MoE model across caching, eviction, overlap, and multi-shard reads; separate speed knobs such as dropping cold experts, cutting active experts, expert substitution, and route-ahead are explicitly documented as lossy, and several prefetching/routing options are marked experimental. Architecture support (Qwen3/Qwen2 MoE, Gemma 4 MoE, gpt-oss, LFM2 MoE, DeepSeek V4 Flash 284B-A13B, Ling 3.0, and a Qwen4 preview) is added as one registry row rather than hardcoded per model. Per its own benchmark protocol -- one 12 GB RAM Android phone, adb shell, 256-token greedy decode, Q4_K_M -- the best streamed setup reaches 2.2 tok/s on gpt-oss-120b (~60 GB, vs. a 0.09 tok/s mmap baseline) and 5.0-5.2 tok/s on the 17-22 GB Qwen3.6-35B-A3B, Qwen3-30B-A3B, and Gemma-4-26B-A4B models; separately, outside that protocol, the README cites demo-app runs of 0.94 tok/s on DeepSeek V4 Flash 0731 (284B, ~91 GB) and 3.48 tok/s on the Qwen3.8-Flash-Next preview (125B, ~80 GB at Q2_K). The bundled Android app adds a model downloader and live telemetry to every tuning flag. The project credits AirLLM, Apple's "LLM in a Flash," FlexGen, and PowerInfer as prior art; its own code, confirmed from the LICENSE file, is Apache-2.0.