Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
FreeToken is an edge-native Mixture-of-Experts serving engine built to run frontier-scale open-weight models on personal and consumer hardware, treating GPUs, CPUs, host memory, and interconnects as one elastic inference platform rather than assuming everything fits in VRAM. Its runtime combines bandwidth-adaptive CPU-GPU co-execution (the paper's q-star policy), full-layer double-buffered prefill streaming, a global LRU expert cache, graph-compatible execution, and an FTW fast weight format. The knob that matters in practice is the MoE backend: fused keeps experts resident on the GPU, offload holds them in host RAM and streams cache misses over PCIe, cpu computes misses on the processor instead of fetching them, hybrid splits misses between the two paths with overlap, and auto picks between them using a per-machine profile collected by ft bench bw. Two further systems features target agent workloads specifically — semantic anchor checkpoints for recurrent state and KV caches, so context edits such as tool calls and thinking blocks avoid redundant recomputation, and runtime VRAM re-allocation between expert caches and KV memory without restarting the engine or reloading weights. Checkpoints load directly from Hugging Face safetensors (with native GGUF for Gemma-4) and the known-good list spans DeepSeek's DeepSeek-V4-Flash, Z.ai's GLM-5.3-Flash, GLM-5.2 and GLM-4.7, Alibaba's Qwen3.8-Flash-Next, Qwen3.6-35B-A3B and Qwen3-30B-A3B, OpenAI's gpt-oss-120b and gpt-oss-20b, Google's Gemma-4, MiniMax-M2.5, and Muse-Glimmer-30B, across MXFP4, NVFP4, FP8, and BF16. Anthropic- and OpenAI-compatible endpoints let coding agents such as Codex, Claude Code, OpenCode, OpenClaw, and DeepSeek Harness point at a local server, and the project ships both a Windows/Linux desktop app and a uv/pip CLI. It accompanies an arXiv paper (2608.16157), acknowledges mini-sglang as its inspiration and reuses design and code from SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM, and llama.cpp, and is licensed Apache-2.0. Multimodal checkpoints are served text-only.