Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
AliceAI-Foundation-80B-A3B-Base is a base language model that Yandex trained entirely from scratch and released under the Apache-2.0 license, alongside two new Russian-focused factual benchmark datasets and their evaluation protocols, WikiWebFacts and HardMultiQA. The model uses a hybrid architecture with 80 billion total parameters and 3 billion activated per token, a 262,144-token context window, 48 layers arranged as 12 repeats of three KDA-linear-attention-then-MoE blocks followed by one gated-attention-then-MoE block, and MoE layers that route each token to 10 of 512 experts plus one shared expert; Yandex validated the architecture and hyperparameters through separate from-scratch training runs of 2 trillion tokens each before the full pretraining run. On Yandex's own internal evaluation infrastructure (vLLM inference, matched settings across models), the model card reports AliceAI-Foundation-80B-A3B-Base scoring 86.5 on the Russian-language WikiWebFacts benchmark versus 62.4 for Qwen3.5-35B-A3B-Base, 70.2 for GLM-4.5-Air-Base (106B-A12B), 72.8 for Nemotron-3-Super-120B-A12B-Base, and 83.2 for DeepSeek-V4-Flash-Base (284B-A13B); 67.9 on HardMultiQA versus those same models' 47.2/48.6/54.5/65.4; and 74.2 on the Russian EduBench education benchmark, ahead of all four comparison models. On English-language benchmarks the results are more mixed: 79.0 on TriviaQA, behind GLM-4.5-Air-Base (83.5), DeepSeek-V4-Flash-Base (89.4), and Nemotron (89.8), and 66.8 on MMLU-Pro CoT, behind Nemotron's 69.9. In a separate reasoning comparison against just Qwen3.5-35B-A3B-Base and Nemotron-3-Super-120B-A12B-Base, it scores 96.9 on HMMT 2026 February pass@32 and 88.7 on IMO Answerbench pass@8, both ahead of both comparison models, while trailing Qwen's 73.7 on CodeForces C++ pass@8. As a base model, it ships without a default conversational chat template; a separate Jinja template for agentic fine-tuning data, using the OpenAI Messages format with system/user/assistant/tool/meta roles, is provided for SFT and RL data preparation instead. The model card documents serving through Transformers 5.16.1 with the flash-linear-attention library for its KDA layers, and through vLLM via a published Docker image with tensor-parallel and multi-token-prediction speculative decoding, plus a LoRA fine-tuning example built on PEFT and FSDP2.