MiniMind: The Complete LLM Training Curriculum on a Single GPU

MiniMind is an educational and practical open-source project that delivers something remarkable: a complete, reproducible pipeline for training a GPT-class language model from absolute zero in approximately 2 hours on a single NVIDIA 3090 GPU, at a compute cost of roughly $3 USD. With 47,000+ GitHub stars, it has become one of the most-starred LLM education repositories available.

The Core Promise: Full LLM Lifecycle in One Codebase

Most tutorials cover one phase of LLM development — pretraining, fine-tuning, or alignment. MiniMind covers them all. The pipeline progresses through:

• Pretraining on a minimal 1.2GB dataset
• Supervised Fine-Tuning (SFT) with 1.6GB of instruction data
• Knowledge Distillation from larger teacher models
• LoRA adaptation for parameter-efficient fine-tuning
• Reinforcement Learning: DPO, PPO, GRPO, CISPO, and Agentic RL variants

This soup-to-nuts coverage means a developer can follow a single repository from random initialization to a model capable of multi-turn dialogue and tool calling.

Architecture Aligned with Modern Standards

Despite its small parameter count, MiniMind implements a proper modern Transformer decoder stack:

• Pre-normalization with RMSNorm for training stability
• SwiGLU activations matching the Llama/Qwen design philosophy
• RoPE positional encoding with YaRN extension for length generalization
• MoE variant: 198M parameters with 4 experts and top-1 routing
• Custom BPE tokenizer with 6,400 vocabulary tokens

The dense base model hits 64M parameters — small enough to train on consumer hardware but architecturally representative of production LLMs.

Native PyTorch First-Principles Approach

A key design choice distinguishes MiniMind from wrapper-heavy educational projects: core algorithms are implemented from scratch in native PyTorch rather than delegating to high-level abstractions. This transparency lets learners trace gradients, understand attention mechanisms, and debug training dynamics without fighting through library indirection.

Production-Compatible Output

Trained MiniMind models integrate with mainstream inference stacks: transformers, llama.cpp, vLLM, Ollama, and Llama-Factory. The models expose OpenAI-compatible API endpoints, support multi-turn dialogue with configurable chain-of-thought via <think> tags, and handle tool calling for function invocation — capabilities that mirror much larger production models.

Distributed Training Support

For teams wanting to scale beyond a single GPU, the framework includes DDP (DistributedDataParallel) and DeepSpeed integration, making it straightforward to move from a single 3090 to a multi-GPU cluster without refactoring the training code.

Benchmark Expectations

On C-Eval and CMMLU benchmarks, the 64M model achieves roughly 25% accuracy — expected performance for an ultrasmall model trained on limited data. The project explicitly frames this as a learning tool rather than a competitive model: the goal is understanding LLM internals, not state-of-the-art scores.

Who It's For

MiniMind is uniquely valuable for ML engineers who want hands-on understanding of LLM training from first principles, researchers exploring parameter scaling laws at small scale, and students who cannot afford cloud compute for larger training runs. The $3 training cost eliminates the financial barrier that has historically gatekept LLM development education.