Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
**MiniMind-O** is the third entry in Jingyao Gong's MiniMind series — after MiniMind (language) and MiniMind-V (vision) — and it does something the open-source ecosystem has largely lacked: it implements a **complete end-to-end Omni model from zero**, where a single set of weights accepts text, audio, and image input and emits both text and streaming speech. The flagship `minimind-3o` is roughly **0.1B parameters (115M)**, trainable on a consumer GPU and fast enough to run inference on CPU, making it plausibly the smallest complete Omni implementation publicly available. The pedagogical framing is the point. Plenty of capable Omni models exist — Mini-Omni2, Moshi, GLM-4-Voice, Qwen3-Omni — but they are large enough that reading, retraining, or modifying one end to end is out of reach for most people. MiniMind-O targets the person who wants to understand the architecture by building it. **All core algorithms are implemented directly in native PyTorch** with no reliance on third-party high-level abstractions, and the project ships two training datasets: a `mini` set that completes the full Thinker-Talker pipeline in about **2 hours on a single RTX 3090**, and a `full` set corresponding to the released weights. The architecture avoids the obvious shortcut. Cascading ASR → LLM → TTS is straightforward engineering, but round-tripping through text discards timing, tone, and emotion. MiniMind-O instead connects speech and text **at the hidden-state level**, using a **Thinker-Talker dual-path design** where the Talker predicts multi-layer Mimi codes via **Multi-Token Prediction (MTP)**. Combined with VAD, this supports 24 kHz streaming speech generation, real-time **barge-in interruption**, and approximate duplex interaction. Audio and vision features come from frozen SenseVoice-Small and SigLIP2 encoders projected into MiniMind's hidden space through two-layer MLPs. The release is unusually complete for an educational project: full structure code, SFT training across T2A / I2T / A2A data types, DDP multi-GPU support, five built-in voice prompts plus seven unseen ones, voice-cloning from arbitrary reference audio, and a WebUI with CLI inference, streaming playback, and a telephone mode. A technical report on arXiv covers architecture, training curves, CER/WER evaluation, and cross-model comparisons. An MoE variant, `minimind-3o-moe`, offers ~0.3B total with 0.1B active. Two caveats matter. At 115M parameters this is a **learning vehicle, not a production model** — output quality sits far below any frontier Omni system, and the project does not pretend otherwise. And the documentation, training data, and evaluation are **oriented toward Chinese**, with an English README available but the primary materials and speech tasks centered on Mandarin. Everything is **Apache-2.0**, with weights distributed through ModelScope and Hugging Face.
hacksider
Real-time AI face swap and one-click video deepfake with only a single image
harry0703
AI-powered short video generator that automates scripting, footage sourcing, subtitles, and composition — supporting 10+ LLM providers and batch production.