Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
MiDashengLM-Gen is a research framework from Xiaomi's MiLM Plus team for unified audio scene generation: instead of using separate models for speech, music, and sound effects, it produces one coherent audio clip blending all three from a single structured text description. The authors point to a limitation in existing unified audio generators, where a frozen, decoupled text encoder feeds a separate audio decoder, limiting cross-modal optimization and producing poor speech intelligibility. MiDashengLM-Gen instead couples a fully fine-tuned Qwen3-1.7B language model with per-token conditional flow matching, generating high-dimensional semantic-acoustic latents that a flow-matching DiT converts into 16 kHz mono audio through a multi-step Euler ODE solver with classifier-free guidance, while a learned stop head enables variable-length output instead of a fixed clip duration. Input is specified through structured multi-view captions: an overall scene caption, an ASR transcript for spoken content, a speech-style description of voice and emotion, plus separate sound-effects, music, and environment descriptions, each replaceable with an unknown token if unspecified. In the accompanying paper, the authors describe the model as a first approach to general text-to-audio generation trained end-to-end as a single model, and report it substantially narrows the intelligibility gap that has held back unified audio generators: on the Seed-TTS benchmark, English word error rate falls from 12.15% for prior unified models to 2.79%, approaching the 1.24% WER of dedicated TTS systems, extending to multilingual settings and, per the README, supporting nine languages with emotion control. The paper also reports competitive mixed-audio quality on the MECAT benchmark. The repository ships a Hugging Face-compatible checkpoint loadable through transformers AutoModel, plus a from-source path that auto-downloads the checkpoint from Zenodo for batch generation. The code is Apache License 2.0, though the README adds a use-restrictions clause barring unlawful, fraudulent, or military use, and the project targets researchers exploring mixed-audio synthesis rather than end users.