Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
ControlFoley is a research framework from MiLM Plus at Xiaomi Inc., developed together with Wuhan University, for unified and controllable video-to-audio (V2A) generation. Rather than treating video, text, and reference-audio conditioning as separate pipelines, the model folds them into a single architecture built to keep working even when the inputs disagree, for example when a text prompt describes a sound that conflicts with what is visible on screen. The design pairs CLIP with CAV-MAE-ST visual encodings to capture both vision-language and audio-visual correlations, extracts a timbre-only representation from reference audio so acoustic style can be transferred without disturbing the video's temporal sync, and trains with an all-modality dropout scheme plus a unified REPA objective so the model degrades gracefully across different combinations of available inputs. The repository ships five inference modes -- TV2A, TC-V2A, AC-V2A, plain V2A, and T2A -- runnable through a single demo.py script, and the authors also release a new evaluation set, VGGSound-TVC, for measuring text controllability under graded levels of visual-text conflict. The paper, accepted to ACM MM 2026, reports state-of-the-art results on VGGSound-Test, Kling-Audio-Eval, and MovieGen-Audio-Bench for the TV2A task, including up to a 27% relative Inception Score improvement (22.08 vs. 17.36) on VGGSound, and the authors report topping the Greatest Hits and AC-VAS datasets for audio-controlled generation, outperforming the specialized CondFoleyGen baseline and, per the README, competitive with the proprietary Kling-Foley system. Beyond the core code, the project has an active downstream ecosystem: an official ComfyUI node pack and workflows covering all five tasks, an interactive Hugging Face Space, and third-party native C++/GGUF inference support added in audio.cpp v0.7. Licensing is split: the inference code in this GitHub repository is Apache-2.0, but the pretrained weights hosted on Hugging Face (YJX-Xiaomi/ControlFoley) are released under CC BY-NC 4.0, a non-commercial license, so deploying the released weights commercially is not permitted without a separate arrangement.