Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
DreamX-Creator is a research framework from the AMAP-ML group for native joint audio-video generation, built around the idea that video and its accompanying audio should be modeled together rather than generated separately and stitched afterward. Given a first frame and a text prompt, its 7B base generator jointly models modality-specialized video and audio latent streams using what the authors call Gated Cross-Modal Attention and Progressive Joint Training, intended to let the two modalities inform each other bidirectionally during generation rather than have one conditioned on a finished version of the other. On top of that base generator, the accompanying technical report describes an Audio-Video Reinforcement Learning stage with Modality-Aware Multimodal Feedback, aimed at improving visual and audio quality, semantic consistency, and audio-video synchronization after the initial joint-modeling stage. A separate component, the Autoregressive 1-Step 2K Refiner (an SR-DiT model with roughly 5B parameters), takes a generated or externally supplied video and super-resolves it toward 2K output in a single autoregressive step while leaving audio and motion timing unchanged, and can run independently on an already-produced clip. The repository, opened on September 1, 2026, released inference code and pretrained weights for both the 7B joint generator and the 2K refiner on September 3, with checkpoints distributed through HuggingFace and ModelScope (built on Wan-AI's Wan2.2-TI2V-5B video VAE and UMT5-xxl text encoder as shared dependencies) rather than bundled in the git repository. Both components are documented as runnable on a single GPU, with the refiner separately supporting multi-GPU sequence-parallel inference and configurable KV-cache/window-attention options. The project's own roadmap marks initializing the repository, publishing the report, and releasing validated weights and inference code as complete, while a further round of distilled, lower-latency models is listed as not yet released. The work accompanies an arXiv technical report (2608.31106) and is distributed under the Apache License 2.0.