Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
OmniVAE is a unified audio-video variational autoencoder released on GitHub by the OpenMOSS team alongside an arXiv paper (2607.23855) and a project page. It runs separate audio and video encoders to extract modality-specific features and aligns them in a shared latent space, so one tokenizer can encode and reconstruct audio alone, video alone, or the two jointly, and hand downstream diffusion transformers latent sequences that are already easier to align across modalities. The repository is organised as three installable packages: vae/ for VAE training, reconstruction inference and reconstruction metrics; generation/ for DiT training plus text-to-audio-video inference; and sync/ for probing how much audio-video synchronisation information survives in frozen OmniVAE features, by training a lightweight sync head that predicts audio-video offset. The code covers the full pipeline rather than inference only — the VAE side ships staged recipes for video and audio reconstruction, distillation, cross-modal alignment and audio-decoder fine-tuning, while the generation side trains in four stages from text-to-image through text-to-video and text-to-audio to text-to-audio-video. Weights live in a separate Hugging Face bundle (OpenMOSS-Team/OmniVAE) that every script resolves through an OMNIVAE_RELEASE_ROOT environment variable, deliberately keeping the Git repository lightweight; that bundle carries eight VAE checkpoint packages, four inference-only T2AV DiT packages, evaluation data and metric models, and the frozen Qwen3.5-0.8B-Base text encoder used during validation. The authors name t2av_recon_distill_avclip as their final T2AV configuration and describe every other released configuration as a baseline or ablation kept for comparison. Two caveats matter in practice: the README publishes no comparison table against external systems, offering instead a release launcher that reproduces VerseBench set3-large validation at CFG 4 and step 200000 and diffs the resulting metric JSONs against a reference directory when one is present; and no training data is bundled, with the documented workflows assuming multi-GPU torchrun jobs rather than a single consumer card.