Emu3 is a suite of open multimodal models from the BAAI (Beijing Academy of Artificial Intelligence) Emu3 Team, built on a single, deceptively simple idea captured by its subtitle: Next-Token Prediction is All You Need. Released under the Apache-2.0 license with an accompanying paper (arXiv:2409.18869), Emu3 tokenizes images, text, and video into one shared discrete space and trains a single transformer from scratch on the resulting mixture of multimodal sequences — no diffusion module, no CLIP encoder, and no separately pretrained language model bolted on top.

One Transformer, One Objective

Modern multimodal systems are usually compositional: a diffusion model for image generation, a CLIP vision encoder feeding a pretrained LLM for understanding, and yet another architecture for video. Emu3 collapses all of that into a single autoregressive transformer trained with the same next-token prediction objective that powers text-only LLMs. A vision tokenizer converts pixels into discrete tokens, so an image or a video frame becomes just another sequence the model predicts one token at a time. This unification is the project's core contribution and the reason it has drawn attention from researchers looking to simplify multimodal stacks.

Generation and Perception in One Model

Emu3 is notable for being strong at both directions of the multimodal problem. On generation, it produces high-quality images by simply predicting the next vision token, naturally supporting flexible resolutions and styles, and it generates video causally by predicting the next token in a video sequence — and can extend an existing clip by continuing the sequence. On perception, it delivers coherent vision-language understanding and text responses without relying on a CLIP encoder or a pretrained LLM. BAAI reports that Emu3 surpasses well-established task-specific open models including SDXL for image generation, LLaVA-1.6 for understanding, and OpenSora-1.2 for video, all while removing the need for diffusion or compositional designs.

Released Models and Tooling

The project ships several checkpoints on the Hugging Face hub: Emu3-Gen for text-to-image generation, Emu3-Chat for vision-language understanding, an image-pretrained Emu3-Stage1, and the Emu3-VisionTokenizer that underpins the discrete image representation. Inference runs through standard Hugging Face Transformers — Emu3-Chat is supported in vLLM as Emu3ForConditionalGeneration for accelerated serving — and the repository includes supervised fine-tuning scripts so teams can adapt the base models to new image-generation or perception tasks.

Trade-offs and Limitations

The unified next-token approach is elegant but has practical costs. Autoregressive image and video generation predicts tokens sequentially, which can be slower at high resolutions than a well-optimized diffusion pipeline that denoises in parallel. Generation and understanding are exposed through separate post-trained checkpoints (Emu3-Gen and Emu3-Chat) rather than a single deployable model, and per the project's roadmap some pieces — full pretraining/DPO training scripts and evaluation code — were still being rolled out. As a research release, it targets practitioners comfortable with model weights and Transformers rather than a turnkey product.

Who Should Use This

Emu3 is compelling for researchers and engineers exploring unified multimodal architectures, anyone who wants a single Apache-2.0 backbone for both image/video generation and vision-language understanding, and teams interested in whether next-token prediction can replace diffusion- and CLIP-based pipelines. If your interest is a clean, permissively licensed testbed for token-based multimodal modeling rather than the absolute fastest production image generator, Emu3 is one of the most interesting open implementations of the idea.