Emu3 is a suite of open multimodal models from the BAAI (Beijing Academy of Artificial Intelligence) Emu3 Team, built on a single, deceptively simple idea captured by its subtitle: *Next-Token Prediction is All You Need*. Released under the Apache-2.0 license with an accompanying paper (arXiv:2409.18869), Emu3 tokenizes images, text, and video into one shared discrete space and trains a single transformer from scratch on the resulting mixture of multimodal sequences — no diffusion module, no CLIP encoder, and no separately pretrained language model bolted on top. ## One Transformer, One Objective Modern multimodal systems are usually compositional: a diffusion model for image generation, a CLIP vision encoder feeding a pretrained LLM for understanding, and yet another architecture for video. Emu3 collapses all of that into a single autoregressive transformer trained with the same next-token prediction objective that powers text-only LLMs. A vision tokenizer converts pixels into discrete tokens, so an image or a video frame becomes just another sequence the model predicts one token at a time. This unification is the project's core contribution and the reason it has drawn attention from researchers looking to simplify multimodal stacks. ## Generation and Perception in One Model Emu3 is notable for being strong at both directions of the multimodal problem. On generation, it produces high-quality images by simply predicting the next vision token, naturally supporting flexible resolutions and styles, and it generates video causally by predicting the next token in a video sequence — and can extend an existing clip by continuing the sequence. On perception, it delivers coherent vision-language understanding and text responses without relying on a CLIP encoder or a pretrained LLM. BAAI reports that Emu3 surpasses well-established task-specific open models including SDXL for image generation, LLaVA-1.6 for understanding, and OpenSora-1.2 for video, all while removing the need for diffusion or compositional designs. ## Released Models and Tooling The project ships several checkpoints on the Hugging Face hub: Emu3-Gen for text-to-image generation, Emu3-Chat for vision-language understanding, an image-pretrained Emu3-Stage1, and the Emu3-VisionTokenizer that underpins the discrete image representation. Inference runs through standard Hugging Face Transformers — Emu3-Chat is supported in vLLM as Emu3ForConditionalGeneration for accelerated serving — and the repository includes supervised fine-tuning scripts so teams can adapt the base models to new image-generation or perception tasks. ## Trade-offs and Limitations The unified next-token approach is elegant but has practical costs. Autoregressive image and video generation predicts tokens sequentially, which can be slower at high resolutions than a well-optimized diffusion pipeline that denoises in parallel. Generation and understanding are exposed through separate post-trained checkpoints (Emu3-Gen and Emu3-Chat) rather than a single deployable model, and per the project's roadmap some pieces — full pretraining/DPO training scripts and evaluation code — were still being rolled out. As a research release, it targets practitioners comfortable with model weights and Transformers rather than a turnkey product. ## Who Should Use This Emu3 is compelling for researchers and engineers exploring unified multimodal architectures, anyone who wants a single Apache-2.0 backbone for both image/video generation and vision-language understanding, and teams interested in whether next-token prediction can replace diffusion- and CLIP-based pipelines. If your interest is a clean, permissively licensed testbed for token-based multimodal modeling rather than the absolute fastest production image generator, Emu3 is one of the most interesting open implementations of the idea.