Claude Video, installed as the `/watch` command, gives an AI agent something it cannot do out of the box: actually watch a video. A coding agent can read a webpage, run a script, or browse a repository, but hand it a YouTube link and it is left guessing from the title or working off a transcript that misses most of what is on screen. Claude Video closes that gap by fetching captions, downloading only what it needs, extracting frames, and handing both the images and a timestamped transcript to the model — so by the time it answers, it has seen the video and heard the audio. Released under an MIT license, the project climbed the GitHub trending charts on the strength of a single, concrete capability, gaining several thousand stars in a week. ## Giving an Agent Eyes and Ears The workflow is deliberately simple. You paste a URL or a local file path, ask a question, and the tool does the rest: `/watch https://youtu.be/... what happens at the 30 second mark?` Behind that one line, `yt-dlp` handles the download (supporting YouTube, Loom, TikTok, X, Instagram, and hundreds of other sources, plus local `.mp4`, `.mov`, `.mkv`, and `.webm` files), `ffmpeg` extracts frames, and the agent reads each frame as an image alongside the transcript. The result is an answer grounded in what is actually on screen rather than in the video's description or title. People use it to break down a competitor's video hook, diagnose a bug from a screen recording, summarize a long talk faster than watching at 2x, or turn a playlist into per-video notes. ## Captions First, Frames on Demand A thoughtful part of the design is how it avoids unnecessary work and cost. At the lightest detail level, if a video already has captions, the tool returns the transcript without downloading the video at all. When audio is needed, it pulls native captions first — free, instant, and reasonably accurate — and only falls back to Whisper (Groq's `whisper-large-v3` or OpenAI's `whisper-1`) when no captions exist, extracting a small mono 16 kHz clip to keep the upload tiny. Frame extraction is tiered too: an `efficient` mode decodes only keyframes near-instantly, while `balanced` and `token-burner` modes prefer scene-change frames and fall back to uniform sampling when they under-produce. A deduplication pass drops near-identical frames — the dozen copies a static slide would otherwise generate — before they ever reach the model. ## The Frame-Budget Problem Because every frame is an image and image tokens add up quickly, token cost is dominated by how many frames you feed in. Claude Video builds its whole sampling strategy around that constraint. An auto-fps system scales the frame budget to a video's length — roughly 30 dense frames for a clip under 30 seconds, tapering to a capped 100 frames for anything over ten minutes, with a "sparse scan" warning when coverage thins out. When you name a moment ("around 2:30", "the last 30 seconds"), passing `--start` and `--end` switches to a focused window with a denser per-second budget. This budgeting is the difference between a usable tool and one that silently burns through a context window on a shallow pass over a long recording. ## Trade-offs and Limitations Claude Video is a wrapper, not a model: it orchestrates `yt-dlp`, `ffmpeg`, and an agent, and its usefulness is bounded by the model reading the frames. Frame-based understanding is inherently lossy — fast motion or fine on-screen detail can fall between sampled frames, and dense visual reasoning still costs real tokens. It depends on external tools that install on first run (via Homebrew on macOS, with printed commands on Linux and Windows), and the Whisper fallback needs an API key for videos without captions. As a young, fast-moving project it is also still maturing, so behavior and flags may shift between releases. ## Who Should Use This The tool is a strong fit for anyone who already works inside an agentic environment — Claude Code, Cursor, Codex, Gemini CLI, or any of the 50-plus Agent Skills hosts it supports — and wants to fold video into that workflow. Marketers analyzing creative, developers triaging bug recordings, and researchers turning long talks into notes get the most obvious value. If you need frame-perfect video analysis or work entirely outside an AI agent, this is not the right layer. But as a pragmatic way to give an agent working eyes and ears for video, Claude Video does one thing and does it well, under a permissive MIT license.