WhisperX is an open-source automatic speech recognition (ASR) pipeline that wraps OpenAI's Whisper to fix two of its biggest practical shortcomings: imprecise timestamps and the lack of speaker labels. Released under a permissive BSD-2 license and now past 23,000 GitHub stars, it delivers accurate word-level timestamps, speaker diarization, and roughly 70x realtime transcription — turning Whisper from a strong-but-coarse transcriber into a production-ready pipeline for subtitles, meeting notes, and audio analytics. The project was introduced in a 2023 paper accepted at INTERSPEECH and took first place in the Ego4d transcription challenge. ## Word-Level Timestamps via Forced Alignment Whisper produces highly accurate transcriptions, but its timestamps are at the utterance level and can drift by several seconds — a problem for subtitling or any task that needs to know exactly when a word was spoken. WhisperX solves this with forced phoneme alignment: after Whisper transcribes the audio, a phoneme-based model such as wav2vec2.0 aligns the text back to the waveform to generate precise per-word timing. The result is subtitle-grade timestamps that Whisper alone cannot provide. ## 70x Realtime with Batched Inference OpenAI's Whisper does not natively support batching, which limits throughput. WhisperX layers voice-activity detection (VAD) preprocessing on top of the faster-whisper backend, cutting the audio into speech regions and batching them for inference. This yields up to 70x realtime transcription with large-v2 while requiring under 8GB of GPU memory at beam size 5. The VAD step also reduces hallucination — a known Whisper failure mode on silence — with no measurable degradation in word error rate. ## Speaker Diarization For multi-speaker audio, WhisperX integrates diarization from pyannote-audio to partition the stream by speaker and attach speaker-ID labels to each segment. Combined with sentence-level segmentation (via NLTK), this produces cleaner subtitles and transcripts that distinguish who said what — a requirement for meeting transcription, interviews, and podcasts. ## Trade-offs and Limitations WhisperX is a pipeline of several models, so setup is more involved than a single `pip install`: GPU acceleration expects a matching CUDA toolkit, and diarization requires accepting the pyannote model terms and supplying a Hugging Face token. Diarization accuracy still degrades with heavy speaker overlap or noisy audio, and the multi-stage design means more moving parts to keep compatible across versions. It also inherits Whisper's language-dependent quality, with best results on well-resourced languages. ## Who Should Use This WhisperX is aimed at developers and teams building subtitling tools, meeting and call transcription, media indexing, or any workflow that needs precise timing and speaker labels rather than a rough transcript. If you only need a quick, single-speaker transcription, plain Whisper or faster-whisper may be simpler — but when accuracy of timing and speaker attribution matters, WhisperX's BSD-2 license and battle-tested pipeline make it a strong open-source foundation.