Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
Building convincing voice AI has long required separate components for speech recognition, dialogue management, and speech synthesis stitched together with latency-introducing glue code. NVIDIA's PersonaPlex takes a fundamentally different approach: a single end-to-end speech-to-speech model that handles full-duplex conversation — speaking and listening simultaneously — with configurable persona control via text prompts and voice embeddings.
Released in April 2026 with MIT-licensed code and an NVIDIA Open Model License for weights, PersonaPlex represents NVIDIA's entry into the open-source conversational speech AI space. With 9.3k GitHub stars in its first weeks, the research community has taken notice.
PersonaPlex is a 7B parameter speech-to-speech model built on the Moshi conversational architecture, extended with NVIDIA's persona control system. Unlike traditional voice assistants that convert speech to text, process it with a language model, then synthesize speech back, PersonaPlex operates directly on audio streams, producing natural, low-latency responses without explicit intermediate text representation.
The system provides two complementary control mechanisms:
The combination produces a consistent persona with a consistent voice — something that has proven difficult to maintain in pipeline-based systems where the TTS voice is disconnected from the LLM's persona definition.
PersonaPlex supports genuinely simultaneous speaking and listening. Unlike voice assistants that use voice activity detection to alternate turns, full-duplex operation allows the system to generate a response while still processing incoming speech. This enables natural interruption handling, backchannel acknowledgments ("uh-huh", "I see"), and overlapping speech patterns that characterize human conversation.
Performance is evaluated against FullDuplexBench, which measures:
| Metric | What It Tests |
|---|---|
| User interruption handling | Graceful response when user cuts off AI |
| Pause handling | Natural behavior during silence |
| Backchannel generation | Appropriate acknowledgment responses |
| Turn-taking smoothness | Natural conversation flow |
PersonaPlex ships with 16 voice embeddings organized by style:
| Category | Female | Male |
|---|---|---|
| Natural (NAT) | NATF0-3 | NATM0-3 |
| Varied (VAR) | VARF0-4 | VARM0-4 |
Natural voices prioritize realistic, conversational quality. Varied voices offer distinct stylistic alternatives for different persona contexts. All voices work with any text-based persona prompt, decoupling voice character from personality definition.
PersonaPlex supports three deployment modes:
accelerate package for hardware-constrained environmentsThe underlying language model is NVIDIA's Helium, which provides robust generalization beyond the training distribution. The research team notes that Helium's breadth "benefits from underlying Helium LLM for handling out-of-distribution prompts" — meaning the model can handle conversation topics and persona types not explicitly seen during training.
Installation requires the Opus audio codec library as a system dependency, then pip installation of the bundled moshi package. GPU setup for Blackwell-generation NVIDIA GPUs requires specifying the CUDA 13.0 index URL. Hugging Face model weights require accepting the NVIDIA Open Model License and setting an HF token.
Once deployed, the web UI provides a browser-based interface for real-time conversation. The system handles audio I/O through WebRTC in the server mode, making it accessible from any browser without additional client software.
The offline evaluation mode is particularly useful for research applications — feeding pre-recorded audio and receiving generated audio output in return allows systematic evaluation of persona consistency and voice quality across conversation scenarios.
Response latency in server mode is competitive with commercial voice AI products for users with appropriate GPU hardware (tested on NVIDIA A100 and H100 configurations).
Pros
Cons
Full-duplex speech AI is one of the most technically challenging frontiers in conversational AI, and PersonaPlex establishes a credible open-source baseline. As demand for voice-native AI interfaces grows across customer service, accessibility tools, and interactive entertainment, the ability to define persona via text prompts — without model fine-tuning — makes PersonaPlex practically deployable for a wide range of applications.
NVIDIA's backing provides long-term credibility, and the association with the Moshi research lineage connects the project to an active academic community pushing the state of the art in end-to-end speech dialogue.
PersonaPlex addresses one of the hardest problems in voice AI: making a system that speaks and listens simultaneously while maintaining a coherent, configurable persona. For researchers working on conversational speech interfaces, developers building voice-native applications, and teams evaluating full-duplex dialogue systems, it is the most capable and accessible open-source option in this architectural class.