VoiceMem is an open-source memory system for real-time voice agents, built on a streaming dual-brain architecture that splits memory into two cooperating stores: a Left Brain that organizes factual information by schema and entity for accurate retrieval while sustaining Mem0's full performance under a top-3 memory limit, and a Right Brain that tracks personality, emotion and relationship context through independent and cross-entity nodes jointly maintained alongside the Left Brain's facts. The pipeline runs while the user is still speaking: audio is segmented, transcribed and mined for memories on the fly and written into a memory graph through information compression, hierarchical storage and streaming retrieval with 0-300 ms speculative prefetching; at query time VoiceMem routes candidates first, ranks second, and injects only the top-K matches into the reply model's context, around 300 tokens per query. The project reports 91.2% accuracy on LoCoMo against 61.68% for Mem0 using only the top 5 memories, 69.44% on PersonaMem, a 134 ms response versus Mem0's 1,440 ms, and roughly 430 tokens per query against 6,956 for Mem0 and 1,899 for EverMemOS. The Python package (pip install voicemem) bundles its own ASR, speaker identification, scene and emotion detection, and local embedding models, and exposes an offline call-and-search interface plus an async streaming interface built around VAD turn state, so retrieval can start on partial speech before a user finishes talking. A bundled interactive demo (web/run.py) serves a local voice chat UI with per-session memory isolation and two-stage barge-in handling that pauses playback on speech and only cancels once a stop command or stable transcript is confirmed. It also publishes a companion model family -- fine-tunes of Qwen2.5-Omni, Qwen3-Omni and Step-Audio2-Mini able to read VoiceMem's memory output, trained on the released ChatMem-400K dataset -- plus finetuning code and an evaluation harness for LoCoMo-style benchmarks. It builds on mem0 for vector memory, FunASR/sherpa-onnx for streaming ASR and speaker verification, and multilingual-e5 for embeddings, while OpenAI's API handles chat, TTS and realtime calls.