E2E fixture copy2 20261112T233300Z

Memory framework

First committed by xzf-thu

Published in xzf-thu/agbr-e2e-20261112t233300z-copy2. Publisher page

0 GitHub starsUpdated 2026-09-14

What it is

agbr-e2e-20261112T233300Z planted copy2 fixture: a copy of ljxqhqkpoxp2 for the R2 staging checks. Removed at teardown.

— the description by xzf-thu

Summary from the brain’s metadata

Infrastructure for the next generation of voice agents, designed to provide universal memory. It is divided into a left brain and a right brain, storing information and emotions respectively, while a fully streaming architecture eliminates latency at the fundamental level. 我们带来 VoiceMem，为语音模型增加最后一个组件：灵魂，让它真正越来越懂你。VoiceMem 建立在「流式双脑」架构之上，提供精准、有情感、懂人格、低延迟且最便宜的记忆服务。本仓库将「永久保持全部开源」。

How it works

Memory model: Vector store

AutoGineer overview, from the brain’s own files

Memory model: Vector store. Folders: assets (attachments, 12 files); docs (docs, 17 files); examples (examples, 8 files); scripts (scripts, 1 file); tests (tests, 2 files); tools (scripts, 1 file).

Folders
Folder	Role	Files
assets	attachments	12
docs	docs	17
examples	examples	8
scripts	scripts	1
tests	tests	2
tools	scripts	1
evaluation	folder	7
finetune	folder	6
voicemem	folder	106
voicemem_memoryspace	folder	3
web	folder	9

Requirements

Runtimes: Python >=3.10

API keys (by name): OPENAI_API_KEY

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Google Antigravity, Kiro, OpenCode, Amp, Roo Code, Cline, JetBrains Junie, Devin (ex-Windsurf)

Set it up

Not scanned by AutoGineerThe author’s README also describes installing it from a clone of its repository. Read its install section: AutoGineer did not scan that path. https://github.com/xzf-thu/VoiceMem/blob/6cacb3c1e7fca2679f3bf95451f6b480efd9a783/README.md#installation
First committed by xzf-thu. Published in xzf-thu/agbr-e2e-20261112t233300z-copy2. Licence: Apache License 2.0.

Usage

AutoGineer’s guidance for this type of brain

Install the library in your own agent code and call it to store and retrieve memories; see its documentation for the API.

Documentation by xzf-thu

中文 | <a href="#english">English</a>

<a href="https://xzf-thu.github.io/VoiceMem/">项目主页 🌐</a> / <a href="https://arxiv.org/pdf/2608.26005">技术报告 📖</a> / <a href="https://huggingface.co/zhifeixie/VoiceMem_Default_Models_Env">VoiceMem Utils 🤗</a> / <a href="https://huggingface.co/zhifeixie/VoiceMem_MF_Qwen3_6_35B_A3B_Qlora">VoiceMem Model Families 🤗</a> / <a href="https://huggingface.co/datasets/zhifeixie/VoiceMem-ChatMem400k">ChatMem-400K 🤗</a>

我们带来 VoiceMem，为语音模型增加最后一个组件：灵魂，让它真正越来越懂你。VoiceMem 建立在「流式双脑」架构之上，提供精准、有情感、懂人格、低延迟且最便宜的记忆服务。本仓库将「永久保持全部开源」。

快速理解 VoiceMem：

左脑： 直接管理信息，在 Top-3 限制下维持 Mem0 的满载性能。
右脑： 用长短期情绪归因管理「情商」，含交叉节点、与左脑信息联合维护。
低延迟： 通过压缩信息、分层存储、流式查询（0–300 ms 投机预取），几乎不增加延迟。
简单实用： 单轮查询约 300 token；架构全部解耦，全部组件（含底层记忆引擎）都可更换。

🔥 News

💬 09/01/2026 · v0.0.2 — 修复事件日期链路，移除右脑冗余记忆类别，开放可插拔语音合成层。
🎉 08/27/2026 · v0.0.1 — 发布初代 VoiceMem 和 Technical Report。
🤖 08/21/2026 — 开源 VoiceMem 模型系列，可直接读取并理解 VoiceMem 提供的记忆。
🛠️ 08/21/2026 — 发布 VoiceMem Utils，开箱即用。
📦 08/20/2026 — 开源 ChatMem-400K 数据集。

🎬 Demo

注意： 播放前需要先取消静音。

https://github.com/user-attachments/assets/0d919f8c-e9ba-4fdb-8078-b049e4b99a28

📚 目录

🚀 快速开始
🧠 VoiceMem 双脑流式架构
🤖 VoiceMem 官方记忆模型
🔌 使用 VoiceMem 定制你的语音智能体
🛠️ 模型微调
📊 评测代码
📖 引用
致谢
许可证

🚀 快速开始

安装

git clone https://github.com/xzf-thu/VoiceMem.git
cd VoiceMem

# 安装记忆系统（含 ASR / 声纹 / 场景 / 情绪 / 本地 embedding 全套内置组件）
pip install voicemem

# 可选：用我们微调的 Qwen 回复模型
pip install "voicemem[slm]"

下载所需模型

pip install -U huggingface_hub

hf download zhifeixie/VoiceMem_Default_Models_Env --local-dir ./models

基础用法 <a id="interfaces"></a>

作为离线记忆引擎运行

from voicemem import VoiceMem

vm = VoiceMem(
    mode="normal",
    openai_key="api_xxx",
    top_k=5,
)

# 本地模型是懒加载的，先热起来，别让第一次调用去等加载
vm.warmup()

# 存：音频文件
# 内部跑 ASR / 声纹 / 场景 / 情绪感知 / Embedding 抽取
print("入库开始")
vm.ingest(audio="assets/input.wav")  # 我是素食主义者，对坚果过敏。
print("入库结束")

# 查：写入慢是因为要抽事实、打标签、建图；查询走的是纯向量检索，跟写入无关
print("检索开始")
result = vm.search("我的饮食禁忌是什么？")
print("检索结束")

print(result.result_leftbrain, result.result_rightbrain)


# 存：左脑信息文本（无情感）
vm = VoiceMem(
    mode="leftbrain_only",
    openai_key="api_xxx",
    top_k=5,
)

vm.ingest("我是素食主义者，对坚果过敏。")

result = vm.search("我的饮食禁忌是什么？")

以流式方式运行 VoiceMem

可以把 VoiceMem 的流式接口看作一个持续处理音频的 VAD 接口。

下面这段：先显式存一条事实，再喂一段问句音频，看记忆是怎么在人还没说完时就查好的；最后照例走一次入库判断。

import asyncio
import os
from pprint import pprint

import numpy as np
import soundfile as sf

from voicemem import VoiceMem

# 沿用上面那个 vm；单独跑这段就自己建一个
vm = VoiceMem(mode="normal", openai_key=os.environ["OPENAI_API_KEY"], top_k=5)

# 本地模型是懒加载的，先热起来，别让第一块音频去等模型加载
vm.warmup()

# 先存一条事实，等下那个问句才有东西可查
vm.ingest("我是素食主义者，对坚果过敏。")

SPEC_MIN_CHARS = 6          
searching = False


def on_partial(text):
    """边说边出字。够长了就说明后台这一刻已经开查了。"""
    global searching
    print(f"\r[partial] {text}", end="", flush=True)
    if not searching and len(text) >= SPEC_MIN_CHARS:
        searching = True
        print("\n[检索开始] 人还没说完，后台已经在查了", flush=True)


async def main():
    # 这段音频里是一个问句：「我的饮食禁忌是什么？」
    audio, sr = sf.read("assets/question.wav", dtype="float32")
    pcm = (np.clip(audio, -1, 1) * 32767).astype(np.int16)

    stream = vm.stream(src_rate=sr, vad_threshold=0.5, on_partial=on_partial)
    step = int(sr * .032)

    for i in range(0, len(pcm), step):
        st = await stream.feed(pcm[i:i + step].tobytes())
        if st.state != "turn_over":                
            continue

        # VAD 确认这一轮说完了。记忆早在说话过程中就查好了，这里直接取，不再等
        print("[检索结束]")
        print("转写  ", st.transcript)
        print("左脑  ", st.result_leftbrain)         
        print("右脑  ", st.result_rightbrain)
        pprint({k: getattr(st, k) for k in
                ["speaker_id", "speaker_voiceprint", "emotion",
                 "entity", "schema", "text_embedding"]})

        # 每一轮都要走一次入库判断
        print("[入库] LLM 正在判断这句话值不值得入库…", flush=True)
        res = vm.ingest(st.transcript)
        print(f"[入库] 抽出 {res['facts_count']} 条事实 -> {res['memory_ids']}")
       


asyncio.run(main())

VoiceMem 交互式演示

演示代码在仓库里（pip 装的包只有库本身），先确认已经克隆并进入仓库目录。

python web/run.py

然后访问：

http://localhost:8787

Demo 默认把终端输出（含 Python logging 和 Uvicorn 的日志）保存一份到 results/logs/voicemem-时间-PID.log，每行带时间戳和 stdout/stderr 标记。启动时终端会打印实际路径。指定文件或临时关闭如下。

python web/run.py --log-file results/logs/debug.log
python web/run.py --no-file-log

回复模型的上下文由当前输入、本次会话尚未入库的对话和检索记忆组成。每轮对话先进入内存 SessionBuffer；异步记忆写入完成并确认产生持久记忆后，对应 turn 从 SessionBuffer 移除。没有产生长期记忆的临时对话会保留到本次会话结束，不同 Memory Space 和不同 WebSocket 会话互相隔离。

播放期间的插话使用两阶段控制：VAD 首先暂停并保留音频队列；明确停止指令或稳定 ASR 文本确认后才清空队列并取消回复；附和、回声、无文字声音和单音节碎片会恢复播放。候选静音回退和最长等待时间可分别通过 BARGE_REJECT_SILENCE_MS、 BARGE_CANDIDATE_TIMEOUT_MS 调整。

两种回复模式共用以 PCM 样本位置为基准的输出时间轴。浏览器 AudioWorklet 回报实际渲染进度，打断时只把已经播放的回复写入 SessionBuffer。TTS 后端可选返回 TimedAudioChunk 提供文字对齐；普通 PCM 后端按分段音频长度和动态语速估算。

🧠 VoiceMem：基于流式双脑架构的记忆系统

VoiceMem 是一个面向实时语音智能体的记忆系统。

VoiceMem 不把所有记忆放进同一个检索数据库，而是将记忆拆分成两个互相配合的部分：

左脑通过 Schema 和 Entity 组织事实记忆，用于更加准确地检索信息。
右脑通过独立节点和跨实体节点管理人格、情绪和关系信息。

整个流程都是流式的。

在用户仍然说话时，VoiceMem 会持续完成音频分段、语音转写、记忆提取，并把结构化信息写入记忆图中。

查询时，VoiceMem 会先路由，再排序，最后只把 Top-K 条记忆注入模型上下文，从而在保留相关信息的同时控制上下文长度。

主要特性

🎯 精准 — 在 LoCoMo 上达到 91.2%，Mem0 为 61.68%，并且只需要 Top-5 条记忆。
❤️ 有情感、懂人格 — 不只记住用户说过什么，还会记住用户是谁、用户有什么感受。在 PersonaMem 上达到 69.44%。
🎧 多模态 — 可以从真实世界音频中记住语音、说话人、声音事件、多人对话和音乐。
⚡ 低延迟 — 响应时间为 134 ms，Mem0 为 1,440 ms，并支持在语音轮次内部进行流式检索。
💰 低 Token 消耗 — 每次只使用 430 个记忆 token，Mem0 为 6,956，EverMemOS 为 1,899。

🤖 VoiceMem 模型系列

我们通过三阶段 OPD 训练流程构建 ChatMem-400K：

Memory-world construction
SLM-validated online on-policy distillation（OPD）
Human refinement

同一套流程在人工编辑后形成 ChatMem-Bench，评测语音模型是否能够在长期沉淀中形成对用户的理解。

VoiceMem 家族开源模型包括 Qwen2.5-Omni、Qwen3-Omni 和 Step-Audio2-Mini。这些模型可以在对话时接受并理解 VoiceMem 提供的记忆信息。

🔌 使用 VoiceMem 定制你的语音智能体

你可以将 VoiceMem 接入自己的语音模型，用于构建带有长期记忆能力的实时语音智能体。

整体流程如下：

麦克风 → VoiceMem 监听语音并提前检索相关记忆 → 你的模型读取这些记忆并生成回答

export OPENAI_API_KEY=sk-...
# 仅在写入记忆时用于事实信息提取。
# 记忆检索完全在本地运行。

python examples/03_simple_agent_with_voicemem_memory.py

换成你自己的模型：把生成那一步换掉就行，记忆那半边一行都不用动。

def my_reply(text, memory_context):        # 同步函数也可以，会自动丢线程
    return my_model.generate(system=memory_context, user=text)

vm = VoiceMem(reply=my_reply)

🛠️ 模型微调

VoiceMem 提供完整的微调代码，可用于训练自己的 VoiceMem Model Family Adapter。

默认训练配置与发布的 checkpoint-3318 使用的配置一致。

使用默认参数运行下面的命令，可以复现相同的 Adapter：

pip install ms-swift==4.5.2 bitsandbytes

python finetune/train.py --data data/train.jsonl

训练数据格式、GPU 显存要求，以及如何更换基础模型，请参阅 finetune/README.md。

📊 评测

评测流程完全开源，并且可以复现。

运行评测

只需要一条命令即可运行 Benchmark：

export OPENAI_API_KEY=sk-...

# 建议先运行仓库中自带的小型示例，
# 确认环境和配置没有问题。
# 2 个对话，5 个问题。
python evaluation/run.py \
    --dataset locomo \
    --data evaluation/examples/locomo_sample.json

# 然后运行完整数据集。
python evaluation/run.py \
    --dataset locomo \
    --data data/locomo.json

示例结果：

LoCoMo: 10 conversations · 152 questions

Score: 139/152 = 91.4%

  multi_hop     88.2%
  temporal      85.7%
  single_hop    95.1%

Median retrieval latency: 12 ms
Median retrieved memory: 298 tokens

在运行完整评测之前，可以加入 --inspect，检查数据集是否被正确解析。

这个模式不会调用模型，因此也不会产生 API 费用：

python evaluation/run.py \
    --dataset locomo \
    --data data/locomo.json \
    --inspect

评测过程中，回答模型只会收到检索得到的记忆，不会收到原始对话历史。

如果直接把完整对话交给模型，Benchmark 测试的就会变成模型的阅读理解能力，而不是记忆系统本身的能力。

完整评测流程，以及添加新 Benchmark 的方法，请参阅 evaluation/README.md。添加一个新的 Benchmark 只需要增加一个文件并实现两个函数。

📖 引用

如果 VoiceMem 对你的研究有帮助，请引用我们的论文：

@misc{2608.26005,
  author = {Zhifei Xie and Jiaqi Lang and Ze An and Yifan Zhao and Dongchao Yang and Kai Li and Ziyang Ma and Mingbao Lin and Chunyan Miao and Shuicheng Yan},
  title = {{V}oice{M}em: {S}treaming {D}ual-{B}rain {M}emory for {R}eal-{T}ime {I}nteraction},
  year = {2026},
  eprint = {2608.26005},
  note = {arXiv:2608.26005v1}
}

致谢

我们感谢以下优秀的开源项目：

mem0 — 向量记忆引擎
FunASR — 基于 paraformer-zh-streaming 的流式 ASR
sherpa-onnx — Silero VAD、3D-Speaker 说话人验证，以及备用流式 ASR
intfloat/multilingual-e5 — 本地 Embedding 和 Slot 分类

VoiceMem 同时使用 OpenAI API 提供 Chat、TTS 和 Realtime 功能。

许可证

VoiceMem 基于 Apache License 2.0 开源。

详细信息请参阅 LICENSE。

<a href="#chinese">中文</a> | English

<a href="https://xzf-thu.github.io/VoiceMem/">Project Page 🌐</a> / <a href="https://arxiv.org/pdf/2608.26005">Technical Report 📖</a> / <a href="https://huggingface.co/zhifeixie/VoiceMem_Default_Models_Env">VoiceMem Utils 🤗</a> / <a href="https://huggingface.co/zhifeixie/VoiceMem_MF_Qwen3_6_35B_A3B_Qlora">VoiceMem Model Families 🤗</a> / <a href="https://huggingface.co/datasets/zhifeixie/VoiceMem-ChatMem400k">ChatMem-400K 🤗</a>

We introduce VoiceMem, adding the final component to voice models: a soul, so they truly come to understand you better over time. VoiceMem is built on a streaming dual-brain architecture and provides accurate, emotional, personality-aware, low-latency, and lowest-cost memory services. This repository will remain fully open source, permanently.

A quick overview of VoiceMem:

Left Brain: Directly manages factual information and sustains Mem0's full performance under a Top-3 memory limit.
Right Brain: Manages emotional intelligence through short-term and long-term emotional attribution, including cross-entity nodes and joint maintenance with Left Brain information.
Low Latency: Uses information compression, hierarchical storage, and streaming retrieval with 0–300 ms speculative prefetching, adding almost no extra latency.
Simple and Practical: Each query uses about 300 tokens. The architecture is fully decoupled, and every component, including the underlying memory engine, can be replaced.

🔥 News

💬 09/01/2026 · v0.0.2 — Fixed the memory event-date path, removed a redundant right-brain memory class, and opened up the speech synthesis layer.
🎉 08/27/2026 · v0.0.1 — Released the first version of VoiceMem and our Technical Report.
🤖 08/21/2026 — Open-sourced the VoiceMem model family (Qwen2.5-Omni / Qwen3-Omni / Step-Audio2-Mini), able to read and use the memory VoiceMem provides.
🛠️ 08/21/2026 — Released VoiceMem Utils, all default local models packaged for out-of-the-box use.
📦 08/20/2026 — Open-sourced ChatMem-400K, built with a three-stage OPD pipeline.

🎬 Demo Video

Note: Please unmute the video before playback. https://github.com/user-attachments/assets/0d919f8c-e9ba-4fdb-8078-b049e4b99a28

📚 Overview

🚀 Quick Start
🧠 VoiceMem Dual-Brain Streaming Architecture
🤖 VoiceMem Model Families
🔌 Customize Your Voice Agent with VoiceMem
🛠️ Finetuning
📊 Evaluation
📖 Citation
Acknowledgements
License

🚀 Quick Start

Installation

Prerequisite: Python 3.10+

git clone https://github.com/xzf-thu/VoiceMem.git
cd VoiceMem

# Install the memory system (bundles ASR / speaker ID / scene / emotion / local embedding)
pip install voicemem

# Optional: run our fine-tuned Qwen reply model
pip install "voicemem[slm]"

Required Model Download

pip install -U huggingface_hub

hf download zhifeixie/VoiceMem_Default_Models_Env --local-dir ./models

Basic Usage <a id="interfaces-en"></a>

Run as an Offline Memory Engine

from voicemem import VoiceMem

vm = VoiceMem(
    mode="normal",
    openai_key="api_xxx",
    top_k=5,
)

# Local models load lazily -- warm them up so the first call doesn't pay for it.
vm.warmup()

# Store an audio file.
# VoiceMem internally runs ASR / speaker ID / scene / emotion / embedding extraction.
print("ingest start")
vm.ingest(audio="assets/input.wav")  # I am vegetarian and allergic to nuts.
print("ingest done")

# Writing is slow because it extracts facts, tags them and builds the graph.
# Reading is a pure vector lookup -- independent of write cost.
print("search start")
result = vm.search("What are my dietary restrictions?")
print("search done")

print(result.result_leftbrain, result.result_rightbrain)


# Store Left Brain factual text directly (no emotional information).
vm = VoiceMem(
    mode="leftbrain_only",
    openai_key="api_xxx",
    top_k=5,
)

vm.ingest("I am vegetarian and allergic to nuts.")

result = vm.search("What are my dietary restrictions?")

Run VoiceMem in Streaming Mode

Think of VoiceMem's streaming interface as a VAD interface that continuously processes audio.

The example below stores one fact explicitly, then feeds a question as audio to show how the memory is already retrieved before the speaker finishes. It ends, as always, with the ingest decision.

import asyncio
import os
from pprint import pprint

import numpy as np
import soundfile as sf

from voicemem import VoiceMem

# Reuses the vm above; building one here so the block runs standalone
vm = VoiceMem(mode="normal", openai_key=os.environ["OPENAI_API_KEY"], top_k=5)

# Local models load lazily -- warm them up so the first audio chunk doesn't wait
vm.warmup()

# Store one fact first, so the question below has something to find
vm.ingest("I am vegetarian and allergic to nuts.")

SPEC_MIN_CHARS = 6
searching = False


def on_partial(text):
    """Partial transcripts as they arrive. Long enough = the search already started."""
    global searching
    print(f"\r[partial] {text}", end="", flush=True)
    if not searching and len(text) >= SPEC_MIN_CHARS:
        searching = True
        print("\n[search start] speaker isn't done yet, retrieval already running", flush=True)


async def main():
    # This audio is a question: "What are my dietary restrictions?"
    audio, sr = sf.read("assets/question.wav", dtype="float32")
    pcm = (np.clip(audio, -1, 1) * 32767).astype(np.int16)

    stream = vm.stream(src_rate=sr, vad_threshold=0.5, on_partial=on_partial)
    step = int(sr * .032)

    for i in range(0, len(pcm), step):
        st = await stream.feed(pcm[i:i + step].tobytes())
        if st.state != "turn_over":
            continue

        # VAD confirmed end of turn. Memory was fetched while the user spoke -- just read it
        print("[search end]")
        print("transcript  ", st.transcript)
        print("left brain  ", st.result_leftbrain)
        print("right brain ", st.result_rightbrain)
        pprint({k: getattr(st, k) for k in
                ["speaker_id", "speaker_voiceprint", "emotion",
                 "entity", "schema", "text_embedding"]})

        # Every turn runs the ingest decision
        print("[ingest] LLM deciding whether this is worth storing...", flush=True)
        res = vm.ingest(st.transcript)
        print(f"[ingest] extracted {res['facts_count']} facts -> {res['memory_ids']}")


asyncio.run(main())

Interactive Demo with VoiceMem

The demo lives in the repo (the pip package ships the library only) — make sure you have cloned it and are in the repo root.

python web/run.py

Then open:

http://localhost:8787

By default, the demo mirrors terminal output — including Python logging and Uvicorn's own logs — to results/logs/voicemem-TIME-PID.log, one timestamped line per record, tagged stdout or stderr. The resolved path is printed at startup. To choose a path or disable file logging:

python web/run.py --log-file results/logs/debug.log
python web/run.py --no-file-log

Reply context combines the current input, turns from this session that are not yet represented by persistent memory, and retrieved memory. Each turn enters an in-memory SessionBuffer first. The asynchronous ingest completion callback removes it only after persistent memory is created. Buffers are isolated by Memory Space and WebSocket session.

Barge-in uses two stages during playback. VAD first pauses playback while preserving the audio queue. An explicit stop command or stable ASR updates confirm cancellation; backchannels, echo, non-text sounds, and isolated syllables resume playback. BARGE_REJECT_SILENCE_MS and BARGE_CANDIDATE_TIMEOUT_MS configure rejection timing.

Both reply modes share a PCM-sample media timeline. The browser AudioWorklet reports actual rendered progress, so interrupted context contains only the heard prefix. TTS providers may return TimedAudioChunk alignment metadata; plain PCM providers use segment duration and an adaptive speech-rate fallback.

🧠 VoiceMem: Memory with a Streaming Dual-Brain Architecture

VoiceMem is a memory system built for real-time voice agents.

Instead of storing every type of memory in a single retrieval database, VoiceMem separates memory into two complementary parts:

Left Brain organizes factual memory using schemas and entities for more accurate retrieval.
Right Brain manages personality, emotion, and relationships using independent and cross-entity memory nodes.

The entire pipeline is streaming.

While the user is still speaking, VoiceMem continuously segments audio, transcribes speech, extracts useful memories, and writes structured information into the memory graph.

At query time, VoiceMem routes first, ranks second, and injects only the Top-K memories into the model context. This keeps the context small while preserving the most relevant information.

Key Features

🎯 Accurate — Reaches 91.2% on LoCoMo, compared with 61.68% for Mem0, using only Top-5 memories.
❤️ Emotional & Personal — Remembers not only what the user said, but also who the user is and how they feel. Reaches 69.44% on PersonaMem.
🎧 Multimodal — Remembers speech, speakers, sound events, multi-speaker conversations, and music from real-world audio.
⚡ Fast — Responds in 134 ms, compared with 1,440 ms for Mem0, with streaming retrieval inside the voice turn.
💰 Low Token Usage — Uses only 430 memory tokens, compared with 6,956 for Mem0 and 1,899 for EverMemOS.

🤖 VoiceMem Model Families

We built ChatMem-400K through a three-stage OPD training pipeline:

Memory-world construction
SLM-validated online on-policy distillation (OPD)
Human refinement

After human editing, the same pipeline produces ChatMem-Bench, which evaluates whether a voice model can build a long-term understanding of the user over time.

The open-source VoiceMem model family includes Qwen2.5-Omni, Qwen3-Omni, and Step-Audio2-Mini. These models can receive and understand memory information provided by VoiceMem during conversations.

🔌 Customize Your Voice Agent with VoiceMem

You can integrate VoiceMem with your own voice model to build a real-time voice agent with long-term memory.

The basic flow is:

microphone → VoiceMem listens and prefetches relevant memories → your model reads those memories and generates a response

export OPENAI_API_KEY=sk-...
# Only used for fact extraction when writing memories.
# Memory retrieval runs entirely locally.

python examples/03_simple_agent_with_voicemem_memory.py

To use your own model, replace the generation step — the memory half stays exactly as is:

def my_reply(text, memory_context):        # a sync function is fine, it runs off-thread
    return my_model.generate(system=memory_context, user=text)

vm = VoiceMem(reply=my_reply)

🛠️ Finetuning

VoiceMem provides the complete finetuning pipeline for training your own VoiceMem Model Family adapter.

The default training configuration matches the one used for the released checkpoint-3318.

Running the following command with the default settings reproduces the same adapter:

pip install ms-swift==4.5.2 bitsandbytes

python finetune/train.py --data data/train.jsonl

See finetune/README.md for the training data format, GPU memory requirements, and instructions for using a different base model.

📊 Evaluation

The evaluation pipeline is fully open source and reproducible.

Run Evaluation

A benchmark can be started with a single command:

export OPENAI_API_KEY=sk-...

# Start with the small example included in the repository
# to make sure everything is set up correctly.
# 2 conversations, 5 questions.
python evaluation/run.py \
    --dataset locomo \
    --data evaluation/examples/locomo_sample.json

# Then run the full dataset.
python evaluation/run.py \
    --dataset locomo \
    --data data/locomo.json

Example result:

LoCoMo: 10 conversations · 152 questions

Score: 139/152 = 91.4%

  multi_hop     88.2%
  temporal      85.7%
  single_hop    95.1%

Median retrieval latency: 12 ms
Median retrieved memory: 298 tokens

Before running a full evaluation, add --inspect to check how the dataset is parsed.

This mode does not call the model and does not incur API costs:

python evaluation/run.py \
    --dataset locomo \
    --data data/locomo.json \
    --inspect

During evaluation, the answering model receives only the retrieved memories, not the original conversation history.

If the model receives the full conversation, the benchmark becomes a reading-comprehension test rather than an evaluation of the memory system itself.

See evaluation/README.md for the complete evaluation protocol and instructions for adding a new benchmark. Adding a benchmark only requires one file and two functions.

📖 Citation

If VoiceMem is useful for your research, please cite our paper:

@misc{2608.26005,
  author = {Zhifei Xie and Jiaqi Lang and Ze An and Yifan Zhao and Dongchao Yang and Kai Li and Ziyang Ma and Mingbao Lin and Chunyan Miao and Shuicheng Yan},
  title = {{V}oice{M}em: {S}treaming {D}ual-{B}rain {M}emory for {R}eal-{T}ime {I}nteraction},
  year = {2026},
  eprint = {2608.26005},
  note = {arXiv:2608.26005v1}
}

Acknowledgements

We thank the following excellent open-source projects:

mem0 — vector memory engine
FunASR — streaming ASR with paraformer-zh-streaming
sherpa-onnx — Silero VAD, 3D-Speaker speaker verification, and fallback streaming ASR
intfloat/multilingual-e5 — local embeddings and slot classification

VoiceMem also uses OpenAI APIs for Chat, TTS, and Realtime functionality.

License

VoiceMem is open source under the Apache License 2.0.

See LICENSE for details.

LICENSE

Shown as plain text.


                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

   APPENDIX: How to apply the Apache License to your work.

      To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "[]"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright [yyyy] [name of copyright owner]

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

177 files · 174 not served here

.gitignore (not served here)
ATTRIBUTION.md
LICENSE
README.md
assets/README.md (not served here)
assets/Voicemem_logo.webp (not served here)
assets/cafe_song.wav (not served here)
assets/evaluation.webp (not served here)
assets/huggingface_paper_gold_day.svg (not served here)
assets/input.wav (not served here)
assets/logo.png (not served here)
assets/question.wav (not served here)
assets/speech.wav (not served here)
assets/teaser.webp (not served here)
assets/wechat.jpg (not served here)
assets/wechat.png (not served here)
docs/images/background.webp (not served here)
docs/images/closing-bg.webp (not served here)
docs/images/cuhk.webp (not served here)
docs/images/fig-architecture.webp (not served here)
docs/images/fig-opd.webp (not served here)
docs/images/image1.webp (not served here)
docs/images/image2.webp (not served here)
docs/images/image3.webp (not served here)
docs/images/image4.webp (not served here)
docs/images/image_all.webp (not served here)
docs/images/ntu.webp (not served here)
docs/images/nus.webp (not served here)
docs/images/stages.webp (not served here)
docs/images/voicemem-logo.webp (not served here)
docs/index.html (not served here)
docs/old_version/index.html (not served here)
docs/output_normal.mp4 (not served here)
evaluation/.gitignore (not served here)
evaluation/README.md (not served here)
evaluation/datasets/__init__.py (not served here)
evaluation/datasets/locomo.py (not served here)
evaluation/examples/locomo_sample.json (not served here)
evaluation/run.py (not served here)
evaluation/score.py (not served here)
examples/01_memory.py (not served here)
examples/02_streaming.py (not served here)
examples/03_simple_agent_with_voicemem_memory.py (not served here)
examples/04_all_local_l40s.py (not served here)
examples/05_realtime_gpt_qwen.py (not served here)
examples/06_mic_memory.py (not served here)
examples/README.md (not served here)
examples/_audio.py (not served here)
finetune/README.md (not served here)
finetune/data/sample.jsonl (not served here)
finetune/dataset.py (not served here)
finetune/eval.py (not served here)
finetune/train.py (not served here)
finetune/utils.py (not served here)
pyproject.toml (not served here)
scripts/download_models.sh (not served here)
tests/test_offline_engine.py (not served here)
tests/test_session_context.py (not served here)
tools/reembed.py (not served here)
voicemem/__init__.py (not served here)
voicemem/assets/cafe_song.wav (not served here)
voicemem/assets/input.wav (not served here)
voicemem/assets/question.wav (not served here)
voicemem/assets/speech.wav (not served here)
voicemem/audio_timing.py (not served here)
voicemem/config.py (not served here)
voicemem/core.py (not served here)
voicemem/lang.py (not served here)
voicemem/leftbrain/__init__.py (not served here)
voicemem/leftbrain/brain.py (not served here)
voicemem/leftbrain/cognitive_graph/__init__.py (not served here)
voicemem/leftbrain/cognitive_graph/annotator.py (not served here)
voicemem/leftbrain/cognitive_graph/local_query_classifier.py (not served here)
voicemem/leftbrain/cognitive_graph/query_slot_classifier.py (not served here)
voicemem/leftbrain/cognitive_graph/slot_v2.py (not served here)
voicemem/leftbrain/cognitive_graph/store.py (not served here)
voicemem/leftbrain/cognitive_graph/store_v2.py (not served here)
voicemem/leftbrain/cognitive_graph/types.py (not served here)
voicemem/leftbrain/data/additive_extraction_prompt.txt (not served here)
voicemem/leftbrain/data/mem0_extract_relations_prompt.txt (not served here)
voicemem/leftbrain/data/mem0_update_memory_prompt.txt (not served here)
voicemem/leftbrain/extract_facts_openai.py (not served here)
voicemem/leftbrain/local_e5_embedder.py (not served here)
voicemem/leftbrain/local_memory_store.py (not served here)
voicemem/leftbrain/mem0_additive_prompt_build.py (not served here)
voicemem/leftbrain/mem0_backend_store.py (not served here)
voicemem/leftbrain/mem0_embedder.py (not served here)
voicemem/leftbrain/memory_repository.py (not served here)
voicemem/leftbrain/memory_repository_v2.py (not served here)
voicemem/leftbrain/merged_extraction.py (not served here)
voicemem/leftbrain/slot_split/__init__.py (not served here)
voicemem/leftbrain/slot_split/graph_entity_store.py (not served here)
voicemem/leftbrain/slot_split/split_manager.py (not served here)
voicemem/leftbrain/slot_split/split_store.py (not served here)
voicemem/leftbrain/slot_split/subgraph_manager.py (not served here)
voicemem/leftbrain/time_expand.py (not served here)
voicemem/llm_config.py (not served here)
voicemem/memory_api.py (not served here)
voicemem/orchestrator.py (not served here)
voicemem/reply.py (not served here)
voicemem/rightbrain/__init__.py (not served here)
voicemem/rightbrain/anchor_router.py (not served here)
voicemem/rightbrain/attribution_manager.py (not served here)
voicemem/rightbrain/brain.py (not served here)
voicemem/rightbrain/experience_repository.py (not served here)
voicemem/rightbrain/graph_store.py (not served here)
voicemem/rightbrain/store.py (not served here)
voicemem/rightbrain/traits_store.py (not served here)
voicemem/rightbrain/types.py (not served here)
voicemem/startup_check.py (not served here)
voicemem/stream.py (not served here)
voicemem/tts.py (not served here)
voicemem/utils/__init__.py (not served here)
voicemem/utils/audio/__init__.py (not served here)
voicemem/utils/audio/asr.py (not served here)
voicemem/utils/audio/audio_archive.py (not served here)
voicemem/utils/audio/emotion/__init__.py (not served here)
voicemem/utils/audio/emotion/attribution_prompt.py (not served here)
voicemem/utils/audio/emotion/attribution_qwen_omni.py (not served here)
voicemem/utils/audio/emotion/graph_memory.py (not served here)
voicemem/utils/audio/emotion/layer.py (not served here)
voicemem/utils/audio/emotion/memory_store.py (not served here)
voicemem/utils/audio/emotion/omni_generate.py (not served here)
voicemem/utils/audio/emotion/paper_emotion_detector.py (not served here)
voicemem/utils/audio/emotion/query_terms.py (not served here)
voicemem/utils/audio/emotion/types.py (not served here)
voicemem/utils/audio/emotion/vad_audio.py (not served here)
voicemem/utils/audio/emotion/vad_qwen_prompt.py (not served here)
voicemem/utils/audio/emotion/vad_trigger.py (not served here)
voicemem/utils/audio/environment/__init__.py (not served here)
voicemem/utils/audio/environment/audioset_labels.py (not served here)
voicemem/utils/audio/environment/environment_detector_ast.py (not served here)
voicemem/utils/audio/environment/environment_detector_clap.py (not served here)
voicemem/utils/audio/environment/music_memory.py (not served here)
voicemem/utils/audio/environment/place_memory.py (not served here)
voicemem/utils/audio/environment/routine_memory.py (not served here)
voicemem/utils/audio/environment/scene_classifier.py (not served here)
voicemem/utils/audio/environment/scene_trigger.py (not served here)
voicemem/utils/audio/perceiver.py (not served here)
voicemem/utils/audio/stream_io.py (not served here)
voicemem/utils/audio/voiceprint/__init__.py (not served here)
voicemem/utils/audio/voiceprint/adaptive_centroid.py (not served here)
voicemem/utils/audio/voiceprint/campplus_worker.py (not served here)
voicemem/utils/audio/voiceprint/speaker_encoder.py (not served here)
voicemem/utils/audio/voiceprint/speaker_identity.py (not served here)
voicemem/utils/audio/voiceprint/voiceprint_store.py (not served here)
voicemem/utils/common/__init__.py (not served here)
voicemem/utils/common/_graph_common.py (not served here)
voicemem/utils/common/cost_log.py (not served here)
voicemem/utils/common/embed_cache.py (not served here)
voicemem/utils/common/paths.py (not served here)
voicemem/utils/common/session_tracker.py (not served here)
voicemem/utils/common/space.py (not served here)
voicemem/utils/common/turn_ids.py (not served here)
voicemem/utils/common/voice_config.py (not served here)
voicemem/utils/common/voice_input.py (not served here)
voicemem/utils/defaults.py (not served here)
voicemem/utils/fusion/__init__.py (not served here)
voicemem/utils/fusion/config.py (not served here)
voicemem/utils/fusion/left_channel.py (not served here)
voicemem/utils/fusion/orchestrator.py (not served here)
voicemem/utils/fusion/prompt_builder.py (not served here)
voicemem/utils/fusion/reply_memory.py (not served here)
voicemem/utils/fusion/right_channel.py (not served here)
voicemem/utils/fusion/types.py (not served here)
voicemem_memoryspace/demo-en/.gitkeep (not served here)
voicemem_memoryspace/demo-zh/.gitkeep (not served here)
voicemem_memoryspace/myvoicemem/.gitkeep (not served here)
web/audio_timeline.py (not served here)
web/images/background.webp (not served here)
web/index.html (not served here)
web/logging_utils.py (not served here)
web/pcm-player-worklet.js (not served here)
web/run.py (not served here)
web/session_context.py (not served here)
web/utils.py (not served here)
web/voicemem.html (not served here)

Advantages

🎯 Accurate — Reaches 91.2% on LoCoMo, compared with 61.68% for Mem0, using only Top-5 memories. evidence: README.md
❤️ Emotional & Personal — Remembers not only what the user said, but also who the user is and how they feel. Reaches 69.44% on PersonaMem. evidence: README.md
🎧 Multimodal — Remembers speech, speakers, sound events, multi-speaker conversations, and music from real-world audio. evidence: README.md
⚡ Fast — Responds in 134 ms, compared with 1,440 ms for Mem0, with streaming retrieval inside the voice turn. evidence: README.md
💰 Low Token Usage — Uses only 430 memory tokens, compared with 6,956 for Mem0 and 1,899 for EverMemOS. evidence: README.md

Limitations

Needs an API key (OPENAI_API_KEY) (a rule-derived fact from the brain’s files, evidence: README.md)
No ZIP download: install it from its source (a rule-derived fact from the brain’s files)

Security

Risk level: Low risk

This brain runs

This brain runs nothing: AutoGineer found no hooks, scripts, plugin code or MCP servers in its files.

Low risk: network-instructions (pyproject.toml:22)

Attribution and licence

First committed by

xzf-thu

Declared authors

Lang Jiaqi (declared in pyproject.toml)

Publisher

Xie Zhifei (Community)

Source

xzf-thu/agbr-e2e-20261112t233300z-copy2 · commit 6cacb3c

Licence

Apache License 2.0 (files served here)

ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-13

Contributors

lang-jiaqi, xzf-thu, Natifishman, Sorbett, YIKUAIBANZI

AutoGineer is not affiliated with or endorsed by xzf-thu.

Install from source

174 of its files are not served here, so there is no ZIP. xzf-thu/agbr-e2e-20261112t233300z-copy2 at 6cacb3c